Pulumi Blog: S3

Uploading documents to your AI Slackbot in real-time using S3, SQS and Pulumi on AWS

Tyler Mulligan — Mon, 03 Jun 2024 17:21:02 +0100

In the introductory blog post, we learned to Create an AI Slack Bot to Chat with Your Data Using Embedchain, Pulumi on AWS, and continued with Adding data to Pinecone using S3, Embedchain, and Pulumi on AWS for an AI Slack bot.

For reference, here’s what our architecture looked like at the end of the second blog post.

To follow along, clone the project, git clone https://github.com/catmeme/arti.git or view it on GitHub.

S3 Events and SQS

AWS provides a few ways to integrate with AWS S3 Events, for this project, we’re going to focus on the latest and most straight-forward, S3 Event Notifications. Depending on your use case, integrating with EventBridge may make more sense.

While we can integrate S3 events directly with Lambda, what happens when we add 100’s of files at the same time? Potentially, hundreds of Lambdas could be invoked simultaneously, which could become an issue depending on other factors. We’ll solve for this by Using Lambda with Amazon SQS, and reserved concurrency.

Our Pulumi program is updated to add a Queue and integrate it with our new Lambda SQS handler (policies left out for brevity, you can view them in the deploy/pulumi/__main__.py in the project).

# Add a queue
sqs_queue = aws.sqs.Queue(...)

# Add a docker image runtime for our Lambda SQS handler
docker_image_lambda_sqs_handler = docker.Image(...)

# Add Lambda SQS handler
app_sqs_lambda = aws.lambda_.Function(
 resource_name=f"{app_name}-sqs-lambda",
 environment=lambda_environment, # We've added the sqs_queue.url here as well
 image_uri=docker_image_lambda_sqs_handler.repo_digest,
 opts=pulumi.ResourceOptions(depends_on=[docker_image_lambda_sqs_handler]),
 **common_lambda_options,
)

# Add event source mapping
event_source_mapping = aws.lambda_.EventSourceMapping(
 f"{app_name}-lambda-sqs-event-source-mapping",
 event_source_arn=sqs_queue.arn,
 function_name=app_sqs_lambda.name,
 enabled=True,
 batch_size=10,
)

Add the application code responsible for processing S3 events coming from SQS.

def handler(event, context):
 for record in event.get("Records", []):
 s3_event = json.loads(record.get("body", "{}"))
 for s3_record in s3_event.get("Records", []):
 object_key = s3_record.get("s3", {}).get("object", {}).get("key")
 if object_key:
 logger.info("Processed %s in bucket %s.", object_key, app_bucket_name)
 load_data(asset_location=f"{app_bucket_name}/{object_key}")

 return {"statusCode": 200, "body": json.dumps("Successfully processed S3 event(s) from SQS messages.")}

We also have a scheduled event that we set up for ingestion in the last post, which we’re going to disable in favor of real-time events.

That’s pretty much it on the infrastructure side, here’s the updated diagram.

Slack to S3 file upload bridge

So what do we do with this new functionality? How do we expose it to the user? The most obvious approach is to reuse our existing Slack integration.

We’ll start by creating a Slack file upload bridge to S3, the bot watches for file_shared events on Slack and re-uploads files to our S3 bucket.

Using Slack Bolt, we add a new function handle_file_upload with the @app.event("file_shared") decorator, binding it to said event. When that event occurs, it’s processed, and the details of our file are retrieved, a new payload is constructed using requests, and the file gets uploaded to S3, this is our bridge.

@app.event("file_shared")
def handle_file_uploads(event, say, client, logger):
 file_id = event['file']['id']
 channel_id = event['channel_id']

 # Fetch file details using the Slack API
 file_info = client.files_info(token=slack_bot_token, file=file_id)['file']
 file_url = file_info['url_private_download']

 # Download file content from Slack
 headers = {'Authorization': f"Bearer {slack_bot_token}"}
 response = requests.get(file_url, headers=headers, stream=True)

 if response.status_code == 200:
 # Upload the file to S3
 s3_client.upload_fileobj(
 Fileobj=response.raw,
 Bucket=os.getenv("APP_BUCKET_NAME"),
 Key=file_info['name']
 )

 # Fetch recent messages to find the correct timestamp
 messages = client.conversations_history(channel=channel_id, limit=10)['messages']
 for msg in messages:
 if msg.get('files') and any(f['id'] == file_id for f in msg.get('files', [])):
 # Found the message with the correct file
 message_ts = msg['ts']
 client.reactions_add(channel=channel_id, name="white_check_mark", timestamp=message_ts)
 # Respond in a thread to the message containing the file
 say(text=f"File `{file_info['name']}` uploaded to S3 successfully! :tada:", thread_ts=message_ts)
 return

 logger.error("No valid message found for reaction.")

Keep in mind, we’re not doing any validation here yet. We’re blindly copying files from Slack to S3 and having our data loader process them. It’s probably not a good idea to deploy this bot to a public Slack channel without more moderation/validation in place.

Here’s what it looks like when we drag/drop a file into Slack now. On the right, a file was shared with the bot in private message. The bot acknowledged this upload with the checkmark reaction, the job is placed on the queue. Shortly after, on the left, the SQS Lambda handler has finished processing the uploaded file. We’re now able to use the bot to talk about the uploaded files.

Slack interactivity

In minutes, we were able to move from a scheduled task for processing files in a bucket, to real-time file processing integrated with Slack. We have our /arti command we can use to interrogate the documents we just uploaded, but we’re going to have a little more fun.

Thanks to Slack Block Kit Builder, we can build a form that exposes all of the configuration options for the query method in Embedchain’s app, providing the user with more options for interacting with the RAG model.

Our code to utilize the JSON we’ve crafted in the builder looks something like (we’ll call this /smarti):

@app.command("/smarti")
def open_smarti_modal(ack, body, client):
 ack()
 print("Opening SMARTI modal...")
 modal_file_path = Path(config.project_root) / 'config' / 'slack_modals' / 'chat.json'

 with open(modal_file_path, 'r') as json_file:
 modal_view = json.load(json_file)

 client.views_open(
 trigger_id=body["trigger_id"],
 view=modal_view
 )

And the code for handling that submission looks something like:

def handle_smarti_submission(client, logger, body, view):
 values = view["state"]["values"]
 question = values["question"]["input"]["value"]
 model = values["model"]["input"]["selected_option"]["value"]
 temperature = float(values["temperature"]["input"]["value"])
 max_tokens = int(values["max_tokens"]["input"]["value"])
 top_p = float(values["top_p"]["input"]["value"])
 prompt = values["prompt"]["input"]["value"]
 system_prompt = values["system_prompt"]["input"]["value"]
 citation_options = values["citations"]["input"].get("selected_options", [])
 citations = any(option['value'] == 'true' for option in citation_options)

 ai_response = ask_ai(
 question=question,
 model=model,
 temperature=temperature,
 max_tokens=max_tokens,
 top_p=top_p,
 prompt=prompt,
 system_prompt=system_prompt,
 citations=citations
 )
 ...

Submitting that form results in a response such as below. The bot honored our prompts, talking like Shakespeare, a thread is created, and citations were noted, showing where the data was retrieved from.

Lastly, if we want interact with the bot more publicly, then we can use the app_mention event.

@app.event("app_mention")
def handle_app_mention(event, say, client):
 text = event['text']
 bot_user_id = client.auth_test()["user_id"]

 # Strip the mention of the bot to isolate the command and arguments
 command_text = text.split(f'<@{bot_user_id}>')[1].strip() if f'<@{bot_user_id}>' in text else text

 # Parse the command and arguments
 parts = command_text.split(maxsplit=1)
 if not parts:
 say("I need a command to do something useful!")
 return

 # Get the command and the remainder of the message as arguments
 command = parts[0].lower()
 args = parts[1] if len(parts) > 1 else ""

 if command == "ask":
 if args:
 response = ask_ai(question=args)
 say(response)
 else:
 say("Please provide a question to ask.")
 elif command == "list":
 list_data_sources(say)
 else:
 say(f"Sorry, I don't recognize the command `{command}`.")

Resulting in an interaction such as

Of course, this same methodology can be used on any of the events exposed by Slack.

To learn more about extending this bot, see Slack Bolt’s documentation.

Conclusion

In this blog post, we took our existing Slack bot which has a data loading process that loads documents from an S3 bucket into a vector database on a schedule, and we wired it up to behave in real time.

To expose this functionality to the end user, we created a bridge between file uploads to Slack and S3, allowing drag/drop capabilities of files in conversational chat.

Furthering our interactions with our application, we leveraged Slack’s Block Kit to create a custom modal, and interact with events.

As always, we welcome your feedback and contributions in the Pulumi Community Slack, GitHub repository, and Pulumi Community Discussions.

New to Pulumi? Signing up is easy and free. Get started today!

Happy AI building!

10x Your Storage Performance with Amazon S3 Express One Zone and Pulumi

Joe Duffy — Mon, 04 Dec 2023 00:00:00 +0000

At AWS re:Invent 2023, AWS announced the new Amazon S3 Express One Zone storage class. This new service provides incredible performance by collocating your S3 buckets closer to the workloads that use the data. Compared to the S3 Standard storage class, the Express One Zone storage class is up to 10x faster, handles 100,000s of requests per second, offers single-digit millisecond latency, and can reduce request costs by 50%. This can be extremely beneficial for data-intensive workloads such as AI/ML, media, finance, realtime, and high-performance computing scenarios. This blog post shows how to get started with Amazon S3 Express One Zone using Pulumi infrastructure as code.

Introducing Amazon S3 Express One Zone

The new Amazon S3 Express One Zone storage class extends S3 with a new low latency option. This can improve runtime performance by spending less time moving data around. When creating a new bucket, you choose an Availability Zone (AZ) in which that bucket will be placed, which of course should be collocated with the compute workload that will consume that data.

A bucket must be designated as an Express One Zone bucket at creation time. There are a few things that are unique to these buckets that make them different from ordinary S3 buckets:

First, they are called “directory buckets” in AWS tools and in your infrastructure as code. This means how you interact with them sometimes differs from general-purpose S3 buckets.
They have a specific naming convention: [name]--[az]--x-s3. The [name] part is a bucket name with the usual restrictions. [az] is the ID of the AZ in which the bucket will reside. Note that this is an ID (e.g. usw2-az1), not the name (e.g. us-west-1c); read more here about the difference.
These buckets use region-specific endpoints, not the region-agnostic aws-global S3 endpoint scheme that general-purpose S3 buckets utilize. This means you’ll need a new AWS CLI or SDK that understands the new endpoint scheme in order to access these buckets (CLI version 2.14.5 and greater supports them). It also means that sometimes when running aws s3 and aws s3api commands, you will need to specify a --region explicitly. Other aspects of accessing these buckets differ, such as authorization, but the AWS SDK generally takes care of these for you.

This new storage class is not universally available in all regions and AZs yet, although Amazon has plans to expand regions over time. At the time of launch, us-east-1 (US North Virginia), us-west-2 (US Oregon), eu-north-1 (Europe Stockholm), and ap-northeast-1 (Asia Pacific Tokyo) are supported. Finally, because One Zone is explicitly single-AZ, there is always the risk of availability issues or data loss in the event that a single AZ becomes unavailable or is damaged; plan accordingly.

New BucketDirectory Resource Type

You can create a new Amazon S3 Express One Zone bucket using the aws s3api create-bucket CLI command or in the AWS console. For this blog post, however, you’ll use infrastructure as code instead, so that provisioning, updating, and managing our buckets is automated, reliable, and easy to scale.

To do this, you will use Pulumi’s new aws.s3.DirectoryBucket resource type in the AWS package. It takes a few parameters:

bucket, which is the name (conforming to the above-mentioned [name]--[az]--x-s3 convention)
dataRedundancy, which currently just accepts the sole value "SingleAvailabilityZone"
location, which specifies the AZ ID to use
forceDestroy, an optional boolean which, when set to true, allows deleting buckets that still have data in them (by default, only empty buckets can be deleted, to prevent accidental data loss)

Here is a simple program that declares a new Amazon S3 Express One Zone bucket:

import * as aws from "@pulumi/aws";
import * as pulumi from "@pulumi/pulumi";
import * as random from "@pulumi/random";

// Pick a single AZ for our S3 Express bucket.
const azId = aws.getAvailabilityZones().then(azs => azs.zoneIds.sort()[0]);

// Name our bucket according to S3 Express requirements ([name]--[az]--x-s3).
const bucketName = "my-bucket";
const suffix = new random.RandomString("bucket-name", {
 length: 6, upper: false, special: false });
const bucketFullName = pulumi.interpolate
 `${bucketName}-${suffix.result}--${azId}--x-s3`;

// Provision the bucket and export its ID.
const bucket = new aws.s3.DirectoryBucket(bucketName, {
 bucket: bucketFullName,
 dataRedundancy: "SingleAvailabilityZone",
 location: {
 type: "AvailabilityZone",
 name: azId,
 },
});

export const bucketID = bucket.id;

import pulumi
import pulumi_aws as aws
import pulumi_random as random

# Pick a single AZ for our S3 Express bucket.
az_ids = aws.get_availability_zones().zone_ids
az_ids.sort()
az_id = az_ids[0]

# Name our bucket according to S3 Express requirements ([name]--[az]--x-s3).
bucket_name = 'my-bucket'
suffix = random.RandomString(
 'bucket-name', length=6, upper=False, special=False)
bucket_full_name = pulumi.Output.format(
 '{0}-{1}--{2}--x-s3', bucket_name, suffix.result, az_id)

# Provision the bucket and export its ID.
bucket = aws.s3.DirectoryBucket(bucket_name,
 bucket = bucket_full_name,
 data_redundancy = 'SingleAvailabilityZone',
 location = {
 'type': 'AvailabilityZone',
 'name': az_id,
 },
)

pulumi.export('bucketID', bucket.id)

There are a few things to note here:

We’re picking the first AZ automatically in whatever region we’ve deployed to. Of course, we could hard-code this, or use a different technique for choosing an AZ. This approach makes it trivial to deploy the same project to multiple regions.
We use a random string suffix for our bucket name, just to avoid accidental collisions if we were to pick the same bucket name as another user (or even ourselves when deploying multiple stacks for the same project). Normally Pulumi uses auto-naming to avoid this problem by default, but because the bucket name must conform to a very precise format, it does not for this resource type.

Now let’s say we actually want to populate our bucket. We can use the standard aws.s3.BucketObjectv2 resource to do this. This program adds an object for every file in our project’s data directory, for example:

import * as fs from "fs";
import * as path from "path";

// ...

// Add a bucket object for every data file.
for (const file of fs.readdirSync("data")) {
 new aws.s3.BucketObjectv2(`my-file-${file}`, {
 bucket: bucket.id,
 source: new pulumi.asset.FileAsset(path.join("data", file)),
 }, { parent: bucket });
}

import os

# ...

# Add a bucket object for every data file.
for file in os.listdir('data'):
 path = os.path.join('data', file)
 if os.path.isfile(path):
 aws.s3.BucketObjectv2(f'my-file-{file}',
 bucket = bucket.id,
 source = pulumi.asset.FileAsset(path),
 )

Creating and Accessing an Express One Zone Bucket

Let’s go ahead and spin up a bucket. Assuming we have the above code in a valid Pulumi project (created with pulumi new), let’s ensure it is configured to one of the regions that supports Amazon S3 Express One Zone, such as us-west-2:

$ pulumi config set aws:region us-west-2

Now let’s create some random files in our data directory:

$ mkdir data
$ for i in {0..9}; do \
 dd if=/dev/urandom of=data/file${i}.txt bs=100K count=1; \
 done

Next, we’ll run pulumi up which will show us a preview of the deployment and ask us to confirm, which we’ll do:

$ pulumi up
 Type Name Status
 + pulumi:pulumi:Stack s3-express-one-zone-py-dev created (4s)
 + ├─ random:index:RandomString bucket-name created (0.09s)
 + ├─ aws:s3:DirectoryBucket my-bucket created (2s)
 + ├─ aws:s3:BucketObjectv2 my-file-file2.txt created (1s)
 + ├─ aws:s3:BucketObjectv2 my-file-file3.txt created (1s)
 + ├─ aws:s3:BucketObjectv2 my-file-file1.txt created (1s)
 + ├─ aws:s3:BucketObjectv2 my-file-file4.txt created (1s)
 + ├─ aws:s3:BucketObjectv2 my-file-file5.txt created (1s)
 + ├─ aws:s3:BucketObjectv2 my-file-file7.txt created (2s)
 + ├─ aws:s3:BucketObjectv2 my-file-file8.txt created (2s)
 + ├─ aws:s3:BucketObjectv2 my-file-file9.txt created (2s)
 + ├─ aws:s3:BucketObjectv2 my-file-file0.txt created (3s)
 + └─ aws:s3:BucketObjectv2 my-file-file6.txt created (3s)

Outputs:
 bucketID: "my-bucket-z5tn74--usw2-az1--x-s3"

Resources:
 + 13 created

Duration: 11s

And there we go – our new Amazon S3 Express One Zone bucket is available and its ID has been printed. We can check out its contents:

$ aws s3api list-objects-v2 \
 --region=us-west-2 \
 --bucket $(pulumi stack output bucketID)

Note that the storage class of these objects is listed as EXPRESS_ONEZONE:

{
"Contents": [
{
"Key": "my-file-file6.txt",
"LastModified": "2023-12-03T17:18:06+00:00",
"ETag": "\"1254d05e685245ef9fdcca10f3c547bc\"",
"ChecksumAlgorithm": [
"CRC32"
],
"Size": 102400,
"StorageClass": "EXPRESS_ONEZONE"
},
...
{
"Key": "my-file-file2.txt",
"LastModified": "2023-12-03T17:18:04+00:00",
"ETag": "\"d5310d3efd394c96bf5bf4379fc81bff\"",
"ChecksumAlgorithm": [
"CRC32"
],
"Size": 102400,
"StorageClass": "EXPRESS_ONEZONE"
}
]
}

We can also see our bucket and its contents in the S3 Management Console:

General-purpose buckets and directory buckets have different tabs. There are also utilities available to import data from other buckets and perform batch operations.

Benchmarking Express One Zone Performance

To demonstrate how much faster Express One Zone buckets are to access, let’s run a few benchmarks. To do this, we set up two buckets – an Express One Zone and another using Standard storage class – along with an EC2 instance in the same AZ as our One Zone bucket.

For the first test, we’ll allocate 100 files, each 10MB in size, for a total of 1GB of data. We’ll then run an aws s3 sync from our EC2 instance to copy the entirety of the bucket locally. We’ll loop this 10 times and simply measure how long it took for the copy to complete. Obviously this only tests read performance, but it will immediately give us a sense of how much faster Express One Zone can be:

Standard class: 8m5s
Express One Zone class: 2m41s
Speedup: 3.0x

This was a nice bump in performance, however, the new storage class really shines for smaller objects. Storage latency typically impacts small files much more than larger files, as storage overheads and latencies become more of a bottleneck than streaming the data over the network. For the second test, we’ll try 100,000 files, each 100K in size instead – still 1GB total, but comprised of many smaller files:

Standard class: 30m45s
Express class: 5m36s
Speedup: 5.5x

From 3x to more than 5x the performance! With more fine tuning of our sync parallelism and other factors, we could have eked out even more gains, but these simple tests quickly showed us how much faster Express One Zone can be for some very basic examples.

In summary, here are the results:

Conclusion

In this post, we learned all about Amazon S3 Express One Zone, how to manage new buckets and their contents with infrastructure as code, and saw the awesome performance gains in action. To be up and running with Amazon S3 Express One Zone and infrastructure as code in minutes, go ahead and get started with Pulumi for free today. Happy cloud spelunking!

Deploying Netlify CMS on AWS with Pulumi

Zephyr Zhou — Tue, 01 Sep 2020 00:00:00 +0000

Netlify CMS is an open-source content management system that provides UI for editing content and adopting Git workflow. Initially, we want to take advantage of it to increase efficiency to edit Pulumi’s website. However, during development, we found few examples are deploying the CMS application on AWS instead of Netlify, its home platform. Therefore, in this blog post, we would like to share how to organize Netlify’s file structure and use Pulumi to store the content on S3 buckets, connect to CloudFront, and configure certificate in Certificate Manager.

Because we are deploying CMS on AWS, we can not use Netlify’s Identity Service, which manages the access token sent by Github API. We will introduce how to write the External OAuth Client-Server and deploy it on AWS in the next post.

For this project, we use Netlify CMS’s Github backend. Our CMS app changes website content stored in another Github repository under the same account. The website uses Hugo as the static site generator.

Extracting CMS As a Stand-alone React App

The starter template provided by Netlify implements the CMS application inside the target repository. We extract the CMS application from the target repository and copy it in another repository under the same Github account to improve the modularity.

We can use the templates from @talves to extract the CMS as a stand-alone React application.

CMS Configuration File

The config.yml file configures Netlify CMS to look for content segments that can be edited and which backend instructions it needs to follow.

The template, cms/public/index.html contains the entry point for React to insert the generated content under <div id=root>. We can change website’s title, but more importantly, we linked the configuration file cms/public/config.yml of the Netlify CMS by following the instructions.

The template uses config.json in ./src/components/NetlifyCMS/config.json, and sets CMS Manual Init in file ./src/components/NetlifyCMS/setup.js and file ./src/components/NetlifyCMS/index.js.

Alternatively, we can add the path to the config.yml file in index.html and put config.yml in the public folder. This reduces the steps to set the CMS init step, which can be confusing to beginners.

Point the CMS to Another Repository

The template separates the CMS as a stand-alone React app, while the CMS remains in the target repository. We can change the value field repo: in the backend: field of the config.yml file to point to the specific repository that the CMS can change.

File Structure

Simplifying the file and deleting unneeded parameters produces this:

The ./src/index.js file is the entry point for React to create the App class inside ./public/index.html’s <div id=root>. App (defined in ./src/App.js) is a React Component that returns the NetlifyCMS React component defined inside ./src/components/NetlifyCMS/index.js. We can also specify the custom preview and custom widget for the CMS app in this file.

To preview the CMS app content, run yarn start.

Now that we have preprocessed the file structure. Let’s deploy the CMS onto the AWS!

Implement Infrastructure

First, we create an infrastructure folder to store the Pulumi project. To create a new Pulumi project, run pulumi new typescript to generate a typescript template.

We can use Pulumi’s static website example as a model for deploying our application.

Step 1: Build the CMS App

We need to build the website with yarn build. This creates a build folder under the root directory that holds the compiled version of the website contents to be uploaded to an S3 bucket. Once uploaded, the S3 bucket contains a working version of our CMS app, which reduces the number of files to upload.

Step 2: Fetch Configuration

The first step is to create a stack configuration with our stack name.

// Replace "pulumi-website-cms" with your stack name
const stackConfig = new pulumi.Config("pulumi-website-cms");
const config = {
 pathToWebsiteContents: stackConfig.require("pathToWebsiteContents"),
 targetDomain: stackConfig.require("targetDomain"),
 certificateArn: stackConfig.get("certificateArn"),
};

The configuration takes three parameters set with the Pulumi CLI

$ pulumi config set pulumi-website-cms:pathToWebsiteContent ../build

The pathToWebsiteContent parameter specifies the path to the directory with the website content. This directory is synched to the S3 bucket by using the crawlDirectory method. We pass in the build folder, which is the directory that stores our built CMS website.

Similarly, we can set the CMS’s target domain name and the optional certificate arn parameter with stackConfig.get instead of stackConfig.require to get the value. For the target domain, we can use any subdomain name of an existing parent domain, and the code creates the subdomain under the parent domain automatically. For the certificate arn, if we already have a certificate registered for CMS, then we can pass it in as the certificate arn. Otherwise, the code generates a new certificate.

# replace the targetDomain with the domain name for your CMS
$ pulumi config set pulumi-website-cms:targetDomain https://some-cms-domain.com
# replace the value of certificateArn with the correct one
$ pulumi config set pulumi-website-cms:certificateArn arnarnarnxxx

Note that the ACM certificate has to be in the same as the project AWS region.

$ pulumi config set aws:region us-east-1

Step 3: Bucket Creation

We create the S3 bucket.

// contentBucket is the S3 bucket that the website's contents will be stored in.
const contentBucket = new aws.s3.Bucket("contentBucket",
 {
 bucket: config.targetDomain,
 acl: "public-read",
 // Configure S3 to serve bucket contents as a website. This way S3 will automatically convert
 // requests for "foo/" to "foo/index.html".
 website: {
 indexDocument: "index.html",
 },
 });

We crawl the content directory and convert them to bucket objects that are the content for the site.

function crawlDirectory(dir: string, f: (_: string) => void) {
 const files = fs.readdirSync(dir);
 for (const file of files) {
 const filePath = `${dir}/${file}`;
 const stat = fs.statSync(filePath);
 if (stat.isDirectory()) {
 crawlDirectory(filePath, f);
 }
 if (stat.isFile()) {
 f(filePath);
 }
 }
}

// Sync the contents of the source directory with the S3 bucket, which will in-turn show up on the CDN.
const webContentsRootPath = path.join(process.cwd(), config.pathToWebsiteContents);
console.log("Syncing contents from local disk at", webContentsRootPath);
crawlDirectory(
 webContentsRootPath,
 (filePath: string) => {
 const relativeFilePath = filePath.replace(webContentsRootPath + "/", "");
 const contentFile = new aws.s3.BucketObject(
 relativeFilePath,
 {
 key: relativeFilePath,

 acl: "public-read",
 bucket: contentBucket,
 contentType: mime.getType(filePath) || undefined,
 source: new pulumi.asset.FileAsset(filePath),
 },
 {
 parent: contentBucket,
 });
 });

We also create a private CDN request log bucket for storing logs that we can use for debugging the application.

// logsBucket is an S3 bucket that will contain the CDN's request logs.
const logsBucket = new aws.s3.Bucket("requestLogs",
 {
 bucket: `${config.targetDomain}-logs`,
 acl: "private",
 });

Step 4: Certificate Creation and Validation

If the certificateArn is not provided as a configuration, the code will create a new certificate. We set the AWS region to match the Project region, which ACM certificate requires.

Then we create a new certificate.

const certificate = new aws.acm.Certificate("certificate", {
 domainName: config.targetDomain,
 validationMethod: "DNS",
 }, { provider: eastRegion });

Next, we create a Route53 DNS Record which is needed when requesting the ACM certificate.

const domainParts = getDomainAndSubdomain(config.targetDomain);
const hostedZoneId = aws.route53.getZone({ name: domainParts.parentDomain }, { async: true }).then(zone => zone.zoneId);

/**
 * Create a DNS record to prove that we _own_ the domain we're requesting a certificate.
 * See https://docs.aws.amazon.com/acm/latest/userguide/gs-acm-validate-dns.html for more info.
 */
const certificateValidationDomain = new aws.route53.Record(`${config.targetDomain}-validation`, {
 name: certificate.domainValidationOptions[0].resourceRecordName,
 zoneId: hostedZoneId,
 type: certificate.domainValidationOptions[0].resourceRecordType,
 records: [certificate.domainValidationOptions[0].resourceRecordValue],
 ttl: tenMinutes,
});

The getDomainAndSubdomain method for separates the parent domain and child domain.

We validate the certificate by setting the CertificateValidation resource and set the certificateArn.

/**
 * This is a _special_ resource that waits for ACM to complete validation via the DNS record
 * checking for a status of "ISSUED" on the certificate itself. No actual resources are
 * created (or updated or deleted).
 *
 * See https://www.terraform.io/docs/providers/aws/r/acm_certificate_validation.html for slightly more detail
 * and https://github.com/terraform-providers/terraform-provider-aws/blob/master/aws/resource_aws_acm_certificate_validation.go
 * for the actual implementation.
 */
 const certificateValidation = new aws.acm.CertificateValidation("certificateValidation", {
 certificateArn: certificate.arn,
 validationRecordFqdns: [certificateValidationDomain.fqdn],
 }, { provider: eastRegion });

 certificateArn = certificateValidation.certificateArn;

Configure the CloudFront distribution

We set the configuration parameters for CloudFront in distributionArgs :

// distributionArgs configures the CloudFront distribution. Relevant documentation:
// https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/distribution-web-values-specify.html
// https://www.terraform.io/docs/providers/aws/r/cloudfront_distribution.html
const distributionArgs: aws.cloudfront.DistributionArgs = {
 enabled: true,
 // Alternate aliases the CloudFront distribution can be reached at, in addition to https://xxxx.cloudfront.net.
 // Required if you want to access the distribution via config.targetDomain as well.
 aliases: [ config.targetDomain ],

 // We only specify one origin for this distribution, the S3 content bucket.
 origins: [
 {
 originId: contentBucket.arn,
 domainName: contentBucket.websiteEndpoint,
 customOriginConfig: {
 // Amazon S3 doesn't support HTTPS connections when using an S3 bucket configured as a website endpoint.
 // https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/distribution-web-values-specify.html#DownloadDistValuesOriginProtocolPolicy
 originProtocolPolicy: "http-only",
 httpPort: 80,
 httpsPort: 443,
 originSslProtocols: ["TLSv1.2"],
 },
 },
 ],

 defaultRootObject: "index.html",

 // A CloudFront distribution can configure different cache behaviors based on the request path.
 // Here we just specify a single, default cache behavior which is just read-only requests to S3.
 defaultCacheBehavior: {
 targetOriginId: contentBucket.arn,

 viewerProtocolPolicy: "redirect-to-https",
 allowedMethods: ["GET", "HEAD", "OPTIONS"],
 cachedMethods: ["GET", "HEAD", "OPTIONS"],

 forwardedValues: {
 cookies: { forward: "none" },
 queryString: false,
 },

 minTtl: 0,
 defaultTtl: tenMinutes,
 maxTtl: tenMinutes,
 },

 // "All" is the most broad distribution, and also the most expensive.
 // "100" is the least broad, and also the least expensive.
 priceClass: "PriceClass_100",

 // You can customize error responses. When CloudFront receives an error from the origin (e.g. S3 or some other
 // web service) it can return a different error code, and return the response for a different resource.
 // customErrorResponses: [
 // { errorCode: 404, responseCode: 404, responsePagePath: "/404.html" },
 // ],

 restrictions: {
 geoRestriction: {
 restrictionType: "none",
 },
 },

 viewerCertificate: {
 acmCertificateArn: certificateArn, // Per AWS, ACM certificate must be in the us-east-1 region.
 sslSupportMethod: "sni-only",
 },

 loggingConfig: {
 bucket: logsBucket.bucketDomainName,
 includeCookies: false,
 prefix: `${config.targetDomain}/`,
 },
};

const cdn = new aws.cloudfront.Distribution("cdn", distributionArgs);

Step 5: Alias Record

Next, we create an alias record that points to the CloudFront distribution.

function createAliasRecord(
 targetDomain: string, distribution: aws.cloudfront.Distribution): aws.route53.Record {
 const domainParts = getDomainAndSubdomain(targetDomain);
 const hostedZoneId = aws.route53.getZone({ name: domainParts.parentDomain }, { async: true }).then(zone => zone.zoneId);
 return new aws.route53.Record(
 targetDomain,
 {
 name: domainParts.subdomain,
 zoneId: hostedZoneId,
 type: "A",
 aliases: [
 {
 name: distribution.domainName,
 zoneId: distribution.hostedZoneId,
 evaluateTargetHealth: true,
 },
 ],
 });
}

const aRecord = createAliasRecord(config.targetDomain, cdn);

We’ve finished implementing our application infrastructure, and we’re ready to deploy. Run pulumi up to execute the Pulumi code to deploy to AWS. However, there is a more convenient way to do so with Github.

Github Workflow (Optional)

Github Actions adopts the Continuous Delivery/Integration concept and specifies a series of steps to run and deploy the program. We can take advantage of it with our CMS.

In the workflow, Github secrets are accessed from ${{ secrets.ACCESS_TOKEN }} where ACCESS_TOKEN is the Github credential. We can specify the secret in the project setting of Github.

PULUMI_STACK is the Pulumi stack name that we are using. PULUMI_ACCESS_TOKEN is required for the automatic process and can be generated by selecting “New Access Token” in the Pulumi Service settings.

Summary and Next Step

We have deployed the stand-alone React CMS application on AWS. The complete code is available in Pulumi’s Example Repository.

The next article will show how to deploy on AWS instead of Netlify. We describe how to substitute the Netlify Identity Service by writing an External OAuth Client-Server and deploying it on AWS. Stay tuned for our next blog post.

Getting Started With PaC

Sophia Parafina — Tue, 11 Feb 2020 00:00:00 +0000

Modern applications have brought many benefits and improvements, including the ability to scale and rapid iterations to update software. However, this has come at the cost of complexity. Modern infrastructure is composed of many resources that require detailed configuration to work correctly and securely. Even managed solutions from cloud service providers need additional configuration to ensure that services are secure and free of defects. Cloud providers, such as AWS, do allow you to create policies to ensure that applications are secure, but they are specific to resources that are already deployed. A significant benefit of Policy as Code is the ability to verify and spot problems before deploying your infrastructure.

Leaky Buckets

What do many of the most innovative companies in the modern cloud era have in common? They’ve all had massive data breaches from leaky AWS S3 Buckets. One of the most common reasons for data breaches is misconfigured AWS S3 Buckets. Both security professionals and bad actors use tools such as slurp to find open S3 buckets by just entering a domain, keyword, or even AWS credentials with the slurp CLI. Ideally, we want to proactively prevent public access to our S3 buckets before they are created.

Pulumi’s CrossGuard helps customers adhere to security best practices by checking the final state of a deployment against a set of policies and revealing any non-compliant services. CrossGuard is an SDK for writing policies for services, and it includes best practices playbooks for cloud providers such as AWS, Azure, and Google Cloud Platform. The Getting Started with PaC covers the case of leaking S3 buckets, but let’s do a quick review with an example showing how we can detect a leaky bucket before it gets deployed.

Note: If you haven’t installed Pulumi, follow the Pulumi Getting Started Start by creating an S3 bucket with Pulumi.

$ pulumi new aws-typescript

Running pulumi new creates the project files as well as the code to create a single S3 bucket. We’ll use an S3 ACL to make the bucket public by setting it to the canned ACL public-read.

const bucket = new aws.s3.Bucket("my-bucket", {
 acl: "public-read",
});

Next, we’ll create a Policy Pack, by creating a directory and making it the current working directory.

$ mkdir policypack && cd policypack

Then create the Policy Pack using pulumi policy new.

$ pulumi policy new aws-typescript

By default, this creates a policy that checks to see if the ACL sets the bucket to be publicly accessible.

new PolicyPack("aws-typescript", {
 policies: [{
 name: "s3-no-public-read",
 description: "Prohibits setting the publicRead or publicReadWrite permission on AWS S3 buckets.",
 enforcementLevel: "mandatory",
 validateResource: validateResourceOfType(aws.s3.Bucket, (bucket, args, reportViolation) => {
 if (bucket.acl === "public-read" || bucket.acl === "public-read-write") {
 reportViolation(
 "You cannot set public-read or public-read-write on an S3 bucket. " +
 "Read more about ACLs here: https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html");
 }
 }),
 }],
});

A policy consists of a name; a description that is displayed when the policy is violated; an enforcement level of either advisory or mandatory; and validateResource which specifies the type of resource, the resource, any arguments, and the message to report the violation.

We can test this locally using pulumi preview

$ pulumi preview --policy-pack <path-to-policy-pack-directory>

The preview will fail and return this output:

Diagnostics:
 pulumi:pulumi:Stack (test-dev):
 error: preview failed

 aws:s3:Bucket (my-bucket):
 mandatory: [s3-no-public-read] Prohibits setting the publicRead or publicReadWrite permission on AWS S3 buckets.
 You cannot set public-read or public-read-write on an S3 bucket. Read more about ACLs here: https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html

CrossGuard validated the desired state of the S3 bucket against the no-public-read policy stopped the preview. If we were deploying the S3 bucket, it would stop the deployment until the policy is met.

Exposed Databases

Like S3 buckets, exposed services are another source of data breaches. Misconfigured databases, failing to set passwords, or dev services open to the public are responsible for releases of Personal Identifiable Information (PII) that range well into the millions of customers. Many of these breaches were due to misconfigured Elasticsearch instances that were public and not secured with a password, such as the recent exposure of a Microsoft customer service database.

We’ll take the same approach that we used to secure S3 buckets. First, we’ll create an Elasticsearch deployment, write a policy, then apply it.

$ pulumi new aws-typescript

Replace index.ts with the following to deploy Elasticsearch:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

const example = new aws.elasticsearch.Domain("example", {
 clusterConfig: {
 instanceType: "r4.large.elasticsearch",
 },
 elasticsearchVersion: "1.5",
 snapshotOptions: {
 automatedSnapshotStartHour: 23,
 },
 tags: {
 Domain: "TestDomain",
 },
});

In this example, we’ll use AWSGuard, which is a set of best practice policies for AWS. First, create a directory, change into it, and create a new policy.

$ mkdir awsguard && cd awsguard
$ pulumi policy new awsguard-typescript

The default policy enforcement level is advisory, but given the frequency of misconfigured Elasticsearch deployments, we’ll set to mandatory.

new AwsGuard({ all: "mandatory" });

When we run pulumi preview with the AWSGuard PolicyPack, we can see how it triggers violations based on best practices that can prevent data exposure.

$ pulumi preview --policy-pack ../awsguard/
Previewing update (dev):

 Type Name Plan Info
 + pulumi:pulumi:Stack elasticsearch-dev create 1 error
 + └─ aws:elasticsearch:Domain example create 2 errors

Diagnostics:
 pulumi:pulumi:Stack (elasticsearch-dev):
 error: preview failed

 aws:elasticsearch:Domain (example):
 mandatory: [elasticsearch-encrypted-at-rest] Checks if the Elasticsearch Service domains have encryption at rest enabled.
 Elasticsearch domain example-5510ef8 must be encrypted at rest.
 mandatory: [elasticsearch-in-vpc-only] Checks that the Elasticsearch domain is only available within a VPC, and not accessible via a public endpoint.
 Elasticsearch domain example-5510ef8 must run within a VPC.


Policy Packs run:
 Name Version
 pulumi-awsguard (../awsguard) (local)

Permalink: https://app.pulumi.com/spara/elasticsearch/dev/previews/bb6df58f-69aa-485a-99a4-d076bc067b84

AWSGuard flagged two recommended Elasticsearch best practices, encryption and deploying only within a VPC and not publicly. By enabling encryption at rest, unauthorized users cannot read data in Elasticseach clusters, storage, indices, and log files. By enabling Elasticsearch’s built-in encryption, your deployment follows recommended practices that comply with HIPAA and GDPR. AWS VPCs can keep the network traffic between your applications and Elasticsearch cluster within your infrastructure’s network and not over the public internet. Isolating Elasticsearch within your infrastructure can help mitigate the likelihood of data exposure.

CrossGuard available in Pulumi 2.0

The examples in this article demonstrate how to avoid common security pitfalls experienced by even large organizations. We’ve looked at the out of the box capability of CrossGuard by securing an S3 bucket and showed how best practices encoded in AWSGuard can warn of common configuration errors before deploying infrastructure. These examples provide a preview of how IaC can improve deployments by keeping your infrastructure secure.

As mentioned previously, CrossGuard is currently a technology preview. It will be GA with the release of Pulumi 2.0 with even more features, such as OPA integration, and more policy packs. We invite you to try it out and send us feedback either through Github issues or on the Pulumi Community Slack Channel.