Pulumi Blog: Redshift

How To Build An ETL Pipeline With Amazon Redshift & AWS Glue

Christian Nunciato — Fri, 23 Dec 2022 00:00:00 +0000

In our last episode, Deploying a Data Warehouse with Pulumi and Amazon Redshift, we covered using Pulumi to load unstructured data from Amazon S3 into an Amazon Redshift cluster. That went well, but you may recall that at the end of that post, we were left with a few unanswered questions:

How do we avoid importing and processing the same data twice?
How can we transform the data during the ingestion process?
What are our options for loading data automatically — for example, on a regular schedule?

These are the kinds of questions you’ll almost always have when setting up a data-processing (or ETL) pipeline — and every platform tends to answer them a little differently.

When your platform of choice is Amazon Redshift, those questions will often be answered by pointing you to another Amazon service (actually a collection of services) called AWS Glue. With Glue, you can define processes that monitor external data sources like S3, keep track of the data that’s already been processed (to avoid corruption from duplicate records, as we saw in the previous post), and write code in general-purpose programming languages like Python to process and transform the data on its way into Redshift.

There’s a lot more you can do with AWS Glue than what we need for this project, but as you’ll see, it’s also an excellent fit. So without further ado, let’s finish things off by using AWS Glue (and Pulumi of course) to set up a full ETL pipeline that loads all of our S3 data into Redshift automatically.

But first …

Set Up Redshift & Glue Integration

If you haven’t already, I’d encourage you to read through the previous post to get up to speed on what we’re building and why. In that post, the situation was such that some hypothetical application was generating “events” — little bits of JSON, essentially — and writing them periodically to a text file in S3, and as data scientists, we needed some way to gather up all of these events and load them into a Redshift cluster in order to analyze them later.

When we left off, we’d gotten Redshift up and running, and we were able to pull the data from S3 into Redshift directly (by running a manual query in the Redshift console), but that’s as far as we got — no automation, no protection from duplicate records, just the absolute basics.

So to pick up from there, let’s start by bootstrapping a new project with the code from the previous post. As always, make sure you’ve installed Pulumi and configured your AWS credentials in the usual way first, then run the following commands to get going:

$ mkdir my-data-warehouse-part2 && cd my-data-warehouse-part2
$ pulumi new aws-typescript

$ mkdir my-data-warehouse-part2 && cd my-data-warehouse-part2
$ pulumi new aws-python

Step through the prompts, choosing the AWS region that works best for you. When the new-project wizard completes, replace the contents of index.js index.ts __main__.py main.go Program.cs Program.fs Program.vb App.java Pulumi.yaml main.tf with the following code to fast-forward to where we left off:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

// Import the stack's configuration settings.
const config = new pulumi.Config();
const clusterIdentifier = config.require("clusterIdentifier");
const clusterNodeType = config.require("clusterNodeType");
const clusterDBName = config.require("clusterDBName");
const clusterDBUsername = config.require("clusterDBUsername");
const clusterDBPassword = config.requireSecret("clusterDBPassword");

// Import the provider's configuration settings.
const providerConfig = new pulumi.Config("aws");
const awsRegion = providerConfig.require("region");

// Create an S3 bucket to store some raw data.
const eventsBucket = new aws.s3.Bucket("events", {
 forceDestroy: true,
});

// Create a VPC.
const vpc = new aws.ec2.Vpc("vpc", {
 cidrBlock: "10.0.0.0/16",
 enableDnsHostnames: true,
});

// Create a private subnet within the VPC.
const subnet = new aws.ec2.Subnet("subnet", {
 vpcId: vpc.id,
 cidrBlock: "10.0.1.0/24",
});

// Declare a Redshift subnet group with the subnet ID.
const subnetGroup = new aws.redshift.SubnetGroup("subnet-group", {
 subnetIds: [
 subnet.id,
 ],
});

// Create an IAM role granting Redshift read-only access to S3.
const redshiftRole = new aws.iam.Role("redshift-role", {
 assumeRolePolicy: {
 Version: "2012-10-17",
 Statement: [
 {
 Action: "sts:AssumeRole",
 Effect: "Allow",
 Principal: {
 Service: "redshift.amazonaws.com",
 },
 },
 ],
 },
 managedPolicyArns: [
 aws.iam.ManagedPolicy.AmazonS3ReadOnlyAccess,
 ],
});

// Create a VPC endpoint so the cluster can read from S3 over the private network.
const vpcEndpoint = new aws.ec2.VpcEndpoint("s3-vpc-endpoint", {
 vpcId: vpc.id,
 serviceName: `com.amazonaws.${awsRegion}.s3`,
 routeTableIds: [
 vpc.mainRouteTableId,
 ],
});

// Create a single-node Redshift cluster in the VPC.
const cluster = new aws.redshift.Cluster("cluster", {
 clusterIdentifier: clusterIdentifier,
 databaseName: clusterDBName,
 masterUsername: clusterDBUsername,
 masterPassword: clusterDBPassword,
 nodeType: clusterNodeType,
 clusterSubnetGroupName: subnetGroup.name,
 clusterType: "single-node",
 publiclyAccessible: false,
 skipFinalSnapshot: true,
 vpcSecurityGroupIds: [
 vpc.defaultSecurityGroupId,
 ],
 iamRoles: [
 redshiftRole.arn,
 ],
});

import json
import pulumi
from pulumi_aws import ec2, iam, redshift, s3, glue

# Import the stack's configuration settings.
config = pulumi.Config()
cluster_identifier = config.require("clusterIdentifier")
cluster_node_type = config.require("clusterNodeType")
cluster_db_name = config.require("clusterDBName")
cluster_db_username = config.require("clusterDBUsername")
cluster_db_password = config.require_secret("clusterDBPassword")

# Import the provider's configuration settings.
provider_config = pulumi.Config("aws")
aws_region = provider_config.require("region")

# Create an S3 bucket to store some raw data.
events_bucket = s3.Bucket("events", s3.BucketArgs(
 force_destroy=True,
))

# Create a VPC.
vpc = ec2.Vpc("vpc", ec2.VpcArgs(
 cidr_block="10.0.0.0/16",
 enable_dns_hostnames=True,
))

# Create a private subnet within the VPC.
subnet = ec2.Subnet("subnet", ec2.SubnetArgs(
 vpc_id=vpc.id,
 cidr_block="10.0.1.0/24",
))

# Declare a Redshift subnet group with the subnet ID.
subnet_group = redshift.SubnetGroup("subnet-group", redshift.SubnetGroupArgs(
 subnet_ids=[
 subnet.id,
 ],
))

# Create an IAM role granting Redshift read-only access to S3.
redshift_role = iam.Role("redshift-role", iam.RoleArgs(
 assume_role_policy=json.dumps({
 "Version": "2012-10-17",
 "Statement": [
 {
 "Action": "sts:AssumeRole",
 "Effect": "Allow",
 "Principal": {
 "Service": "redshift.amazonaws.com",
 },
 },
 ],
 }),
 managed_policy_arns=[
 iam.ManagedPolicy.AMAZON_S3_READ_ONLY_ACCESS,
 ],
))

# Create a VPC endpoint so the cluster can read from S3 over the private network.
vpc_endpoint = ec2.VpcEndpoint("s3-vpc-endpoint", ec2.VpcEndpointArgs(
 vpc_id=vpc.id,
 service_name=f"com.amazonaws.{aws_region}.s3",
 route_table_ids=[
 vpc.main_route_table_id,
 ],
))

# Create a single-node Redshift cluster in the VPC.
cluster = redshift.Cluster("cluster", redshift.ClusterArgs(
 cluster_identifier=cluster_identifier,
 database_name=cluster_db_name,
 master_username=cluster_db_username,
 master_password=cluster_db_password,
 node_type=cluster_node_type,
 cluster_subnet_group_name=subnet_group.name,
 cluster_type="single-node",
 publicly_accessible=False,
 skip_final_snapshot=True,
 vpc_security_group_ids=[
 vpc.default_security_group_id,
 ],
 iam_roles=[
 redshift_role.arn,
 ],
))

At the top of the program, you’ll see that the code assumes you’ve set several Pulumi configuration values — clusterIdentifier, clusterNodeType, and others — so take a moment to do that now with the Pulumi CLI as well:

$ pulumi config set clusterIdentifier my-redshift-cluster
$ pulumi config set clusterNodeType ra3.xlplus
$ pulumi config set clusterDBName dev
$ pulumi config set clusterDBUsername admin
$ pulumi config set clusterDBPassword StrongPass1 --secret

That should be all you need to pick up where we left off. Deploy with an initial pulumi up to bring this project back to life:

$ pulumi up
...
Updating (dev)
Type Name Status
+ pulumi:pulumi:Stack my-data-warehouse-part2-dev created (5s)
+ ├─ aws:ec2:Vpc vpc created (13s)
+ ├─ aws:iam:Role redshift-role created (2s)
+ ├─ aws:s3:Bucket events created (3s)
+ ├─ aws:ec2:VpcEndpoint s3-vpc-endpoint created (6s)
+ ├─ aws:ec2:Subnet subnet created (1s)
+ ├─ aws:redshift:SubnetGroup subnet-group created (1s)
+ └─ aws:redshift:Cluster cluster created (105s)
Resources:
+ 8 created
Duration: 2m7s

When the deployment completes, you should see that eight new resources were created, including a VPC with a private subnet, a Redshift cluster deployed into the subnet, an S3 bucket that we’ll use to hold some raw data, and a VPC endpoint allowing direct access to the bucket over the private network.

Now it’s time to start adding the Glue components to — well, glue it all together.

Automating ETL Jobs in AWS Glue

At a high-level, we’ll need three components to complete our ETL pipeline-to-be:

A Glue crawler. The crawler is responsible for fetching data from some external source (for us, an S3 bucket) and importing it into a Glue catalog. A catalog is a kind of staging database that tracks your data through the ingestion process. Our particular crawler will poll our S3 bucket for new data and import that data into a catalog table.
A Glue job. The job is responsible for running an ETL script (e.g., on a schedule) to process the data imported by the crawler into the catalog.
A Glue script. The script is where the ETL magic happens. Ours will be written in Python and be responsible for extracting data from the catalog, applying some slight transformations to it (mostly just for demonstration purposes), and loading the transformed data into Redshift.

We’ll translate these high-level components into concrete Pulumi resources in the steps ahead.

Step 1: Add a Glue crawler

The first thing we’ll need is a unique name for our Glue catalog database. Rather than hard-code this name into our program code (which would be problematic for projects consisting of multiple stacks), let’s instead use Pulumi to set this catalog name stack-specifically:

$ pulumi config set glueDBName my-glue-db

Next, back in your editor, add the following code to read this new setting and declare the new AWS resources you’ll need to enable the crawler. The comments inline explain each one: there’s a glue.CatalogDatabase to house imported data, a glue.Crawler to poll our bucket every 15 minutes for new data, and an iam.Role granting the Glue service the appropriate set of privileges to read from S3 and write temporary data as needed:

// ...
const glueDBName = config.require("glueDBName");

// ...

// Define an AWS cron expression of "every 15 minutes".
// https://docs.aws.amazon.com/lambda/latest/dg/services-cloudwatchevents-expressions.html
const every15minutes= "cron(0/15 * * * ? *)";

// Create a Glue catalog database.
const glueCatalogDB = new aws.glue.CatalogDatabase("glue-catalog-db", {
 name: glueDBName,
});

// Define an IAM role granting AWS Glue access to S3 and other Glue-required services.
const glueRole = new aws.iam.Role("glue-role", {
 assumeRolePolicy: JSON.stringify({
 Version: "2012-10-17",
 Statement: [
 {
 Action: "sts:AssumeRole",
 Effect: "Allow",
 Principal: {
 Service: "glue.amazonaws.com",
 },
 },
 ],
 }),
 managedPolicyArns: [
 aws.iam.ManagedPolicy.AmazonS3FullAccess,
 "arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole",
 ],
});

// Create a Glue crawler to process the contents of the data bucket on a schedule.
// https://docs.aws.amazon.com/glue/latest/dg/monitor-data-warehouse-schedule.html
const glueCrawler = new aws.glue.Crawler("glue-crawler", {
 databaseName: glueCatalogDB.name,
 role: glueRole.arn,
 schedule: every15minutes,
 s3Targets: [
 {
 path: pulumi.interpolate`s3://${eventsBucket.bucket}`,
 }
 ],
});

# ...
glue_db_name = config.require("glueDBName")

# ...

# Define an AWS cron expression of "every 15 minutes".
# https://docs.aws.amazon.com/lambda/latest/dg/services-cloudwatchevents-expressions.html
every_15_minutes = "cron(0/15 * * * ? *)"

# Create a Glue catalog database.
glue_catalog_db = glue.CatalogDatabase("glue-catalog-db", glue.CatalogDatabaseArgs(
 name=glue_db_name,
))

# Define an IAM role granting AWS Glue access to S3 and other Glue-required services.
glue_role = iam.Role("glue-role", iam.RoleArgs(
 assume_role_policy=json.dumps({
 "Version": "2012-10-17",
 "Statement": [
 {
 "Action": "sts:AssumeRole",
 "Effect": "Allow",
 "Principal": {
 "Service": "glue.amazonaws.com",
 },
 },
 ],
 }),
 managed_policy_arns=[
 iam.ManagedPolicy.AMAZON_S3_FULL_ACCESS,
 "arn:aws:iam::aws:policy/service-role/AWSGlueServiceRole",
 ],
))

# Create a Glue crawler to process the contents of the data bucket on a schedule.
# https://docs.aws.amazon.com/glue/latest/dg/monitor-data-warehouse-schedule.html
glue_crawler = glue.Crawler("glue-crawler", glue.CrawlerArgs(
 database_name=glue_catalog_db.name,
 role=glue_role.arn,
 schedule=every_15_minutes,
 s3_targets=[
 glue.CrawlerS3TargetArgs(
 path=events_bucket.bucket.apply(lambda name: f"s3://{name}")
 ),
 ],
))

Run a quick pulumi up to deploy these new additions:

$ pulumi up
...

Updating (dev)

 Type Name Status
 pulumi:pulumi:Stack my-data-warehouse-part2-dev
 + ├─ aws:iam:Role glue-role created (2s)
 + ├─ aws:glue:CatalogDatabase glue-catalog-db created (1s)
 + └─ aws:glue:Crawler glue-crawler created (10s)

Resources:
 + 3 created
 8 unchanged

Duration: 15s

Now let’s add the Glue job.

Step 2: Add a Glue job

The job we need is a fairly simple one, but it still requires a handful of AWS resources.

First, we’ll need a glue.Connection to allow Glue to connect to the Redshift database to load the data processed by our TBD Python script. We’ll also need the script itself, which Glue expects to be able to find in an S3 bucket, so we’ll need to define a Bucket and BucketObject for that as well. We’ll need a glue.Job resource to define the compute settings and other parameters with which to run our Python script (passing it various arguments such as which Glue catalog to read from and Redshift database to write to), and then finally, we’ll need a glue.Trigger to kick off the process every 15 minutes. It’s only a few resources, but there’s a fair bit of configuration baked into some of them, so I’d encourage you to spend a little time with the AWS Glue docs to gain a better understanding of what each option does and what else can be done with them.

Believe it or not, that’s all we’ll need for the Pulumi program itself. So to wrap up, add the following code to index.js index.ts __main__.py main.go Program.cs Program.fs Program.vb App.java Pulumi.yaml main.tf , taking note of the last line, which exports the name of the S3 bucket to give us something to upload to:

// ...

// Create a Glue connection to the Redshift cluster.
const glueRedshiftConnection = new aws.glue.Connection("glue-redshift-connection", {
 connectionType: "JDBC",
 connectionProperties: {
 JDBC_CONNECTION_URL: pulumi.interpolate`jdbc:redshift://${cluster.endpoint}/${clusterDBName}`,
 USERNAME: clusterDBUsername,
 PASSWORD: clusterDBPassword,
 },
 physicalConnectionRequirements: {
 securityGroupIdLists: cluster.vpcSecurityGroupIds,
 availabilityZone: subnet.availabilityZone,
 subnetId: subnet.id,
 },
});

// Create an S3 bucket for Glue scripts and temporary storage.
const glueJobBucket = new aws.s3.Bucket("glue-job-bucket", {
 forceDestroy: true,
});

// Upload a Glue job script.
const glueJobScript = new aws.s3.BucketObject("glue-job.py", {
 bucket: glueJobBucket.id,
 source: new pulumi.asset.FileAsset("./glue-job.py"),
});

// Create a Glue job that runs our Python ETL script.
const glueJob = new aws.glue.Job("glue-job", {
 roleArn: glueRole.arn,
 glueVersion: "3.0",
 numberOfWorkers: 10,
 workerType: "G.1X",
 defaultArguments: {
 // Enabling job bookmarks helps you avoid loading duplicate data.
 // https://docs.aws.amazon.com/glue/latest/dg/monitor-continuations.html
 "--job-bookmark-option": "job-bookmark-enable",

 "--ConnectionName": glueRedshiftConnection.name,
 "--GlueDBName": glueDBName,
 "--GlueDBTableName": eventsBucket.bucket.apply(name => name.replace('-', '_')),
 "--RedshiftDBName": clusterDBName,
 "--RedshiftDBTableName": "events",
 "--RedshiftRoleARN": redshiftRole.arn,
 "--TempDir": pulumi.interpolate`s3://${glueJobBucket.bucket}/glue-job-temp`,
 },
 connections: [
 glueRedshiftConnection.name,
 ],
 command: {
 scriptLocation: pulumi.interpolate`s3://${glueJobBucket.bucket}/glue-job.py`,
 pythonVersion: "3",
 },
});

// Create a Glue trigger to run the job every 15 minutes.
const glueJobTrigger = new aws.glue.Trigger("trigger", {
 schedule: every15minutes,
 type: "SCHEDULED",
 actions: [{
 jobName: glueJob.name,
 }],
});

// Export the name of the data bucket.
export const dataBucketName = eventsBucket.bucket;

# ...

# Create a Glue connection to the Redshift cluster.
glue_redshift_connection = glue.Connection("glue-redshift-connection", glue.ConnectionArgs(
 connection_type="JDBC",
 connection_properties={
 "JDBC_CONNECTION_URL": cluster.endpoint.apply(lambda endpoint: f"jdbc:redshift://{endpoint}/{cluster_db_name}"),
 "USERNAME": cluster_db_username,
 "PASSWORD": cluster_db_password,
 },
 physical_connection_requirements=glue.ConnectionPhysicalConnectionRequirementsArgs(
 security_group_id_lists=cluster.vpc_security_group_ids,
 availability_zone=subnet.availability_zone,
 subnet_id=subnet.id,
 ),
))

# Create an S3 bucket for Glue scripts and temporary storage.
glue_job_bucket = s3.Bucket("glue-job-bucket", s3.BucketArgs(
 force_destroy=True,
))

# Upload a Glue job script.
glue_job_script = s3.BucketObject("glue-job.py", s3.BucketObjectArgs(
 bucket=glue_job_bucket.id,
 source=pulumi.asset.FileAsset("./glue-job.py"),
))

# Create a Glue job that runs our Python ETL script.
glue_job = glue.Job("glue-job", glue.JobArgs(
 role_arn=glue_role.arn,
 glue_version="3.0",
 connections=[
 glue_redshift_connection.name,
 ],
 number_of_workers=10,
 worker_type="G.1X",
 default_arguments={
 # Enabling job bookmarks helps you avoid loading duplicate data.
 # https://docs.aws.amazon.com/glue/latest/dg/monitor-continuations.html
 "--job-bookmark-option": "job-bookmark-enable",

 "--ConnectionName": glue_redshift_connection.name,
 "--GlueDBName": glue_db_name,
 "--GlueDBTableName": events_bucket.bucket.apply(lambda name: name.replace('-', '_')),
 "--RedshiftDBName": cluster_db_name,
 "--RedshiftDBTableName": "events",
 "--RedshiftRoleARN": redshift_role.arn,
 "--TempDir": glue_job_bucket.bucket.apply(lambda name: f"s3://{name}/glue-job-temp"),
 },
 command=glue.JobCommandArgs(
 script_location=glue_job_bucket.bucket.apply(lambda name: f"s3://{name}/glue-job.py"),
 python_version="3",
 ),
))

# Create a Glue trigger to run the job every 15 minutes.
glue_job_trigger = glue.Trigger("trigger", glue.TriggerArgs(
 schedule=every_15_minutes,
 type="SCHEDULED",
 actions=[
 glue.TriggerActionArgs(
 job_name=glue_job.name,
 ),
 ],
))

# Export the name of the data bucket.
pulumi.export("dataBucketName", events_bucket.bucket)

We’re not quite ready to deploy yet though — we need a script!

Step 3: Add a Glue script

To finish things off, create a new file called glue-job.py (alongside index.js index.ts __main__.py main.go Program.cs Program.fs Program.vb App.java Pulumi.yaml main.tf ) and paste in the following code. This is the file that’ll be run by the glue.Job we just declared. There’s a lot going on in this script, but the gist is that when it runs, it’ll extract all unprocessed data from our Glue catalog, transform that data using a simple mapping to tweak some field names (again, just to demonstrate how that’s done), and load the resulting data into Redshift. A Glue job bookmark guarantees that we don’t process the same data more than once:

import sys
from awsglue.utils import getResolvedOptions
from awsglue.transforms import ApplyMapping
from awsglue.context import GlueContext
from awsglue.job import Job
from pyspark.context import SparkContext

# Collect the arguments passed in by the glue.Job run.
args = getResolvedOptions(
 sys.argv,
 [
 "JOB_NAME",
 "TempDir",
 "ConnectionName",
 "GlueDBName",
 "GlueDBTableName",
 "RedshiftRoleARN",
 "RedshiftDBName",
 "RedshiftDBTableName",
 ],
)

glueContext = GlueContext(SparkContext.getOrCreate())

job = Job(glueContext)
job.init(args["JOB_NAME"], args)

# Extract all unprocessed data from the Glue catalog.
source0 = glueContext.create_dynamic_frame.from_catalog(
 database=args["GlueDBName"],
 table_name=args["GlueDBTableName"],
 additional_options={
 "jobBookmarkKeys": ["id"],
 "jobBookmarkKeysSortOrder": "asc",
 },
 transformation_ctx="source0",
)

# Transform the data (mostly just to show how to do so).
transformed0 = ApplyMapping.apply(
 frame=source0,
 mappings=[
 ("id", "int", "event_id", "int"),
 ("name", "string", "event_name", "string"),
 ],
)

# Load the data into the Redshift database.
glueContext.write_dynamic_frame.from_jdbc_conf(
 frame=transformed0,
 catalog_connection=args["ConnectionName"],
 connection_options={
 "database": args["RedshiftDBName"],
 "dbtable": args["RedshiftDBTableName"],
 "aws_iam_role": args["RedshiftRoleARN"],
 },
 redshift_tmp_dir=args["TempDir"],
)

# Call commit() to reset the job bookmark for the next run.
job.commit()

And that’s it — you’re ready to deploy.

Deploy!

With everything now in place — VPC, private network, Redshift cluster, and the Glue componentry to handle all of the automation (crawler, job, Python ETL script), you’re ready to deploy. Do that now with a final pulumi up:

$ pulumi up
...

Updating (dev)

 Type Name Status
 pulumi:pulumi:Stack my-data-warehouse-part2-dev
 + ├─ aws:s3:Bucket glue-job-bucket created (2s)
 + ├─ aws:glue:Connection glue-redshift-connection created (1s)
 + ├─ aws:glue:Job glue-job created (0.95s)
 + ├─ aws:s3:BucketObject glue-job.py created (0.63s)
 + └─ aws:glue:Trigger trigger created (0.89s)


Outputs:
 + dataBucketName: "events-e713608"

Resources:
 + 5 created
 11 unchanged

Duration: 7s

You should see that the Glue components were created. Now let’s load some sample data.

Load Sample Data into AWS Glue

As before, we’ll simulate this part, since we don’t have an actual application generating real data for us. Run the following command to write a few JSON records to a file called events-1.txt:

$ echo '{"id": 1, "name": "An interesting event"}
{"id": 2, "name": "Another interesting event"}
{"id": 3, "name": "An event of monumental importance"}' > events-1.txt

With the AWS CLI, upload the file into the S3 data bucket using the name we exported at the end of the last deployment:

$ aws s3 cp events-1.txt s3://$(pulumi stack output dataBucketName)

And now … we wait.

Grab a coffee, take a walk, watch an episode of your favorite sitcom, whatever you like — just be gone for a good half-hour or so to give the crawler and job enough time to run so that when you get back, you’ll have something to look at. (Of course, if you’d rather not wait, you can always just log into the AWS console and kick off the crawler and job manually — but it’s definitely more fun to let the glue.Crawler a glue.Trigger do that for you.)

Soon enough, you should be able to verify, by navigating to Crawlers in the AWS Glue console, that the crawler indeed ran, creating a new table in the Glue catalog database:

As well, you should see (at AWS Glue Studio > Jobs) that the job ran successfully as well:

A successful job run, of course, means that our data must’ve made it into Redshift, too — and you can verify by running a query that in the Redshift query editor:

And with that, your work is done. You’re now running a fully-automated, Pulumi-managed ETL pipeline with Amazon Redshift and AWS Glue.

If you like, you can create a second data file — call it events-2.txt:

$ echo '{"id": 4, "name": "A fourth thing"}
{"id": 5, "name": "The fifth and final thing"}' > events-2.txt

upload that file to S3 as well:

$ aws s3 cp events-2.txt s3://$(pulumi stack output dataBucketName)

and sit back while the pipeline ingests this new data, blissfully ignoring the file that’s already been processed (thanks to the Glue job bookmark) and dropping the new records right into Redshift as you’d expect:

Clean Up AWS Resources with Pulumi

When you’re finished experimenting, be sure to tear down the Redshift cluster and all other resources with a single pulumi destroy:

$ pulumi destroy
...

Destroying (dev)

 Type Name Status
 - pulumi:pulumi:Stack my-data-warehouse-part2-dev deleted
 - ├─ aws:glue:Trigger trigger deleted (36s)
 - ├─ aws:glue:Job glue-job deleted (0.32s)
 - ├─ aws:glue:Connection glue-redshift-connection deleted (0.44s)
 - ├─ aws:redshift:Cluster cluster deleted (229s)
 - ├─ aws:redshift:SubnetGroup subnet-group deleted (0.33s)
 - ├─ aws:ec2:VpcEndpoint s3-vpc-endpoint deleted (6s)
 - ├─ aws:s3:BucketObject glue-job.py deleted (0.88s)
 - ├─ aws:ec2:Subnet subnet deleted (1s)
 - ├─ aws:glue:Crawler glue-crawler deleted (1s)
 - ├─ aws:s3:Bucket events deleted (1s)
 - ├─ aws:iam:Role glue-role deleted (1s)
 - ├─ aws:s3:Bucket glue-job-bucket deleted (2s)
 - ├─ aws:ec2:Vpc vpc deleted (1s)
 - ├─ aws:glue:CatalogDatabase glue-catalog-db deleted (1s)
 - └─ aws:iam:Role redshift-role deleted (1s)


Outputs:
 - dataBucketName: "events-e713608"

Resources:
 - 16 deleted

Watch the AWS Glue ETL Pipeline in Action

If you’d like to see a demonstration of what you’ve read in this blog post, then this Modern Infrastructure video covers the whole process:

What’s next?

I hope you learned as much as I did with this post — we covered a lot, and we’ve only scratched the surface of what’s possible with these tools. Once again, I’d encourage you to spend some time digging into the Redshift and Glue documentation to explore both in more depth, as there’s lots more to learn than what we’ve covered so far.

Once you’re up and running with Redshift, you might also want to check out the Metabase package in the Pulumi Registry. The Metabase package makes it easy to deploy Metabase alongside Redshift on AWS, allowing you to build powerful data-analysis and visualization experiences with your data.

You’ll find the full source of this walkthrough (in TypeScript and Python) on GitHub as well.

Happy data-warehousing!

Deploying a Data Warehouse with Pulumi and Amazon Redshift

Christian Nunciato — Tue, 29 Nov 2022 00:00:00 +0000

It’s fun to think about how much data there is swirling around in the global datasphere these days. However you choose to measure it (and there are various ways), it’s a quantity so massive — hundreds of zettabytes, by some estimates — that it’s kind of a hard thing to quite get your head around.

If you could convert all the world’s data into droplets of water, for instance, at one megabyte per drop, you’d have enough 1MB drops to fill two more Lake Washingtons. If you could store all that data on 3.5" floppies, you’d need more than a hundred quadrillion floppies to capture it all — enough to cover the planet entirely (with much room for overlap) or to pave a nice bridge for yourself from your front porch well into interstellar space. If you could pull all that data into an HD movie, and you sat down to start watching that movie 2.5 million years ago (with your favorite saber-toothed friend, say), you’d still be watching the same movie today.

That’s a lot of data — and there’s more every day.

Of course, the volume of data you care about is probably a lot more modest. But even so, there’s a good chance whatever system you’re building can and will produce useful data of its own, and there’s an equally good chance you’ll want to learn something from it. That data will likely exist in many forms and in multiple places — flat file here, transactional database there, the REST API of some third-party service — so before you can really look at it, you’ll need to figure out how to get it all into one place. And for that, you might want to consider a data warehouse.

A data warehouse is a specialized database that’s purpose built for gathering and analyzing data. Unlike general-purpose databases like MySQL or PostgreSQL, which are designed to meet the real-time performance and transactional needs of applications, a data warehouse is designed to collect and process the data produced by those applications, collectively and over time, to help you gain insight from it. Examples of data-warehouse products include Snowflake, Google BigQuery, Azure Synapse Analytics, and Amazon Redshift — all of which, incidentally, are easily managed with Pulumi.

Today, though, we’re going to focus on Amazon Redshift. Specifically, we’re going to walk through the process of writing a Pulumi program that provisions a single-node Redshift cluster in an Amazon VPC, then we’ll load some sample data into the warehouse from Amazon S3. We’ll load this data manually at first, just to get a sense of how everything works when it’s all wired up, and then later, in a follow-up post, we’ll go a step further and weave in some automation to load the data on a schedule.

Let’s get going!

Create a new project

We’ll begin, as always, by creating a new Pulumi project. (And if you haven’t already, make sure you’ve installed Pulumi and configured your AWS credentials in the usual way.)

$ mkdir my-data-warehouse && cd my-data-warehouse
$ pulumi new aws-typescript

$ mkdir my-data-warehouse && cd my-data-warehouse
$ pulumi new aws-python

Step through the new-project wizard, accepting the defaults if they’ll work for you. For this project, we’ll deploy both cluster and data into the us-west-2 region of AWS, but you can change the target to region you like, as both S3 and Redshift are supported in all AWS regions.

With the project and stack in place, it’s time to start building.

Configure the stack

To launch a new Redshift cluster, you’ll need to give AWS a few details:

A unique (in the scope of your AWS account) name for the Redshift cluster
A name for the cluster’s initial database
A username and password for the cluster’s administrative user
The node type to use for the Redshift cluster

The following configuration settings should work for this walkthrough (but again, feel free to adjust them if you’d like):

$ pulumi config set clusterIdentifier my-redshift-cluster
$ pulumi config set clusterNodeType ra3.xlplus
$ pulumi config set clusterDBName dev
$ pulumi config set clusterDBUsername admin
$ pulumi config set clusterDBPassword StrongPass1 --secret

Compose the program

With these settings in place, you can start writing the Pulumi program.

Open index.js index.ts __main__.py main.go Program.cs Program.fs Program.vb App.java Pulumi.yaml main.tf in your editor of choice and replace its contents with the following code, which imports the Pulumi and AWS libraries, reads the configuration values you just set, and provisions an S3 bucket that we’ll use to store some raw, unstructured data later:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

// Import the stack's configuration settings.
const config = new pulumi.Config();
const clusterIdentifier = config.require("clusterIdentifier");
const clusterNodeType = config.require("clusterNodeType");
const clusterDBName = config.require("clusterDBName");
const clusterDBUsername = config.require("clusterDBUsername");
const clusterDBPassword = config.requireSecret("clusterDBPassword");

// Import the provider's configuration settings.
const providerConfig = new pulumi.Config("aws");
const awsRegion = providerConfig.require("region");

// Create an S3 bucket to store some raw data.
const eventsBucket = new aws.s3.Bucket("events", {
 forceDestroy: true,
});

import json
import pulumi
from pulumi_aws import ec2, iam, redshift, s3

# Import the stack's configuration settings.
config = pulumi.Config()
cluster_identifier = config.require("clusterIdentifier")
cluster_node_type = config.require("clusterNodeType")
cluster_db_name = config.require("clusterDBName")
cluster_db_username = config.require("clusterDBUsername")
cluster_db_password = config.require_secret("clusterDBPassword")

# Import the provider's configuration settings.
provider_config = pulumi.Config("aws")
aws_region = provider_config.require("region")

# Create an S3 bucket to store some raw data.
events_bucket = s3.Bucket("events", s3.BucketArgs(
 force_destroy=True,
))

Next, define a new VPC and the associated network resources for the Redshift cluster. Since the aim is to launch the cluster into a VPC (providing the network isolation we need to keep the cluster from being accessed over the internet), you’ll define the VPC first, then define a private subnet within it, and then finally designate a Redshift subnet group to tell AWS where to provision the cluster:

// ...

// Create a VPC.
const vpc = new aws.ec2.Vpc("vpc", {
 cidrBlock: "10.0.0.0/16",
 enableDnsHostnames: true,
});

// Create a private subnet within the VPC.
const subnet = new aws.ec2.Subnet("subnet", {
 vpcId: vpc.id,
 cidrBlock: "10.0.1.0/24",
});

// Declare a Redshift subnet group with the subnet ID.
const subnetGroup = new aws.redshift.SubnetGroup("subnet-group", {
 subnetIds: [
 subnet.id,
 ],
});

# ...

# Create a VPC.
vpc = ec2.Vpc("vpc", ec2.VpcArgs(
 cidr_block="10.0.0.0/16",
 enable_dns_hostnames=True,
))

# Create a private subnet within the VPC.
subnet = ec2.Subnet("subnet", ec2.SubnetArgs(
 vpc_id=vpc.id,
 cidr_block="10.0.1.0/24",
))

# Declare a Redshift subnet group with the subnet ID.
subnet_group = redshift.SubnetGroup("subnet-group", redshift.SubnetGroupArgs(
 subnet_ids=[
 subnet.id,
 ],
))

Now, on to the cluster itself.

The plan, you’ll recall, is to pull data from an S3 bucket into Redshift (using the Redshift service itself, for now), so you’ll need to give Redshift the appropriate permissions to read from Amazon S3. You can do that with an IAM role and AWS-managed policy granting Redshift read-only access to the S3 service:

// ...

// Create an IAM role granting Redshift read-only access to S3.
const redshiftRole = new aws.iam.Role("redshift-role", {
 assumeRolePolicy: {
 Version: "2012-10-17",
 Statement: [
 {
 Action: "sts:AssumeRole",
 Effect: "Allow",
 Principal: {
 Service: "redshift.amazonaws.com",
 },
 },
 ],
 },
 managedPolicyArns: [
 aws.iam.ManagedPolicy.AmazonS3ReadOnlyAccess,
 ],
});

# ...

# Create an IAM role granting Redshift read-only access to S3.
redshift_role = iam.Role("redshift-role", iam.RoleArgs(
 assume_role_policy=json.dumps({
 "Version": "2012-10-17",
 "Statement": [
 {
 "Action": "sts:AssumeRole",
 "Effect": "Allow",
 "Principal": {
 "Service": "redshift.amazonaws.com",
 },
 },
 ],
 }),
 managed_policy_arns=[
 iam.ManagedPolicy.AMAZON_S3_READ_ONLY_ACCESS,
 ],
))

That takes care of the permissions part — but there’s one last thing you’ll need to do to make this work. Because the cluster will reside in a private subnet, and that subnet won’t have access to the public internet, you’ll need to give the cluster a way to communicate with S3 without having to leave the VPC network. You can do this by adding a VPC gateway endpoint:

// ...

// Create a VPC endpoint so the cluster can read from S3 over the private network.
const vpcEndpoint = new aws.ec2.VpcEndpoint("s3-vpc-endpoint", {
 vpcId: vpc.id,
 serviceName: `com.amazonaws.${awsRegion}.s3`,
});

# ...

# Create a VPC endpoint so the cluster can read from S3 over the private network.
vpc_endpoint = ec2.VpcEndpoint("s3-vpc-endpoint", ec2.VpcEndpointArgs(
 vpc_id=vpc.id,
 service_name=f"com.amazonaws.{aws_region}.s3",
))

Finally, add the cluster itself, using everything you’ve defined up to now — config settings, network settings, IAM role, and the VPC’s default security group, which enables traffic to and from the cluster on all ports and subnet IPs. Finish things off by exporting the name of the S3 bucket and Redshift ARN, as you’ll need both in order to post and then import the S3 data (both will be auto-named):

// ...

// Create a single-node Redshift cluster in the VPC.
const cluster = new aws.redshift.Cluster("cluster", {
 clusterIdentifier: clusterIdentifier,
 databaseName: clusterDBName,
 masterUsername: clusterDBUsername,
 masterPassword: clusterDBPassword,
 nodeType: clusterNodeType,
 clusterSubnetGroupName: subnetGroup.name,
 clusterType: "single-node",
 publiclyAccessible: false,
 skipFinalSnapshot: true,
 vpcSecurityGroupIds: [
 vpc.defaultSecurityGroupId,
 ],
 iamRoles: [
 redshiftRole.arn,
 ],
});

// Export the bucket name and role ARN.
export const dataBucketName = eventsBucket.bucket;
export const redshiftRoleArn = redshiftRole.arn;

# ...

# Create a single-node Redshift cluster in the VPC.
cluster = redshift.Cluster("cluster", redshift.ClusterArgs(
 cluster_identifier=cluster_identifier,
 database_name=cluster_db_name,
 master_username=cluster_db_username,
 master_password=cluster_db_password,
 node_type=cluster_node_type,
 cluster_subnet_group_name=subnet_group.name,
 cluster_type="single-node",
 publicly_accessible=False,
 skip_final_snapshot=True,
 vpc_security_group_ids=[
 vpc.default_security_group_id,
 ],
 iam_roles=[
 redshift_role.arn,
 ],
))

# Export the bucket name and role ARN.
pulumi.export("dataBucketName", events_bucket.bucket)
pulumi.export("redshiftRoleArn", redshift_role.arn)

And with that, you’re ready to deploy.

Deploy the stack

Back at the command line, deploy the stack in the usual way with pulumi up:

$ pulumi up
...
Updating (dev)

 Type Name Status
 + pulumi:pulumi:Stack my-data-warehouse-dev created (3s)
 + ├─ aws:s3:Bucket events created (2s)
 + ├─ aws:ec2:Vpc vpc created (12s)
 + ├─ aws:iam:Role redshift-role created (1s)
 + ├─ aws:ec2:Subnet subnet created (0.81s)
 + ├─ aws:ec2:VpcEndpoint s3-vpc-endpoint created (6s)
 + ├─ aws:redshift:SubnetGroup subnet-group created (0.80s)
 + └─ aws:redshift:Cluster cluster created (270s)

Outputs:
 dataBucketName : "events-50d8c96"
 redshiftRoleArn: "arn:aws:iam::616138583583:role/redshift-role-47ad8f0"

Resources:
 + 8 created

Duration: 4m48s

After a few moments (deploying a new cluster takes about five minutes), you should see that the new cluster and all of its resources were created. Again, you won’t be able to access the cluster over the internet, as it’ll have been deployed into an isolated VPC network, but you should be able to browse to the cluster in the AWS Console and start running some queries with it.

Choose Query data > Query in query editor v2 to open the Redshift query editor:

Then, from the query editor, find your newly created cluster in the left-hand pane and connect using the database, username, and password you configured earlier:

Choose Create connection, expand my-redshift-cluster, and you should see the dev database there, ready to go (only without any tables — we’ll get to that next):

Now let’s load some data!

Getting data into Redshift

In a real-world situation, you’d probably already have some data to load — web server logs, for example, or some other sort of raw or unstructured data residing in an S3 bucket, a DynamoDB table, an RDS database, etc. But we’re starting from scratch, so we’ll create and publish that data manually.

Copy and paste the following command to generate a text file containing a few lines of JSON, each representing a fictitious event to be loaded into the data warehouse:

$ echo '{"id": 1, "name": "An interesting event"}
{"id": 2, "name": "Another interesting event"}
{"id": 3, "name": "An event of monumental importance"}' > events-1.txt

Notice there aren’t any commas between these events, so it’s not actually a valid JSON file – it’s just a text file containing some individually valid JSON events, one event per line. Adhering to this format allows Redshift to ingest and map the data to table we create next using one of its built-in parsers.

Upload the file to S3 with the AWS CLI, using Pulumi to supply the name of the destination bucket:

$ aws s3 cp events-1.txt "s3://$(pulumi stack output dataBucketName)"

At this point, you have all you need in terms of infrastructure: You’ve got a single-node Redshift cluster, an empty Redshift database, some unstructured S3 data, and the right set of permissions and network settings to allow the cluster to reach and import the data. All that’s left is to run a couple of SQL queries to make that happen: one to create a new table to hold the data, another to load it.

Back in the Redshift query editor, create the new table by choosing the dev database, pasting the following query into the editor, and hitting Run:

CREATE TABLE events (
 id INT,
 name VARCHAR(255)
);

You should see that the query succeeded and that the table was created:

Now you’re ready to import the data. Replace the contents of the query editor with the following Redshift COPY command, replacing the {bucket-name}, {iam-role}, and {aws-region} placeholders with the corresponding values from your Pulumi stack:

COPY events
FROM 's3://{bucket-name}/events-1.txt'
IAM_ROLE '{iam-role}'
FORMAT AS JSON 'auto'
REGION AS '{aws-region}';

Here’s what I see, for example, after running that command:

After the COPY command completes (which may take a few seconds), you should be able to run a new query the events table and see the new data:

SELECT *
 FROM events
 ORDER BY id;

Huzzah! You’ve now got yourself a working data warehouse.

See it in action

To demonstrate the entire process of building a data warehouse with Redshift and Pulumi, we created this Modern Infrastructure video that shows all of this in action:

Taking stock, and looking ahead

If you’re like me, you’re probably happy to be up and running — but you’ve also got a few questions:

What would happen if you added a few more rows to events-1.txt, then ran that COPY command a second time? Would Redshift ignore the records it’s already processed, and only import the new ones?
Running SQL commands in a browser like this seems less than ideal. Can I run this process automatically somehow — like once a day, or in response to some other event?
My source data’s kind of a mess. Can I process the data somehow when I load it — like drop certain fields, rename columns, split records into multiple tables, and so on?
Can I pull data from other sources that aren’t Amazon S3?

All good questions, and I’m glad that you asked them.

First, as you’ll notice, adding lines to events.txt and running COPY a second time probably doesn’t behave as you might hope:

Duplicate records — the bane of the data scientist.

As of now, your only real recourse is to leave events-1.txt alone, create new data files for each batch of events, and only COPY those files that haven’t been processed already. For small projects that don’t generate much data, this might be a viable option, but over time, it’d probably become pretty tedious, and it’d be all too easy to make a mistake that leaves you with messed-up data. Not ideal.

Instead, what you’ll ultimately want is some process by which your data can be extracted (duplicate-free and from multiple sources), transformed according to rules you specify (keeping what you want, dropping what you don’t), and loaded into the warehouse without your involvement. What you’ll want, in other words, is an ETL pipeline.

The good news is that with the data warehouse up and running, you’re well on your way already. The bad news is that Redshift alone won’t finish the job for you. Redshift is just the destination — the target of your ETL pipeline-to-be. The pipeline itself still needs to be written.

So in the next post, we’ll do that: We’ll take what we’ve done here, add a few more components with Pulumi and AWS Glue, and wire it all up with a few magical lines of Python scripting.

Until then, happy coding — and stay tuned!

Using AWS Quick Starts with the Pulumi Registry

Aidan Hoolachan — Wed, 05 Jan 2022 00:00:00 +0000

As somebody who works on AWS projects across numerous projects, teams, and industries; I see the following three common types of infrastructure problems. I think the Pulumi Registry provides an incredible solution to each of these problems and will fundamentally change how people interact with AWS.

Common AWS infrastructure problems:

AWS Console Wizard Fever - These companies use the AWS Console to deploy resources. They move very quickly initially but eventually their AWS accounts become cluttered with untracked resources and they need to do some heavy lifting to set up robust dev / staging / prod environments. Many AWS developers scoff at the idea of using the Console, but I actually think this is currently an okay approach for a lot of people / companies in the early stages of AWS development. The console provides a lot of assistance when doing deployments and most people don’t need very complicated infrastructure initially. It’s not ideal because of the tech debt but it’s the right choice for many situations (at least, it was the right choice before the Pulumi registry).
YAML/JSON CloudFormation Template Ceiling - These companies have the right idea to use Infrastructure as Code, but CloudFormation YAML/JSON templates or other limited frameworks like Amplify are unwieldy and have their own scaling problems. Most people who work closely with AWS advocate for Infrastructure as Code (I do too), but it really is a steep learning curve and can be a lot of overhead for many simple applications. Setting up email alarms when a database goes down requires specialized AWS knowledge to do correctly – which makes a basic feature like email/slack alarm notifications feel like a sophisticated luxury. It shouldn’t be complicated, but it is.
AWS Lock-in - These are companies who lean into the AWS ecosystem for convenience sake. They might choose Cognito over a third-party Identity Provider even though the third-party Identity Provider might be a superior service.

The underlying cause of these problems is that AWS’s core services are designed for modularity within the AWS ecosystem, which is great for flexibility and experienced AWS developers, but it makes building best-practice AWS environments surprisingly difficult. Most “best practice” features on AWS, like email alarms when a database goes down, should really be the standard practice, but those features currently require complicated cross-service relationships that are difficult to implement correctly. The AWS ecosystem has been waiting for a layer of abstraction on top of these resources.

Pulumi and the AWS Quick Starts in the Pulumi Registry use higher-order components to abstract away that underlying complexity which makes configuring, deploying, and managing critical AWS resources straightforward – like it should be.

Configuring an Aurora Postgresql database with best practices (again, what should really just be the standard practices) requires specialized AWS knowledge and provisioning a zoo of other critical resources:

Encryption (2 resources) - KMS Key + KMS Key Policy
Access logging (4 resources) - S3 + S3 Bucket Policy + Encryption (KMS, again) + IAM Role
Alarms with notifications (3 Resources) - Cloudwatch Events + SNS + SNS Subscription
Networking infrastructure - (4+ resources) VPCs + Subnets + Gateways, etc.
Username and password - (1 resource) Secrets Manager Secret
Redundancy - (multiple resources) RDS multi-AZ clusters
IAM - (too many resources) Various IAM roles and policies

If “Aurora Postgres RDS” were a third-party SaaS or PaaS service provider, the service would abstract away the above details and provide simple parameters for enabling encryption, alarms, notifications, network security, redundancy, monitoring, and logging. With Pulumi, that abstraction is possible while providing open-source access to the underlying AWS resource definitions and with much more flexibility than traditional CloudFormation or Terraform. Those features can even be enabled by default. I believe that AWS “best practices” will also become the easiest approach with Pulumi as the ecosystem matures around the best packages.

When writing the Pulumi providers for AWS Quick Start VPC, AWS Quick Start Aurora Postgres, and AWS Quick Start Redshift, we were able to fight through the complexity and implement the best practices listed above. Those practices are enabled by default but can be disabled using a flag (eg enableEncryption: false). The result is a simple interface for deploying a VPC, an Aurora Postgresql cluster, and a Redshift cluster along with all of the other resources needed for best practice deployments. The process for writing these AWS providers was straightforward and intuitive. The Pulumi deployment and debug process was pleasant and I was able to go through most of the deploy + debug + tear down lifecycle without leaving the terminal.

The following code deploys a VPC, an Aurora Postgresql Cluster, a Redshift Cluster, and the various helper resources needed for encryption, logging, database alarms, vpc flow logs, etc. – for a total of 47 resources deployed using only three functions with simple parameters.

import * as pulumi from "@pulumi/pulumi";
import * as vpcQuickstart from "@pulumi/aws-quickstart-vpc";
import * as redshiftQuickstart from "@pulumi/aws-quickstart-redshift";
import * as auroraQuickstart from "@pulumi/aws-quickstart-aurora-postgres";

const notificationEmail = process.env.notificationEmail;
const dbPasswordSecret = pulumi.secret(process.env.databasePassword);

// Create a best-practices VPC
// Deploys 21 resources
const multiAvailabilityZoneVpc = new vpcQuickstart.Vpc("example-aurora-vpc", {
 cidrBlock: "10.0.0.0/16",
 availabilityZoneConfig: [{
 availabilityZone: "us-east-1a",
 publicSubnetCidr: "10.0.128.0/20",
 privateSubnetACidr: "10.0.32.0/19",
 }, {
 availabilityZone: "us-east-1b",
 privateSubnetACidr: "10.0.64.0/19",
 }]
})

// Create a best-practices Aurora Postgresql cluster
// Deploys 17 resources
const multiAvaialabilityZoneAuroraCluster = new auroraQuickstart.Cluster("example-aurora-cluster", {
 vpcID: multiAvailabilityZoneVpc.vpcID,
 dbName: "myDemoDatabase",
 dbEngineVersion: "11.9",
 dbInstanceClass: "db.t3.medium",
 availabilityZoneNames: ["us-east-1a", "us-east-1b"],
 dbNumDbClusterInstances: 2,
 dbMasterUsername: "mainuser",
 snsNotificationEmail: databaseNotificationEmail,
 enableEventSubscription: true,
 dbMasterPassword: dbPasswordSecret,
 dbParameterGroupFamily: "aurora-postgresql11",
 privateSubnetID1: multiAvailabilityZoneVpc.privateSubnetIDs.apply(x => x![0]),
 privateSubnetID2: multiAvailabilityZoneVpc.privateSubnetIDs.apply(x => x![1]),
});

// Create a best-practices Redshift cluster
// Deploys 8 resources
const cluster = new redshiftQuickstart.Cluster("example-redshift-cluster", {
 vpcID: multiAvailabilityZoneVpc.vpcID,
 dbPort: 5432,
 dbClusterIdentifier: "example-redshift-cluster",
 dbMasterUsername: "mainuser",
 notificationEmail: notificationEmail,
 enableEventSubscription: true,
 dbMasterPassword: dbPasswordSecret,
 dbName: "main",
 dbNodeType: "dc2.large",
 subnetIDs: multiAvailabilityZoneVpc.privateSubnetIDs as pulumi.Output<string[]>
})

These AWS Quick Start components are recommended for developers with all levels of AWS skill. For developers who don’t work with AWS frequently, these Quick Starts can replace the AWS Console Wizards for confidently provisioning resources.

For experienced AWS developers who want to use these components and are concerned about abstraction, there are several options: (1) review the underlying open source code, (2) do a quick proof-of-concept deployment and use Pulumi’s resource management tools to review the resources, and (3) when necessary, alter the underlying code! In my experience, because I was so accustomed to defining every painful detail of AWS infrastructure, the abstractions felt almost too easy to use. However, once I became comfortable with the underlying design patterns, these packages made deploying complicated AWS architectures incredibly easy, quick, and reliable.

For service providers considering providing a Pulumi package for their service offering – as an AWS developer, the most exciting part of this toolset is the ability to choose non-AWS tools without sacrificing convenience. I sometimes choose Cognito over Okta simply because Cognito is in the AWS ecosystem. If service provider packages are as convenient as using native AWS tools, they become more competitive with AWS’ native services.

Returning to the initial three problems:

AWS Console Wizard Fever - By creating many of the “helper” resources (like email notifications when the database goes down), the Pulumi registry serves the same purpose as the AWS Console wizards – “best practice” infrastructure / applications are available at the click of a button.
YAML/JSON CloudFormation Template Ceiling - Software / programming languages are much easier to work with than YAML or JSON templates. Programming languages provide simpler logic, abstraction, auto-complete, guardrails, open-source imports, and all of the other benefits of programming languages. Furthermore, the Pulumi framework provides infrastructure review tools during deployments, which mitigates the risks of programming language bugs affecting deployments (eg. there are robust safeguards against an infinite “for” loop spinning up an infinite number of EC2 instances).
AWS Lock-in - In addition to Pulumi making it substantially easier to use AWS services, Pulumi makes it substantially easier to use other SAAS and cloud providers alongside AWS resources. As the registry matures around standard abstractions, I can imagine constructs for “Postgres Database” or “Serverless Website” or “Next.js Deployment” that are implemented by multiple SAAS or cloud providers such that a developer can switch clouds with a few minor configuration changes.

For getting started, I recommend starting with the examples for the AWS Quick Start VPC, AWS Quick Start Aurora Postgres, and Aurora Quick Start Redshift.