ETL pipeline with Amazon Redshift and AWS Glue

This example lives in the pulumi/examples repository. Check out just this directory to use it:

$ git clone --filter=blob:none --sparse https://github.com/pulumi/examples pulumi-examples
$ git -C pulumi-examples sparse-checkout set aws-py-redshift-glue-etl
$ cd pulumi-examples/aws-py-redshift-glue-etl

This example creates an ETL pipeline using Amazon Redshift and AWS Glue. The pipeline extracts data from an S3 bucket with a Glue crawler, transforms it with a Python script wrapped in a Glue job, and loads it into a Redshift database deployed in a VPC.

Prerequisites#

1. Install Pulumi
2. Configure AWS credentials
3. Install Python

Deploying the example#

1. Create a new stack:

   pulumi stack init dev

2. Set the AWS region to deploy into:

   pulumi config set aws:region us-west-2

3. Install dependencies:

   python3 -m venv venv
   source venv/bin/activate
   pip install -r requirements.txt

4. Deploy the stack:

   pulumi up

5. In a few moments, the Redshift cluster and Glue components will be up and running and the S3 bucket name emitted as a Pulumi stack output:

   Outputs:
       dataBucketName: "events-56e424a"

6. Upload the included sample data file to S3 to verify the automation works as expected:

   aws s3 cp events-1.txt s3://$(pulumi stack output dataBucketName)

Cleaning up#

Once you’re finished experimenting, destroy your stack and remove it to avoid incurring any additional cost:

pulumi destroy
pulumi stack rm