Automating file transfers to S3 with AWS Transfer workflows

Question

Pulumi · Accepted Answer

Automating file transfers to Amazon S3 using AWS Transfer Family allows you to securely move files into and out of Amazon S3 storage. AWS Transfer Family provides fully managed support for file transfers directly into and out of Amazon S3 using SFTP, FTPS, and FTP.

To automate this operation with Pulumi, you can create a workflow that defines a series of steps to process the files (e.g., copy, tag, or delete), as well as a Transfer server, which will handle the file transfers, and optionally, a user that will be allowed to perform the operations.

Below is a Pulumi program written in TypeScript that sets up an AWS Transfer Workflow including a server to handle file transfers and S3 buckets for storage. The program will create a simple workflow with a single step to copy files into an S3 bucket whenever they are uploaded.

```typescript
import * as aws from "@pulumi/aws";

// Create an S3 bucket to store files.
const s3Bucket = new aws.s3.Bucket("my-transfer-bucket", {
    acl: "private", // Using 'private' access control to keep the files secure.
});

// Create an AWS Transfer Server to handle SFTP transfers.
const sftpServer = new aws.transfer.Server("my-sftp-server", {
    protocols: ["SFTP"], // Setting up the server to use SFTP protocol.
    endpointType: "PUBLIC", // Making the server public to accept connections.
    identityProviderType: "SERVICE_MANAGED", // Using service-managed authentication.
    loggingRole: undefined, // Optional: Set up a logging role for audit purposes.
});

// Define a workflow to handle file uploads.
const workflow = new aws.transfer.Workflow("my-transfer-workflow", {
    steps: [{
        type: "COPY", // Defining a copy step to move files to S3 bucket.
        copyStepDetails: {
            name: "CopyToS3",
            destinationFileLocation: {
                s3FileLocation: {
                    bucket: s3Bucket.id, // Using the previously created S3 bucket.
                    key: "${Transfer:Input[*]}" // The key for the object in S3 (uses wildcard to denote dynamic filenames).
                }
            },
            sourceFileLocation: "${Transfer:HomeDirectory}/${Transfer:FileName}", // The original file location in the user's directory.
        }
    }],
    onExceptionSteps: [], // Optional: Define steps to handle exceptions.
    tags: {
        "Name": "MyTransferWorkflow",
    },
});

// Attach the workflow to the SFTP server for when files are uploaded.
// This requires setting up an IAM role that gives the server permission to access the S3 bucket and execute the workflow.
const executionRolePolicy = new aws.iam.Policy("executionRolePolicy", {
    policy: s3Bucket.arn.apply(arn => JSON.stringify({
        Version: "2012-10-17",
        Statement: [{
            Effect: "Allow",
            Action: ["s3:GetObject", "s3:PutObject", "s3:ListBucket"],
            Resource: [`${arn}/*`, arn],
        }],
    })),
});

const executionRole = new aws.iam.Role("executionRole", {
    assumeRolePolicy: JSON.stringify({
        Version: "2012-10-17",
        Statement: [{
            Effect: "Allow",
            Principal: {
                Service: "transfer.amazonaws.com",
            },
            Action: "sts:AssumeRole",
        }],
    }),
});

const executionRolePolicyAttachment = new aws.iam.RolePolicyAttachment("executionRolePolicyAttachment", {
    role: executionRole.name,
    policyArn: executionRolePolicy.arn,
});

const workflowAttachment = new aws.transfer.ServerWorkflowDetails("my-workflow-details", {
    serverId: sftpServer.id,
    workflowId: workflow.id,
    executionRole: executionRole.arn,
});

// Export the S3 bucket name and the SFTP server endpoint.
export const bucketName = s3Bucket.bucket;
export const sftpServerEndpoint = sftpServer.endpoint;

// You can now provide users with SFTP access details (after setting up an SFTP user) and they can upload files that get copied to the specified S3 bucket.
```

This script is ready to run with Pulumi and will deploy the AWS resources necessary to set up an automated file transfer to S3 via AWS Transfer Workflows. You will need to set up an SFTP user separately, either through the AWS Console or via Pulumi, and then provide your users with credentials for uploading files.

Each resource is annotated with comments to explain what it's for. After creating the resources, we export the S3 bucket name and SFTP server endpoint, which you can share with your users. To fully deploy this infrastructure, ensure you have authenticated with the AWS provider and have sufficient permissions to create the resources.