Managed SFTP Access for Training Data Upload

Question

Pulumi · Accepted Answer

To set up managed SFTP access for uploading training data, we can use AWS Transfer Family, which is a fully managed service by AWS that enables the transfer of files into and out of Amazon S3 storage. We will utilize Pulumi to create an AWS Transfer for SFTP server that will allow users to upload data securely to an S3 bucket.

Here's how we will achieve this:

1. Create an S3 bucket where the training data will be uploaded.
2. Set up an AWS Transfer for SFTP server.
3. Attach the S3 bucket to the SFTP server as the storage location for the uploaded files.

Let's go through each of these steps in a Pulumi Python program.

First, we create an S3 bucket where training data will be stored:

```python
import pulumi
import pulumi_aws as aws

# Create an S3 bucket to store the training data.
training_data_bucket = aws.s3.Bucket("training-data-bucket")

# Export the name of the bucket
pulumi.export("bucket_name", training_data_bucket.id)
```

In this code snippet, we are using the `pulumi_aws` package to create an S3 bucket. The bucket's ID is then exported so that it can be easily referenced or retrieved after deployment.

Next, we will set up the SFTP server and integrate it with the S3 bucket we created:

```python
# Set up AWS Transfer for SFTP server.
sftp_server = aws.transfer.Server("sftp-server",
                                  protocols=["SFTP"],
                                  domain="S3",
                                  endpoint_type="PUBLIC",
                                  identity_provider_type="SERVICE_MANAGED")

# Attach the S3 bucket to the SFTP server with a role that has access to the bucket.
sftp_role = aws.iam.Role("sftp-role",
                         assume_role_policy=sftp_server.arn.apply(
                             lambda arn: json.dumps({
                                 "Version": "2012-10-17",
                                 "Statement": [{
                                     "Effect": "Allow",
                                     "Principal": {"Service": "transfer.amazonaws.com"},
                                     "Action": "sts:AssumeRole",
                                     "Condition": {"StringEquals": {"aws:SourceArn": arn}}
                                 }]
                             })))

sftp_bucket_access_policy = aws.iam.Policy("sftp-bucket-access-policy",
                                           policy=training_data_bucket.arn.apply(
                                               lambda arn: json.dumps({
                                                   "Version": "2012-10-17",
                                                   "Statement": [{
                                                       "Effect": "Allow",
                                                       "Action": ["s3:ListBucket", "s3:GetBucketLocation"],
                                                       "Resource": arn
                                                   }, {
                                                       "Effect": "Allow",
                                                       "Action": ["s3:PutObject", "s3:GetObject", "s3:DeleteObject"],
                                                       "Resource": f"{arn}/*"
                                                   }]
                                               })))

sftp_bucket_access_role_policy_attachment = aws.iam.RolePolicyAttachment("sftp-bucket-access-role-policy-attachment",
                                                                       role=sftp_role.name,
                                                                       policy_arn=sftp_bucket_access_policy.arn)

# Connect the S3 bucket as the storage location for the SFTP server.
sftp_user = aws.transfer.User("sftp-user",
                              server_id=sftp_server.id,
                              role=sftp_role.arn,
                              home_directory=f"/{training_data_bucket.id}")

# Export the SFTP server endpoint.
pulumi.export("sftp_endpoint", sftp_server.endpoint)
```

In this section of the program:

- We create an SFTP server using the AWS Transfer Family service.
- We set up an IAM role (`sftp_role`) with a policy that enables the SFTP service to assume the role.
- We define the permissions (`sftp_bucket_access_policy`) allowing the role to access the S3 bucket.
- We associate the permissions with the IAM role (`sftp_bucket_access_role_policy_attachment`).
- Finally, we create an SFTP user that is associated with the SFTP server and has the IAM role attached, thereby providing access to the S3 bucket. The `home_directory` points to the root of the S3 bucket, meaning when users connect to the SFTP server, they will be within the S3 bucket's namespace.

With this Pulumi program, once deployed, you will get an SFTP endpoint which can be shared with users to upload training data securely to the specified S3 bucket. When users connect to the SFTP server using their SFTP client, they'll be directed to the S3 bucket where they can upload and manage their files.