Using aws datasync with ses-service

Question

Pulumi · Accepted Answer

AWS DataSync is a data transfer service that simplifies, automates, and accelerates moving data between on-premises storage systems and AWS storage services, as well as between AWS storage services. On the other hand, Amazon Simple Email Service (SES) is a cloud-based email sending service designed to help digital marketers and application developers send marketing, notification, and transactional emails.

If you are looking to use AWS DataSync to move data to or from AWS services and to trigger email notifications via SES based on certain events related to the DataSync operation (like task completion), you can do so by setting up an AWS DataSync task and utilizing AWS Lambda functions along with Amazon SES for notifications.

Below is a program written in TypeScript that will set up an AWS DataSync task to copy data from an NFS location to an S3 bucket, and an AWS Lambda function to send an email notification using Amazon SES when the DataSync task execution is successful.

I will explain each part of the code to ensure you understand what’s happening.

```typescript
import * as aws from "@pulumi/aws";

// Note: To use this, you must have configured the AWS provider with the proper access key, secret, and region.

// DataSync NFS Location - This is your on-premises NFS filesystem that you want to sync from.
const nfsLocation = new aws.datasync.NfsLocation("nfsLocation", {
    serverHostname: "nfs-server.example.com", // Replace with your actual NFS server hostname
    subdirectory: "/exported/path", // Replace with the NFS directory you want to sync
    onPremConfig: {
        agentArns: ["arn:aws:datasync:region:account-id:agent/agent-id"], // Replace with your actual DataSync Agent ARN
    },
});

// DataSync S3 Location - This is the S3 bucket where you will copy the data from your NFS location.
const s3Location = new aws.datasync.S3Location("s3Location", {
    s3BucketArn: "arn:aws:s3:::my-datasync-bucket", // Replace with your actual S3 bucket ARN
    s3Config: {
        bucketAccessRoleArn: "arn:aws:iam::account-id:role/role-name", // Replace with the ARN for the IAM role with permissions for S3 access
    },
    // You can specify tags here if you want to organize your resources.
});

// DataSync Task - This is the task that will perform the data synchronisation from NFS to S3.
const task = new aws.datasync.Task("task", {
    sourceLocationArn: nfsLocation.arn,
    destinationLocationArn: s3Location.arn,
    // Specifies how and what to copy - more elaborate configuration could be added here
    // Optionally, you can configure CloudWatchLogs for logging, an execution schedule, and advanced task settings.
});

// An IAM role that Lambda will assume to execute your function.
const role = new aws.iam.Role("lambdaRole", {
    assumeRolePolicy: {
        Version: "2012-10-17",
        Statement: [{
            Action: "sts:AssumeRole",
            Effect: "Allow",
            Principal: {
                Service: "lambda.amazonaws.com",
            },
        }],
    },
});

// Attach a policy to the Lambda role for permission to use SES for sending emails.
const policy = new aws.iam.RolePolicyAttachment("lambdaPolicyAttachment", {
    role: role,
    policyArn: aws.iam.ManagedPolicy.AmazonSESFullAccess, // Provides full access to SES.
});

// An AWS Lambda function that will send an email via SES when triggered.
const lambda = new aws.lambda.Function("sendSESEmail", {
    code: new pulumi.asset.AssetArchive({
        ".": new pulumi.asset.FileArchive("./lambda"), // Your Lambda function code should be in the 'lambda' directory.
    }),
    runtime: aws.lambda.Runtime.NodeJS12dX, // Choose the runtime that matches your function code.
    role: role.arn,
    handler: "index.handler", // The filename is `index.js` and `handler` is the exported function that gets called.
    // More configurations like environment variables can be added here.
});

// Here is where we set up Lambda to listen for the DataSync Task Completion event
const eventRule = new aws.cloudwatch.EventRule("datasyncTaskCompletionRule", {
    eventPattern: JSON.stringify({
        source: ["aws.datasync"],
        detailType: ["DataSync Task Execution State Change"],
        detail: {
            state: ["SUCCESS"], // You can also add additional states like FAILURE for different notifications.
        },
    }),
});

// Grant CloudWatch Events permission to invoke your Lambda function.
const permission = new aws.lambda.Permission("lambdaInvokePermission", {
    action: "lambda:InvokeFunction",
    principal: "events.amazonaws.com",
    sourceArn: eventRule.arn,
    function: lambda,
});

// Target that links the Lambda function and the CloudWatch event rule.
const target = new aws.cloudwatch.EventTarget("datasyncTaskCompletionTarget", {
    rule: eventRule.name,
    arn: lambda.arn,
    // You can define input, input path, etc. to send to your Lambda function if needed.
});

// Output the ARNs of the DataSync Location, DataSync Task, and Lambda function to access them easily if needed.
export const nfsLocationArn = nfsLocation.arn;
export const s3LocationArn = s3Location.arn;
export const dataSyncTaskArn = task.arn;
export const lambdaFunctionArn = lambda.arn;
```

Before running the above program, you’ll need to make sure that you have:

1. Configured your Pulumi CLI for the AWS provider.
2. An existing NFS server with a configured DataSync agent.
3. An existing S3 bucket.
4. Set `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_REGION` environment variables or configured them in the local AWS config file.

This code defines the resources needed to set up a DataSync task with related cloud resources, including a Lambda function to handle SES-specific email notifications.

Remember to replace values for `serverHostname`, `subdirectory`, `agentArns`, `s3BucketArn`, `bucketAccessRoleArn`, and other placeholders with your specific information prior to running the program. Additionally, the actual Lambda function code ('index.js') that will use the SES service to send an email must be written and available in a directory named 'lambda'.

In the Lambda function code, you need to use the AWS SES API via the AWS SDK for JavaScript to send an email. Follow AWS SES documentation on how to format that request.

Lastly, if you wish to trigger emails based on other states (like failure), adjust the `eventPattern` to include those states and potentially modify your Lambda function to handle different types of events accordingly.