Using aws lambda with datasync

Question

Pulumi · Accepted Answer

To integrate AWS Lambda with AWS DataSync, you typically use Lambda to perform actions triggered by DataSync operations. AWS DataSync is a service used to transfer data between on-premises storage and AWS storage services such as Amazon S3, EFS, and FSx for Windows File Server.

Here is how AWS Lambda can interact with AWS DataSync:

1. **AWS Lambda**: You can create a Lambda function to process data once it has been transferred by DataSync. For example, you might trigger a Lambda function after DataSync has finished transferring files to an S3 bucket.

2. **DataSync Events**: DataSync can send events to Amazon EventBridge when tasks start and complete. You can set up a rule in EventBridge to trigger your Lambda function in response to these events.

3. **Lambda Permissions**: In order to allow Lambda to interact with DataSync, you need to set up the correct IAM permissions.

In the following TypeScript program, I'll create:
- An AWS DataSync Task that synchronizes data from an S3 bucket to an NFS server.
- An AWS Lambda function which will be triggered by EventBridge when a DataSync task status changes.
- EventBridge rules to trigger the Lambda function on DataSync task status changes.

```typescript
import * as aws from '@pulumi/aws';

// Assume we already have an S3 bucket and NFS file system as source and destination locations
const sourceLocation = 'arn:aws:datasync:region:123456789012:location/loc-12345678901234567';
const destinationLocation = 'arn:aws:datasync:region:123456789012:location/loc-76543210987654321';

// Create a DataSync task between the S3 bucket and NFS file system
const datasyncTask = new aws.datasync.Task('my-datasync-task', {
    sourceLocationArn: sourceLocation,
    destinationLocationArn: destinationLocation,
    name: 'MyDataSyncTask',
    options: {
        // Configure the options as needed
        transferMode: 'CHANGED', // Only transfer changed files
    },
});

// Create an IAM role for the Lambda function
const lambdaRole = new aws.iam.Role('lambda-datasync-role', {
    assumeRolePolicy: {
        Version: '2012-10-17',
        Statement: [{
            Action: 'sts:AssumeRole',
            Effect: 'Allow',
            Principal: {
                Service: 'lambda.amazonaws.com',
            },
        }],
    },
});

// Attach the AWSLambdaBasicExecutionRole policy
new aws.iam.RolePolicyAttachment('lambda-basic-execution', {
    role: lambdaRole,
    policyArn: 'arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole',
});

// Create the Lambda function
const lambda = new aws.lambda.Function('datasyncLambda', {
    runtime: aws.lambda.Runtime.NodeJS12dX, // Select the proper runtime
    handler: 'index.handler',
    role: lambdaRole.arn,
    code: new pulumi.asset.AssetArchive({
        '.': new pulumi.asset.FileArchive('./path-to-lambda-code'), // Specify the path to your Lambda code
    }),
});

// Grant permission to the Lambda function from EventBridge (CloudWatch Events)
new aws.lambda.Permission('lambda-eventbridge-permission', {
    action: 'lambda:InvokeFunction',
    function: lambda,
    principal: 'events.amazonaws.com',
    sourceArn: datasyncTask.arn,
});

// Create an EventBridge rule to trigger the Lambda function on DataSync task status changes
new aws.cloudwatch.EventRule('datasync-event-rule', {
    description: 'Triggers a Lambda function on AWS DataSync task status changes',
    eventPattern: JSON.stringify({
        source: ["aws.datasync"],
        detail: {
            "task-status": ["SUCCESS", "ERROR"],
        },
    }),
});

// Add the Lambda function as a target for the EventBridge rule
new aws.cloudwatch.EventTarget('datasync-event-target', {
    rule: 'datasync-event-rule',
    arn: lambda.arn,
});

// Output the Lambda function ARN
export const lambdaFunctionArn = lambda.arn;

// Output the DataSync task ARN
export const datasyncTaskArn = datasyncTask.arn;
```

### Explanation

- We begin by defining an AWS DataSync Task using the `aws.datasync.Task` resource. Here, we are providing ARNs for the source and destination locations. The source is an S3 bucket, and the destination is an NFS server.

- We create an IAM role for our Lambda function using the `aws.iam.Role` resource. We attach the `AWSLambdaBasicExecutionRole` policy which allows our Lambda function to log to CloudWatch.

- The Lambda function itself is created using the `aws.lambda.Function` resource. It references the previously created IAM role and the code archive, which should contain your Lambda function's code.

- Next, we add a `aws.lambda.Permission` resource that grants AWS EventBridge permission to invoke our Lambda function in response to events.

- An AWS EventBridge rule is set up using `aws.cloudwatch.EventRule` to filter for the success or error status updates from AWS DataSync tasks.

- The `aws.cloudwatch.EventTarget` integrates our Lambda function with our EventBridge rule, effectively setting the function as a target to be invoked when the rule's conditions are met.

- Finally, we export the ARNs of the Lambda function and the DataSync task for easy reference.

### What to Do Next

- Replace `region` and `123456789012` with your AWS region and account ID, respectively.
- Provide the actual source and destination ARN values for the DataSync task.
- Change the path `'./path-to-lambda-code'` to the location of your Lambda function code.

When you deploy this Pulumi program, it will set up the DataSync task, the Lambda function, and the EventBridge rule to invoke the Lambda function whenever the DataSync task status changes. Ensure your Lambda function is coded to handle the input from EventBridge events and perform the desired actions upon task completion.