Event-driven Video Analysis Workflow with S3 and AWS Lambda

Question

Pulumi · Accepted Answer

Creating an event-driven video analysis workflow on AWS using Pulumi involves several steps and AWS services. The general idea is to have video files uploaded to S3 trigger an AWS Lambda function, which can perform video analysis using AWS services like Amazon Rekognition. In this setup:

- **Amazon S3** is used to store video files. Events are generated from S3 when a new video file is uploaded.
- **AWS Lambda** is a serverless compute service that can run code in response to events, like S3 events, and automatically manage the compute resources.
- **IAM Roles and Policies** are required for securely granting the necessary permissions to Lambda and other AWS services involved.
- **Amazon Rekognition** can be used by the Lambda function to analyze the video content.

I'll guide you through a Pulumi Python program that sets up this workflow:

1. Define an S3 bucket for storing the video files.
2. Create an IAM role and policy that gives the necessary permissions to the Lambda function.
3. Define a Lambda function that gets triggered when a new video is uploaded to the S3 bucket.
4. Set up S3 event notifications to trigger the Lambda function.
5. Grant the Lambda function permission to be invoked by S3.

Note that this assumes you have AWS credentials configured for Pulumi to use. Here is a program that accomplishes this:

```python
import pulumi
import pulumi_aws as aws

# Create an S3 bucket to store video files
video_bucket = aws.s3.Bucket("videoBucket")

# Create an IAM Role for the AWS Lambda function
lambda_role = aws.iam.Role("lambdaRole", assume_role_policy="""{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Action": "sts:AssumeRole",
         "Principal": {
            "Service": "lambda.amazonaws.com"
         },
         "Effect": "Allow",
         "Sid": ""
      }
   ]
}
""")

# Attach the AWSLambdaBasicExecutionRole policy to the role created above
# This provides Lambda permission to write logs to CloudWatch
basic_execution_policy_attachment = aws.iam.RolePolicyAttachment("basicExecutionPolicyAttachment",
    role=lambda_role.name,
    policy_arn="arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole")

# Allow the Lambda function to call AWS Rekognition and to read from the S3 bucket
rekognition_policy = aws.iam.RolePolicy("rekognitionPolicy",
    role=lambda_role.name,
    policy=pulumi.Output.all(video_bucket.arn).apply(lambda arn: f"""{{
       "Version": "2012-10-17",
       "Statement": [
          {{
             "Effect": "Allow",
             "Action": "rekognition:*",
             "Resource": "*"
          }},
          {{
             "Effect": "Allow",
             "Action": [
                "s3:GetObject",
                "s3:ListBucket"
             ],
             "Resource": [
                "{arn}",
                "{arn}/*"
             ]
          }}
       ]
    }}
    """))

# Define your Lambda function that will be triggered by S3 events
video_analysis_lambda = aws.lambda_.Function("videoAnalysisLambda",
    code=pulumi.FileArchive("./lambda"), # This is a local path to the Lambda code
    role=lambda_role.arn,
    handler="index.handler", # The entrypoint into your Lambda code e.g. `index.handler`
    runtime=aws.lambda_.Runtime.PYTHON_3_8, # The runtime for the Lambda function
    environment=aws.lambda_.FunctionEnvironmentArgs(
        variables={
            "BUCKET": video_bucket.bucket, # Pass environment variables to the Lambda function
        },
    ),
)

# Set up S3 to Lambda notifications
notification = aws.s3.BucketNotification("bucketNotification",
    bucket=video_bucket.bucket,
    lambda_functions=[aws.s3.BucketNotificationLambdaFunctionArgs(
        lambda_function_arn=video_analysis_lambda.arn,
        events=["s3:ObjectCreated:*"],
        filter_prefix="videos/",
    )],
)

# Grant the S3 bucket permission to invoke the Lambda function
permission = aws.lambda_.Permission("permission",
    action="lambda:InvokeFunction",
    function=video_analysis_lambda.name,
    principal="s3.amazonaws.com",
    source_arn=video_bucket.arn,
)

# Export the name of the bucket
pulumi.export('video_bucket_name', video_bucket.bucket)
```

In this program:

- We create an S3 `Bucket` to store our video files. Whenever a new video file is uploaded to a specific prefix ("videos/") in the bucket, it will trigger an event.
- A `Role` for the Lambda function is created with a trust relationship policy that allows Lambda service to assume the role.
- We attach a `RolePolicyAttachment` granting the AWSLambdaBasicExecutionRole, which allows our Lambda function to write logs to CloudWatch.
- A `RolePolicy` for the Lambda role is created to allow interaction with Rekognition and operations on the S3 bucket.
- The `Function` defines the Lambda function we'll use for video analysis. The `handler` specifies the function entry point in our Lambda code, and `FileArchive` is the deployment package.
- An `BucketNotification` sets up the S3 to Lambda event notifications.
- A `Permission` resource grants S3 service the permission to invoke the Lambda function.
- Finally, we export the S3 bucket name as a stack output.

Remember to replace `"./lambda"` with the actual path to your Lambda function source code. Also, ensure that the IAM Role has the necessary permissions for any other AWS services you might call from within the Lambda function.