Event-driven Image Recognition Workflows with AWS Lambda

Question

Pulumi · Accepted Answer

To create an event-driven image recognition workflow with AWS Lambda, you'll essentially be creating a serverless function that responds to events. Here's the high-level flow of how such a system would work:

1. Images are uploaded to Amazon S3 (Simple Storage Service), which stores and retrieves any amount of data, at any time, from anywhere on the web.
2. An event notification is emitted from the S3 bucket upon each new image upload.
3. AWS Lambda is configured to trigger from the S3 event notification.
4. The Lambda function processes the image using image recognition, perhaps through Amazon Rekognition or a custom model.

Here's how you might structure a Pulumi program to set up such a workflow:

- **AWS S3 Bucket**: This resource acts as your source of image files which triggers the workflow.
- **AWS Lambda Function**: This function contains the logic for your image recognition tasks. It can use the AWS SDK to interface with Amazon Rekognition for image recognition.
- **AWS Lambda Permission**: This resource grants S3 the permission to invoke your Lambda function.
- **S3 Bucket Events**: You'll configure these events in S3 to trigger the Lambda function.

Below is a detailed Pulumi program written in Python that sets up an event-driven image recognition workflow using AWS Lambda:

```python
import json
import pulumi
import pulumi_aws as aws

# Create an AWS S3 Bucket that will store your images.
images_bucket = aws.s3.Bucket('imagesBucket')

# IAM role which dictates what the Lambda function will have permissions to do.
lambda_role = aws.iam.Role('lambdaRole', assume_role_policy=json.dumps({
    "Version": "2012-10-17",
    "Statement": [{
        "Action": "sts:AssumeRole",
        "Principal": {"Service": "lambda.amazonaws.com"},
        "Effect": "Allow",
        "Sid": "",
    }]
}))

# Attaches the AWSLambdaBasicExecutionRole which grants the lambda permissions to write logs to CloudWatch.
attach_lambda_exec_policy = aws.iam.RolePolicyAttachment("lambdaExecPolicyAttachment",
    role=lambda_role.name,
    policy_arn="arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
)

# The actual Lambda function. The code is assumed to be in 'image_recognition_handler.py'.
lambda_function = aws.lambda_.Function('imageRecognitionFunction',
    code=pulumi.FileArchive('./image_recognition_handler.py'),
    role=lambda_role.arn,
    handler='image_recognition_handler.handler',  # This is the entrypoint into your code.
    runtime='python3.8',
)

# Gives the necessary permissions for the S3 bucket to trigger the Lambda function.
bucket_permission = aws.lambda_.Permission('bucketPermission',
    action='lambda:InvokeFunction',
    function=lambda_function.name,
    principal='s3.amazonaws.com',
    source_arn=images_bucket.arn,
)

# Adds the S3 bucket event that triggers the lambda function upon image upload.
bucket_event = aws.s3.BucketNotification('bucketEvent',
    bucket=images_bucket.id,
    lambda_functions=[aws.s3.BucketNotificationLambdaFunctionArgs(
        lambda_function_arn=lambda_function.arn,
        events=["s3:ObjectCreated:*"],
        filter_prefix="images/",
        filter_suffix=".jpg",
    )],
    opts=pulumi.ResourceOptions(depends_on=[bucket_permission]),
)

# Export the bucket name and Lambda function name as stack outputs.
pulumi.export('bucket_name', images_bucket.id)
pulumi.export('lambda_function_name', lambda_function.name)
```

Before deploying this Pulumi stack, ensure you have the AWS CLI configured with the appropriate credentials and permissions.

This script does the following:

- It sets up a new S3 bucket to store images.
- It creates an IAM role for the Lambda function with permissions to run and also to log to CloudWatch.
- It defines the Lambda function, pointing to the local file `image_recognition_handler.py` where your image recognition code lives.
- It attaches that role to the Lambda function, ensuring it has the necessary permissions when it executes.
- It configures S3 to trigger the Lambda function whenever new `.jpg` images are uploaded to the `images/` prefix of the bucket.
- Finally, it exports the names of the S3 bucket and the Lambda function for easy reference later.

Your `image_recognition_handler.py` is an independent Python file that you would have to create which includes the logic for handling an S3 event and calling Amazon Rekognition or any other service you use for image recognition. It should define a function `handler`, which will be the entry point when the Lambda function is invoked. The content of this file would depend on the specifics of your image recognition task.