Event-Driven Image Recognition with Amazon Rekognition and SQS

Question

Pulumi · Accepted Answer

To create an event-driven image recognition system using Amazon Rekognition and Amazon SQS (Simple Queue Service), you would need to set up the following components:

1. An Amazon S3 bucket – to store the images that you want to be recognized.
2. Amazon Rekognition – an image recognition service that will detect objects, scenes, and faces in images.
3. Amazon SQS – a message queuing service that will handle the communication between your services.
4. AWS Lambda – a compute service that you can use to run your recognition logic without managing servers.
5. IAM Roles – to grant necessary permissions for services to access each other.

Here's a high-level overview of how these components work together:
- Images are uploaded to the S3 bucket.
- The upload event triggers a Lambda function.
- The Lambda function sends the image to Rekognition for analysis.
- Rekognition processes the image and the analysis results are sent to SQS.
- An application can then pull the message from SQS and use the analysis data.

Let's construct a Pulumi program to create this architecture in Python. We'll use the Pulumi AWS SDK because it's the high-level SDK that provides rich Pulumi components for AWS resources.

```python
import pulumi
import pulumi_aws as aws

# Define an Amazon S3 bucket where images will be stored
s3_bucket = aws.s3.Bucket('image-bucket')

# Define an IAM Role for AWS Lambda, allowing it to call Rekognition and send messages to SQS
lambda_execution_role = aws.iam.Role('lambda-execution-role', assume_role_policy="""{
   "Version": "2012-10-17",
   "Statement": [{
       "Action": "sts:AssumeRole",
       "Effect": "Allow",
       "Principal": {
           "Service": "lambda.amazonaws.com"
       }
   }]
}""")

rekognition_policy = aws.iam.Policy('rekognition-policy',
    policy=s3_bucket.arn.apply(lambda arn: f"""{{
        "Version": "2012-10-17",
        "Statement": [
            {{
                "Effect": "Allow",
                "Action": "rekognition:*",
                "Resource": "*"
            }},
            {{
                "Effect": "Allow",
                "Action": "s3:GetObject",
                "Resource": "{arn}/*"
            }},
            {{
                "Effect": "Allow",
                "Action": "sqs:SendMessage",
                "Resource": "*"  # Replace with SQS ARN after creating the queue
            }}
        ]
    }}""")
)

rekognition_role_policy_attachment = aws.iam.RolePolicyAttachment('rekognition-role-policy-attachment',
    role=lambda_execution_role.name,
    policy_arn=rekognition_policy.arn
)

# Create an Amazon SQS queue that will receive the analysis results
sqs_queue = aws.sqs.Queue('image-analysis-results')

# Define a Lambda function that will be triggered by S3 events
lambda_function = aws.lambda_.Function('image-analysis-function',
    code=pulumi.AssetArchive({
        '.': pulumi.FileArchive('./lambda')  # Assumes your Lambda code is packaged in the 'lambda' directory
    }),
    role=lambda_execution_role.arn,
    handler='lambda_function.handler',
    runtime='python3.8',  # Choose the appropriate runtime for your Lambda function
    environment=aws.lambda_.FunctionEnvironmentArgs(
        variables={
            "SQS_QUEUE_URL": sqs_queue.url,
        }
    ),
    opts=pulumi.ResourceOptions(depends_on=[rekognition_role_policy_attachment])
)

# Allow the Lambda function to be invoked by S3 bucket events
lambda_permission = aws.lambda.Permission('lambda-permission',
    action='lambda:InvokeFunction',
    function=lambda_function.name,
    principal='s3.amazonaws.com',
    source_arn=s3_bucket.arn,
    source_account=lambda_execution_role.arn.apply(lambda arn: arn.split(':')[4])
)

# Configure the S3 bucket to trigger the Lambda function upon image upload event
s3_bucket_notification = aws.s3.BucketNotification('s3-bucket-notification',
    bucket=s3_bucket.id,
    lambda_functions=[
        aws.s3.BucketNotificationLambdaFunctionArgs(
            lambda_function_arn=lambda_function.arn,
            events=["s3:ObjectCreated:*"],
            filter_prefix="images/"
        )
    ],
    opts=pulumi.ResourceOptions(depends_on=[lambda_permission])
)

# Export relevant data
pulumi.export('s3_bucket_name', s3_bucket.bucket)
pulumi.export('sqs_queue_url', sqs_queue.url)
```

In this program, we:

- Create an Amazon S3 bucket (which acts as a trigger for the Lambda function when an object is created).
- Define an IAM role for Lambda execution with a policy granting access to Amazon Rekognition and Amazon SQS.
- Establish an SQS queue to hold the results from the Rekognition analysis.
- Construct an AWS Lambda function with the necessary permissions, which is triggered by S3 object creation. It pulls the image from S3 and sends it off to Rekognition, and then passes the results to the SQS queue. Note that you should replace `'./lambda'` with the path to your Lambda code, and `'python3.8'` with the language version that you're using.
- Configure event notifications on the S3 bucket so that the Lambda function is invoked when a new image is uploaded to the bucket.

After deploying this Pulumi program, your event-driven image recognition infrastructure will be ready. When you upload images to the designated S3 bucket, it will kick off the process that leads to image analysis and result queuing.