Creating a media content analytics platform with AWS Elemental, processing logs with Lambda, and storing in Redshift
PythonTo accomplish this goal, you could leverage various AWS resources. Specifically, you can use an AWS Elemental MediaConvert to convert your media files, AWS Lambda to process the logs, and AWS Redshift to store the logs. Here's how you could do this with Pulumi:
import pulumi from pulumi_aws import lambda_, redshift, iam # Create an IAM role that AWS Elemental MediaConvert can assume elemental_role = iam.Role('MediaConvertRole', assume_role_policy="""{ "Version": "2012-10-17", "Statement": [ {"Sid": "", "Effect": "Allow", "Principal": {"Service": "mediaconvert.amazonaws.com"}, "Action": "sts:AssumeRole"} ] }""") # Create an Elemental MediaConvert Job here # The details are skipped since Pulumi does not support AWS Elemental yet # Create an IAM role that AWS Lambda can assume lambda_role = iam.Role('LambdaRole', assume_role_policy="""{ "Version": "2012-10-17", "Statement": [ {"Sid": "", "Effect": "Allow", "Principal": {"Service": "lambda.amazonaws.com"}, "Action": "sts:AssumeRole"} ] }""") # Create an IAM policy attachment that gives the Lambda function access to CloudWatch Logs log_policy_attachment = iam.RolePolicyAttachment('LambdaCloudWatchLogsPolicy', role=lambda_role.name, policy_arn='arn:aws:iam::aws:policy/AWSLambdaBasicExecutionRole') # Create the Lambda function to process the logs lambda_function = lambda_.Function('ProcessLogsFunction', code=pulumi.AssetArchive({ '.': pulumi.FileArchive('./path-to-your-lambda-handler-directory')}), role=lambda_role.arn, handler='index.handler', # replace 'index.handler' if your handler is called differently in your code runtime='nodejs12.x') # Create a Redshift Cluster cluster = redshift.Cluster('LoggingCluster', cluster_identifier='logging-cluster', database_name='process-logs', master_username='username', master_password='password', node_type='dc2.large', cluster_type='single-node', skip_final_snapshot=True) # Export the Redshift Cluster endpoint pulumi.export('redshiftClusterEndpoint', cluster.endpoint)
This Pulumi program has 5 main resource steps:
-
Creates an IAM role that AWS Elemental MediaConvert can assume. You should replace
"mediaConvertRole"
with your desired IAM role name. -
Creates an IAM role that AWS Lambda can assume, and attaches the policy that allows the Lambda function to write logs to CloudWatch.
-
Defines a AWS Lambda function that will process your logs. You must replace
'./path-to-your-lambda-handler-directory'
with the path to your Node.js AWS Lambda handler. -
Creates an AWS Redshift Cluster that will store your processed logs. Be sure to provide a strong password in the
"master_password"
field or usepulumi.config.Config.require_secret
to store it securely. -
Exports the Redshift Cluster's endpoint so you can access your stored logs.
Before you run this program, ensure you have installed the required Python modules in your virtual environment:
pip install pulumi pulumi_aws
Unfortunately, Pulumi does not support AWS Elemental yet, the part with MediaConvert creation is skipped and you need to create it manually or by using some automation scripts. Also, you should modify this program to suit your specific needs, for example, adding tags or configuring optional properties.
Look at the following resource documentation for more details:
-