Host Python Libraries for Data Preprocessing on AWS Lambda

Question

Pulumi · Accepted Answer

To host Python libraries for data preprocessing on AWS Lambda, you will need to package those libraries into a Lambda Layer. Lambda Layers are a way to manage your function's dependencies separately, enabling you to keep your deployment package small. This is essential because AWS Lambda has a limit on the deployment package size.

Here are the key resources we'll use to achieve this:

1. **AWS Lambda Layer (`aws.lambda.LayerVersion`)** - This resource will hold the Python libraries you need for data preprocessing.

2. **AWS Lambda Function (`aws.lambda.Function`)** - This represents your main Lambda function that will process the data. It will use the libraries from the Lambda Layer.

I will guide you through a Pulumi Python program that will create a Lambda Layer with the required Python libraries and an AWS Lambda Function that will use this layer for its operation.

First, you need a way to package your Python libraries. You can do that with a script that installs the libraries into a directory and packages them in a ZIP file that you can upload to AWS Lambda as a Layer. Typically, you would use a `requirements.txt` file to list your Python dependencies and then run:

```bash
pip install -r requirements.txt -t python/lib/python3.8/site-packages/
zip -r my-python-libraries-layer.zip python
```

Where `python/lib/python3.8/site-packages/` is a path that matches the Python execution environment on the Lambda service. Adjust the path if you're using a different Python runtime version.

After your ZIP file is created, you can use it in a Pulumi program as follows:

```python
import pulumi
import pulumi_aws as aws

# Create an AWS Lambda Layer with the Python libraries ZIP file.
python_libs_layer = aws.lambda_.LayerVersion("pythonLibsLayer",
    compatible_runtimes=["python3.8"],
    code=pulumi.FileAsset("my-python-libraries-layer.zip"),
    layer_name="python-libs-layer"
)

# Create an IAM role for the Lambda function.
lambda_role = aws.iam.Role("lambdaRole",
    assume_role_policy="""{
       "Version": "2012-10-17",
       "Statement": [{
           "Action": "sts:AssumeRole",
           "Principal": {
               "Service": "lambda.amazonaws.com"
           },
           "Effect": "Allow",
           "Sid": ""
       }]
    }"""
)

# Attach the AWS managed LambdaBasicExecutionRole so that your Lambda
# function can log to CloudWatch.
attachment = aws.iam.RolePolicyAttachment("lambdaLogs",
    role=lambda_role.name,
    policy_arn="arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
)

# Create the Lambda function, referencing the created IAM role and Lambda Layer.
lambda_function = aws.lambda_.Function("myLambdaFunction",
    code=pulumi.AssetArchive({
        ".": pulumi.FileArchive("./lambda_function")  # Directory with your Lambda handler
    }),
    handler="handler.main",  # Assuming "main" function in "handler.py"
    role=lambda_role.arn,
    runtime="python3.8",
    layers=[python_libs_layer.arn]
)

# Export the Lambda function's name and the layer's ARN.
pulumi.export('lambda_function_name', lambda_function.name)
pulumi.export('lambda_layer_arn', python_libs_layer.arn)
```

In this program:
- `pulumi.FileAsset("my-python-libraries-layer.zip")` uploads the ZIP file created earlier to AWS Lambda as a Layer.
- `pulumi.FileArchive("./lambda_function")` is a placeholder directory for your Lambda function's code. It should contain all the files needed by the Lambda, including a `handler.py` with a `main` function.
- `lambda_function` is created with the Python 3.8 runtime and `python_libs_layer.arn` as a layer, granting it access to the Python libraries in the layer.

Ensure you have Pulumi installed and configured with appropriate AWS credentials before running this program. After running the program with `pulumi up`, Pulumi will provision these resources in AWS on your behalf. Your Lambda will now be able to use the libraries in the Layer for data preprocessing tasks.