Low-latency Model Serving Using AWS CloudFront

Question

Pulumi · Accepted Answer

To serve a machine learning model with low-latency using AWS CloudFront, you would follow these high-level steps:

1. Store your trained machine learning model artifacts in a location that can be integrated with CloudFront, such as an Amazon S3 bucket.
2. Deploy an inference service, which will load the model and perform predictions. This service can be deployed on AWS Lambda, Amazon ECS, or Amazon EKS, depending on your requirements. AWS Lambda is a common choice due to its serverless nature, ease of scaling, and tight integration with CloudFront via Lambda@Edge.
3. Set up a CloudFront distribution. CloudFront can serve content from AWS services like S3 or a custom origin like an EC2 instance or ECS service. In this case, you would use the inference service as the origin.
4. Optionally, configure caching policies and other settings to optimize latency and performance. For example, you can adjust Time-to-Live (TTL) settings, secure your content with HTTPS, and add custom headers.

With Pulumi, you can define all the required infrastructure using code. Below is a Pulumi Python program that sets up an S3 bucket to store your model, deploys an AWS Lambda function for inference, and creates a CloudFront distribution to serve predictions with low latency.

```python
import pulumi
import pulumi_aws as aws

# Create an S3 bucket to store the machine learning model
model_bucket = aws.s3.Bucket("model-bucket")

# Upload your trained machine learning model to the S3 bucket
# (Assuming you have a file "./model.tar.gz" that contains your trained model)
model_file = aws.s3.BucketObject("model-file",
    bucket=model_bucket.id,
    key="model.tar.gz",
    source=pulumi.FileAsset("./model.tar.gz") # Local file path of the model
)

# Create a Lambda function that will serve as the inference service
# This assumes you have a zipped archive of your Lambda function code
lambda_function = aws.lambda_.Function("model-inference-lambda",
    code=pulumi.AssetArchive({"index.zip": pulumi.FileAsset("./lambda/inference.zip")}),
    role=some_lambda_role.arn,  # IAM role with permissions for the Lambda function
    handler="index.handler",    # The handler to invoke in your Lambda code
    runtime="python3.8",        # Replace with your Lambda's runtime, e.g., 'python3.8'
    timeout=30,                 # Lambda function timeout (in seconds)
    memory_size=1024,           # Memory allocated to the Lambda function (in MB)
)

# Create an API Gateway to invoke the Lambda function via HTTP(S)
# This will be used as the origin for CloudFront
api_gateway = aws.apigatewayv2.Api("model-inference-api",
    protocol_type="HTTP",
    route_key=f"POST /infer",
    target=lambda_function.invoke_arn
)

# Create a CloudFront distribution to serve the inference requests with low latency
# This will route requests to the API Gateway which triggers the Lambda function
cloudfront_distribution = aws.cloudfront.Distribution("model-serving-cdn",
    enabled=True,
    # Point the CloudFront distribution to the API Gateway as the origin
    origins=[aws.cloudfront.DistributionOriginArgs(
        domain_name=api_gateway.api_endpoint.apply(lambda endpoint: endpoint.split("//")[1]),
        origin_id="api-gateway-origin",
    )],
    default_cache_behavior=aws.cloudfront.DistributionDefaultCacheBehaviorArgs(
        viewer_protocol_policy="redirect-to-https",
        allowed_methods=["GET", "HEAD", "OPTIONS", "PUT", "POST", "PATCH", "DELETE"],
        cached_methods=["GET", "HEAD", "OPTIONS"],
        target_origin_id="api-gateway-origin",
        forwarded_values=aws.cloudfront.DistributionDefaultCacheBehaviorForwardedValuesArgs(
            query_string=True,
        ),
        lambda_function_associations=[aws.cloudfront.DistributionDefaultCacheBehaviorLambdaFunctionAssociationArgs(
            event_type="origin-request",
            lambda_arn=lambda_function.arn,
        )],
    ),
    # Configuration for other CloudFront settings (optional)
    restrictions=aws.cloudfront.DistributionRestrictionsArgs(
        geo_restriction=aws.cloudfront.DistributionRestrictionsGeoRestrictionArgs(
            restriction_type="none",
        ),
    ),
    viewer_certificate=aws.cloudfront.DistributionViewerCertificateArgs(
        cloudfront_default_certificate=True,
    ),
)

# Export the CloudFront distribution domain name
pulumi.export("cloudfront_domain_name", cloudfront_distribution.domain_name)
```

In this program:

- We first created an S3 bucket to store our machine learning model artifacts.
- We then uploaded the model to this bucket using `BucketObject`, providing the path to the model file on the Pulumi host.
- We set up a `Lambda Function` which serves as the inference endpoint. It's packaged as a `.zip` file and deployed to AWS Lambda.
- An `API Gateway` was created to allow a standardized HTTP(S) endpoint that will trigger our Lambda function.
- Lastly, we crafted a `CloudFront Distribution` which directs traffic to our API Gateway, effectively reducing latency and distributing requests at scale.

Upon deploying this with Pulumi, the `cloudfront_domain_name` export at the bottom of the script will display the domain name that can be used to invoke the model inference service with low latency around the world.

Remember to replace `"./lambda/inference.zip"` and `some_lambda_role` with the actual file path to your Lambda function's zipped code and a reference to the IAM role you want the Lambda function to assume.

This is a starting point for serving models using CloudFront and AWS Lambda. You might need to fine-tune the caching settings or tweak Lambda configurations depending on the specific nature of your model serving needs.