Canary Deployments for AI Applications using AWS App Mesh

Question

Pulumi · Accepted Answer

Canary deployments are a strategy for deploying applications by gradually shifting traffic from an old to a new version. It allows you to test the new version with a subset of live traffic to verify its behavior before fully committing to the release. AWS App Mesh is a service mesh provided by AWS that makes it easy to monitor and control microservices. App Mesh standardizes how your services communicate, providing end-to-end visibility and ensuring high-availability for your applications.

In the context of AI applications, you might use App Mesh to handle the complex data flows and service-to-service communications that are typical in AI systems, often comprised of multiple specialized microservices.

Here's how you would use Pulumi to configure AWS App Mesh for a canary deployment:

1. **Mesh**: This is the service mesh that all services will reside in.
2. **Virtual Nodes**: These represent the backends your microservices run on. You would typically have a virtual node for each version of your AI service - one for the stable version and one for the new canary version.
3. **Virtual Services**: These define an interface for common access points to the backends represented by virtual nodes.
4. **Virtual Router**: This routes incoming requests to different virtual nodes. With weighted routing, we can direct a small fraction of the traffic to the canary.
5. **Routes**: These are associated with virtual routers and contain the logic on how requests are directed to different virtual nodes based on weights or other criteria.

Let's translate that into a Pulumi program:

```python
import pulumi
import pulumi_aws as aws

# Create an AWS App Mesh mesh. This is the service mesh that all services will reside in.
mesh = aws.appmesh.Mesh("aiMesh")

# Define a stable version of the AI service as a virtual node.
stable_virtual_node = aws.appmesh.VirtualNode("stableVirtualNode",
    mesh_name=mesh.name,
    spec=aws.appmesh.VirtualNodeSpecArgs(
        service_discovery=aws.appmesh.VirtualNodeSpecServiceDiscoveryArgs(
            aws_cloud_map=aws.appmesh.VirtualNodeSpecServiceDiscoveryAwsCloudMapArgs(
                namespace_name="my-namespace",
                service_name="stable-service",
            ),
        ),
    ),
)

# Define a new canary version of the AI service as a virtual node.
canary_virtual_node = aws.appmesh.VirtualNode("canaryVirtualNode",
    mesh_name=mesh.name,
    spec=aws.appmesh.VirtualNodeSpecArgs(
        service_discovery=aws.appmesh.VirtualNodeSpecServiceDiscoveryArgs(
            aws_cloud_map=aws.appmesh.VirtualNodeSpecServiceDiscoveryAwsCloudMapArgs(
                namespace_name="my-namespace",
                service_name="canary-service",
            ),
        ),
    ),
)

# Define a virtual service that the AI clients will point to.
virtual_service = aws.appmesh.VirtualService("aiVirtualService",
    mesh_name=mesh.name,
    spec=aws.appmesh.VirtualServiceSpecArgs(
        provider=aws.appmesh.VirtualServiceSpecProviderArgs(
            virtual_router=aws.appmesh.VirtualServiceSpecProviderVirtualRouterArgs(
                virtual_router_name="aiVirtualRouter",
            ),
        ),
    ),
)

# Define a virtual router to route traffic between stable and canary versions of the AI service.
virtual_router = aws.appmesh.VirtualRouter("aiVirtualRouter",
    mesh_name=mesh.name,
    spec=aws.appmesh.VirtualRouterSpecArgs(
        listeners=[aws.appmesh.VirtualRouterSpecListenerArgs(
            port_mapping=aws.appmesh.VirtualRouterSpecListenerPortMappingArgs(
                port=8080,
                protocol="http",
            ),
        )],
    ),
)

# Define routes for directing traffic to either the stable or canary virtual node.
route = aws.appmesh.Route("aiRoute",
    mesh_name=mesh.name,
    virtual_router_name=virtual_router.name,
    spec=aws.appmesh.RouteSpecArgs(
        http_route=aws.appmesh.RouteSpecHttpRouteArgs(
            match=aws.appmesh.RouteSpecHttpRouteMatchArgs(
                prefix="/",
            ),
            action=aws.appmesh.RouteSpecHttpRouteActionArgs(
                weighted_targets=[
                    aws.appmesh.RouteSpecHttpRouteActionWeightedTargetArgs(
                        virtual_node=stable_virtual_node.name,
                        weight=90,
                    ),
                    aws.appmesh.RouteSpecHttpRouteActionWeightedTargetArgs(
                        virtual_node=canary_virtual_node.name,
                        weight=10,
                    ),
                ],
            ),
        ),
    ),
)

# Exporting relevant outputs so they can be used elsewhere as needed.
pulumi.export("mesh_name", mesh.name)
pulumi.export("stable_virtual_node_name", stable_virtual_node.name)
pulumi.export("canary_virtual_node_name", canary_virtual_node.name)
pulumi.export("virtual_service_name", virtual_service.name)
```

In the above code:

- We created a mesh named `aiMesh`.
- We set up two virtual nodes to represent the stable and canary version of our AI service.
- We then defined a virtual service that acts as an abstraction and interface for these services. This is the endpoint your application clients will call.
- We created a virtual router that will decide whether to route incoming requests to the stable node or the canary node based on the weights we define (90% stable, 10% canary in this case).
- A route provides the defined behavior of sending traffic to designated virtual nodes.
- Finally, the names of our mesh and virtual nodes are exported as outputs, which can be helpful for dynamically retrieving these values in other parts of our Pulumi program or for use in other tools and scripts.

This setup allows us to safely test our canary deployment by exposing it to a subset of our live traffic. If the canary version behaves as expected, we can proceed to shift more traffic over time until it completely replaces the stable version. If issues arise, we can quickly revert back to the stable service with minimal impact on our users.