Implementing canary releases using Kubernetes upgrade plans

Question

Pulumi · Accepted Answer

Canary releases are a deployment strategy that allows you to test a new version of your software by gradually rolling it out to a small subset of users before making it available to everybody. This strategy helps detect any issues in the new release early in the deployment process, reducing the impact of any potential problems.

Using Kubernetes and Pulumi, you can implement canary releases by updating the Deployment resources and controlling the rollout using Kubernetes' built-in mechanisms to manage the versions and replicas.

Below is an example of how to use Pulumi with TypeScript to implement a canary release strategy in Kubernetes:

1. We will create two deployments: one for the stable release and one for the canary release.
2. The stable release will have most of the traffic, while the canary deployment will start with a smaller number of replicas.
3. We will use the `matchLabels` selector to differentiate between stable and canary pods.
4. We'll manage the rollout incrementally by adjusting the number of replicas of the canary deployment based on observed behavior.
5. To automate the process, we'll use Pulumi to keep track of the state of our deployments and ensure they are updated correctly.

Let's start with the Pulumi program written in TypeScript to achieve the above steps.

```typescript
import * as k8s from "@pulumi/kubernetes";

// Define stable deployment.
const stableDeployment = new k8s.apps.v1.Deployment("stable-deployment", {
    spec: {
        replicas: 5, // Number of replicas for the stable version
        selector: { matchLabels: { app: "your-app", release: "stable" } },
        template: {
            metadata: { labels: { app: "your-app", release: "stable" } },
            spec: { containers: [{ name: "your-app", image: "your-app:v1.0.0" }] },
        },
        strategy: {
            rollingUpdate: { maxSurge: 1, maxUnavailable: 1 }, // Rolling update config
        },
    },
});

// Define canary deployment.
const canaryDeployment = new k8s.apps.v1.Deployment("canary-deployment", {
    spec: {
        replicas: 1, // Start with a smaller replica for canary
        selector: { matchLabels: { app: "your-app", release: "canary" } },
        template: {
            metadata: { labels: { app: "your-app", release: "canary" } },
            spec: { containers: [{ name: "your-app", image: "your-app:v1.1.0-canary" }] },
        },
    },
});

// Export the Deployment names
export const stableDeploymentName = stableDeployment.metadata.name;
export const canaryDeploymentName = canaryDeployment.metadata.name;

// To perform a canary release, start by deploying this stack. Then, after observing the canary behavior,
// you can adjust the `replicas` of the canaryDeployment resource to increase or decrease the proportion
// of traffic it receives. Once you're satisfied with the canary release, you can update the stableDeployment
// to use the new image version and adjust its replicas as necessary.
```

In this program, we define two `Deployment` resources:
- `stableDeployment` for the current stable release of your application. It's configured to run 5 replicas with the `app:your-app` and `release:stable` labels, and to use the `your-app:v1.0.0` image.
- `canaryDeployment` for the canary release. It starts with just 1 replica and uses the `your-app:v1.1.0-canary` image which represents the new version you want to test.

Both deployments define a rolling update strategy with `maxSurge` and `maxUnavailable` parameters to control the rollout process. The `maxSurge` indicates how many pods we can add at a time, and `maxUnavailable` specifies the number of pods that can be unavailable during the update.

After deploying this code with Pulumi, here's how you can proceed with the canary release:
- Monitor the canary deployment's behavior. If it performs well, you can gradually increase the number of replicas for the canary deployment, directing more traffic to the new version.
- If at any point the canary version seems to have issues, you can immediately roll back by reducing or removing the canary replicas without affecting all users.
- Once you're confident that the canary version is stable, you can start updating the stable deployment with the new version (`your-app:v1.1.0`) and increase its replicas accordingly.

To apply the changes, you would modify the Pulumi code to reflect the desired state for your replicas and image versions, and then run `pulumi up` to update the resources. Pulumi tracks the desired and actual state, making sure the Kubernetes resources match your definitions in code.