Scaling Azure Kusto clusters dynamically based on load

Question

Pulumi · Accepted Answer

To scale an Azure Kusto cluster (also known as Azure Data Explorer) dynamically based on load, we will use the Pulumi `azure-native` package, specifically the `Cluster` and `OptimizedAutoscale` resources.

Azure Kusto clusters can be configured to automatically scale based on the load by enabling the optimized autoscale feature. This feature allows the cluster to scale in or out within a specified range of instances. It adjusts the number of instances according to the rules set up based on metrics like CPU, memory consumption, or other performance indicators.

The `OptimizedAutoscale` property is a part of the `Cluster` resource, where you define the parameters for autoscaling, such as the minimum and maximum count of instances, whether the feature is enabled, and the version of the autoscaler.

Let's write a Pulumi program in TypeScript to define an Azure Kusto cluster with autoscaling enabled. The program will:

- Define the Kusto cluster with desired starting configurations.
- Set up the optimized autoscale settings that define the minimum and maximum number of instances, and enable autoscaling.

Please make sure you have the appropriate Azure credentials configured on the machine where you're running Pulumi.

Here is the Pulumi program:

```typescript
import * as pulumi from "@pulumi/pulumi";
import * as azure_native from "@pulumi/azure-native";

const resourceGroup = new azure_native.resources.ResourceGroup("resourceGroup");

const kustoCluster = new azure_native.kusto.Cluster("kustoCluster", {
    resourceGroupName: resourceGroup.name,
    location: resourceGroup.location,
    sku: {
        name: "Standard_D13_v2",
        tier: "Standard",
        capacity: 2 // Initial capacity. Autoscaling will adjust this as needed.
    },
    // The optimizedAutoscale setting allows the cluster to scale dynamically based on load.
    optimizedAutoscale: {
        version: 1, // The autoscale settings version. This is required and should be set to 1.
        isEnabled: true, // Enabling autoscale.
        minimum: 2, // Minimum number of instances to scale down to.
        maximum: 10, // Maximum number of instances to scale up to.
    },
    // Optional settings can be configured here, such as tags, zones, etc.
});

// Export the Kusto Cluster name and the Azure Resource Group name
// This allows us to easily identify the Kusto Cluster in the Azure portal or via the Azure CLI
export const kustoClusterName = kustoCluster.name;
export const azureResourceGroupName = resourceGroup.name;
```

In this program:

- We create a resource group that the Kusto cluster will reside in.
- We then create the Kusto cluster with a specified SKU (change this to match your desired performance and cost constraints).
- We set the initial capacity, which is the number of instances that the cluster starts with.
- We configure the `optimizedAutoscale` property with `isEnabled` set to `true` to turn on autoscaling. The minimum and maximum values define the scaling range for the instances.
- The `export` statements at the end of the program allow you to retrieve the name of the Kusto cluster and the resource group after deployment, which is useful for further referencing in the Azure portal or other tools.

To run this Pulumi program, save it to a file (e.g., `index.ts`), then execute it using the Pulumi CLI:

```sh
pulumi up
```

This command will provision the resources as defined in your TypeScript program. If you need to make any changes to the configuration, update your Pulumi program, and then run `pulumi up` again to apply the changes.

Remember to monitor your Kusto cluster's performance and cost after enabling autoscaling to ensure it meets your application requirements and stays within your budget.