Deploying Kafka brokers on EKS for scalable message streaming

Question

Pulumi · Accepted Answer

Deploying Kafka on Amazon EKS (Elastic Kubernetes Service) involves several steps, including setting up an EKS cluster, configuring networking, creating IAM roles, and then deploying Kafka brokers. Pulumi enables the creation and management of these resources through code, making the process repeatable and predictable.

Here, I'll walk you through building an EKS cluster and deploying Kafka brokers using Pulumi. You'll need to have Pulumi CLI installed and configured to work with AWS.

First, we'll create an EKS cluster using the `eks` package, which provides a high-level abstraction over EKS-related details, making the cluster setup easier. We'll also use the `aws` package to set up necessary IAM Roles for EKS and a VPC CNI plugin for networking.

For Kafka, we will use a Helm chart to simplify the Kafka deployment process on Kubernetes. Pulumi supports deploying Helm charts directly within the infrastructure code. We will also need a VPC and subnets set up for the EKS cluster, which Pulumi can help us create or select from existing resources.

Let's start with the Pulumi program in TypeScript:

```typescript
import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as eks from "@pulumi/eks";
import * as k8s from "@pulumi/kubernetes";
import * as awsx from "@pulumi/awsx";

// Create a VPC for our cluster
const vpc = new awsx.ec2.Vpc("kafka-vpc", {
    numberOfAvailabilityZones: 2,
});

// Create an EKS cluster with the default configuration
const cluster = new eks.Cluster("kafka-cluster", {
    vpcId: vpc.id,
    subnetIds: vpc.privateSubnetIds,
    instanceType: "t3.medium", // Choose a suitable instance type
    desiredCapacity: 3, // Set the desired number of worker nodes
    minSize: 1, // Set the minimum size for auto-scaling
    maxSize: 4, // Set the maximum size for auto-scaling
    storageClasses: "gp2", // General Purpose SSD volume type
    deployDashboard: false, // Kubernetes dashboard is optional
});

// Export the cluster's kubeconfig
export const kubeconfig = cluster.kubeconfig;

// Create a Kubernetes provider instance using the EKS cluster's kubeconfig
const provider = new k8s.Provider("k8s-provider", {
    kubeconfig: cluster.kubeconfig.apply(JSON.stringify),
});

// Deploy Kafka using the Bitnami Helm chart
const kafkaChart = new k8s.helm.v3.Chart("kafka", {
    chart: "kafka",
    version: "12.7.1", // Specify the version of the Helm chart
    fetchOpts: {
        repo: "https://charts.bitnami.com/bitnami",
    },
    values: {
        replicationFactor: 3, // Set the replication factor for Kafka brokers
        // Additional configuration parameters can be set here
    },
}, { provider });

// Export the Kafka broker service endpoint
const kafkaService = kafkaChart.getResource("v1/Service", "kafka-kafka-headless");
export const kafkaEndpoints = kafkaService.status.apply(status => status.loadBalancer.ingress.map(ingress => ingress.ip));

```

In the above program, we start by creating a new VPC with the `awsx.ec2.Vpc` resource. We then create an EKS cluster with the `eks.Cluster` resource, specifying the VPC and subnets where it should be provisioned, as well as the desired size and type of the nodes.

After the EKS cluster is created, we define a `k8s.Provider` resource that uses the kubeconfig of the created EKS cluster. This provider is used to interface with the Kubernetes cluster and deploy resources on it.

We then proceed to deploy Kafka using a Helm chart by creating a `k8s.helm.v3.Chart` resource. The `values` field within the chart configuration allows us to set custom parameters, like the replication factor for Kafka brokers. These values should be adjusted according to your use case.

Lastly, the program exports the kubeconfig of the cluster and the endpoints of the Kafka brokers. The `kubeconfig` is needed to interact with your Kubernetes cluster, and the `kafkaEndpoints` might be used by your applications or services to interact with Kafka brokers.

To launch and manage this stack, you would save this code in a file, for instance `index.ts`, run `pulumi up` to preview and deploy the changes, and `pulumi destroy` when you want to tear down the resources. You can adjust the values to fit the requirements of your workload, like changing the instance types, replication factors, and so on.