Scalable AI Pipelines on EKS with Kafka as Message Queue

Question

Pulumi · Accepted Answer

To create scalable AI pipelines on Amazon EKS (Elastic Kubernetes Service) with Apache Kafka as a message queue, you will need to:

Provision an EKS cluster: This is the managed Kubernetes service that will run your containerized AI workloads.
Set up Kafka: Kafka will act as a distributed streaming platform which can be used as a message queue for your pipelines. Kafka can either be set up within the Kubernetes cluster or as a separate managed service which your Kubernetes services can interact with.

For this setup, we will:

Create an EKS Cluster using Pulumi's AWS Native provider.
Create a Kafka service using Pulumi's Aiven provider, which offers Kafka as a managed service.
Configure an instance of Kafka Topic, to define how messages are stored and accessed.

Here's a Pulumi program written in Python that illustrates these steps:

import pulumi
import pulumi_aws_native as aws_native
import pulumi_aiven as aiven

# Create an AWS EKS Cluster
eks_cluster = aws_native.eks.Cluster("eksCluster",
    role_arn="arn:aws:iam::<account-id>:role/<EKS-service-role>",  # Replace <account-id> and <EKS-service-role> with appropriate values
    resources_vpc_config=aws_native.eks.ClusterResourcesVpcConfigArgs(
        subnet_ids=["subnet-12345678", "subnet-87654321"],  # Replace with your VPC subnets
    ),
    version="1.21",  # Replace with your desired Kubernetes version
)

# Create Kafka on Aiven
kafka_service = aiven.Kafka("kafkaService",
    plan="business-4",  # Choose the appropriate plan for your needs
    service_name="kafka-service",
    project="<your-aiven-project-name>",  # Replace with your Aiven project name
    kafka_user_config=aiven.KafkaUserConfigArgs(
        kafka=aiven.KafkaUserConfigKafkaArgs(
            kafka_version="2.8",  # Specify Kafka version
            kafka_rest=True,
            public_access=aiven.KafkaUserConfigPublicAccessArgs(
                prometheus=True,
            ),
        ),
    ),
)

# Define Kafka Topics
kafka_topic = aiven.KafkaTopic("kafkaTopic",
    topic_name="ai-events",
    partitions=3,
    replication=2,
    project="<your-aiven-project-name>",  # Replace with your Aiven project name
    service_name="kafka-service",
)

# Export the cluster name and Kafka service URI
pulumi.export("eks_cluster_name", eks_cluster.name)
pulumi.export("kafka_service_uri", kafka_service.service_uri.apply(lambda uri: uri))

Let's break down what the code does:

EKS Cluster: This section of the code sets up an EKS cluster with the required VPC configurations and IAM role. You need to provide your AWS account ID, VPC subnet IDs, and create an IAM role for EKS with the necessary permissions. You also select the Kubernetes version you wish to use.
Kafka Service: Here, we create a Kafka service instance on Aiven, which is a managed cloud service that makes it easy to set up, manage, and scale Kafka clusters. You need to select a plan and provide your Aiven project name. We've also configured the Kafka's version and enabled Prometheus public access for monitoring.
Kafka Topic: This resource defines a topic in the Kafka service that will store the messages or events. You can set the number of partitions and replication factor based on your scalability and redundancy needs.
Exports: At the end of the program, we export the cluster name and Kafka service URI so that you can use them in other parts of your infrastructure or application.

Remember to replace placeholders in angle brackets with actual values corresponding to your AWS and Aiven configurations. This Pulumi program assumes you have already configured both AWS and Aiven providers. If not, follow the respective configuration guides for AWS and Aiven.