Using kubernetes kafka.strimzi.io with access

Question

Pulumi · Accepted Answer

To use `kafka.strimzi.io` on Kubernetes with Pulumi, you'll first need to have a Kubernetes cluster set up and configured with `kubectl`. Strimzi provides container images and Operators for running Apache Kafka on Kubernetes. The Strimzi Kafka Operator simplifies the process of running Apache Kafka on Kubernetes by automating common tasks involved in managing a Kafka cluster.

The Pulumi program would involve the following steps:
1. Installing the Strimzi Kafka Operator on your Kubernetes cluster.
2. Deploying a Kafka cluster using the Custom Resource Definitions provided by Strimzi.
3. Configuring Kafka access and user permissions within the cluster.

Below is a TypeScript program using Pulumi which automates these steps. It assumes that you have a running Kubernetes cluster and your `kubeconfig` is set up correctly. The program uses the Pulumi Kubernetes provider to deploy resources.

```typescript
import * as k8s from "@pulumi/kubernetes";

// 1. Install the Strimzi Kafka Operator.
// This could be done by applying the YAML manifests provided by Strimzi,
// which usually involves creating several resources such as ClusterRoles,
// ClusterRoleBindings, and Custom Resource Definitions (CRDs) for the Kafka resources.

const strimziOperatorUrl = "https://strimzi.io/install/latest?namespace=kafka";

const strimziOperator = new k8s.yaml.ConfigGroup("strimzi-operator", {
    files: [strimziOperatorUrl],
});

// 2. Deploy a Kafka cluster using Strimzi's Custom Resource Definitions.
// You can define your Kafka cluster, including its configuration and the necessary
// amount of brokers.

const kafkaCluster = new k8s.yaml.ConfigGroup("kafka-cluster", {
    files: ["strimzi-kafka-cluster.yaml"],
}, { dependsOn: strimziOperator });

// An example Kafka cluster YAML could look like this:
// strimzi-kafka-cluster.yaml
/*
apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
  name: my-cluster
  namespace: kafka
spec:
  kafka:
    version: 2.6.0
    replicas: 3
    listeners:
      plain: {}
      tls: {}
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      log.message.format.version: "2.6"
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 100Gi
        deleteClaim: false
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 100Gi
      deleteClaim: false
  entityOperator:
    topicOperator: {}
    userOperator: {}
*/

// 3. Configure Kafka User and Access.
// After deploying your Kafka cluster, you can configure Kafka user and ACLs using Strimzi's User Custom Resource
// to manage authentication and authorization for Kafka clients.

const kafkaUser = new k8s.yaml.ConfigGroup("kafka-user", {
    files: ["strimzi-kafka-user.yaml"],
}, { dependsOn: kafkaCluster });

// An example Kafka user YAML:
// strimzi-kafka-user.yaml
/*
apiVersion: kafka.strimzi.io/v1beta1
kind: KafkaUser
metadata:
  name: my-user
  namespace: kafka
  labels:
    strimzi.io/cluster: my-cluster
spec:
  authentication:
    type: tls
  authorization:
    type: simple
    acls:
      - resource:
          type: topic
          name: my-topic
          patternType: literal
        operation: Read
        host: "*"
*/

// Export the URLs to the Kafka brokers.
export const kafkaBrokers = kafkaCluster.getResource("v1/Service", "kafka", "my-cluster-kafka-external-bootstrap").status.loadBalancer.ingress[0].hostname;

```

This program does the following things:
- It installs the Strimzi Kafka Operator on your Kubernetes cluster by applying the installation YAML from the Strimzi website. This sets up the necessary roles, role bindings, and CRDs needed to manage Kafka clusters in Kubernetes.
- It deploys a Kafka cluster as specified in a YAML file. This would include the Kafka version, the number of brokers, storage configuration, and any other Kafka configurations you want to apply.
- It creates a Kafka user with access to specific Kafka topics using the KafkaUser resource. Access control can be adjusted based on the requirements you have for topic access and user permissions.

Remember to replace the `strimzi-kafka-cluster.yaml` and `strimzi-kafka-user.yaml` with your own configurations according to your cluster's requirements. After applying this program using Pulumi, you will have a Kafka cluster running on Kubernetes managed by Strimzi, along with a user configured with access to Kafka resources.