Using kubernetes kafka.strimzi.io with security.istio.io

Question

Pulumi · Accepted Answer

Integrating Kafka with Istio on Kubernetes involves setting up a Kafka cluster managed by the Strimzi operator and configuring Istio to manage network communication for security, including enabling mTLS (mutual Transport Layer Security) for secure communication between Kafka brokers and clients.

The Strimzi Kafka operator simplifies the process of running Apache Kafka in Kubernetes. It provides a method to easily deploy and manage Kafka clusters as well as monitor and adjust their configuration.

On the other hand, Istio is a service mesh that provides a way to control how microservices share data with one another. Within a Kubernetes cluster, it can be used to secure communication between pods, inject policies, and aggregate telemetry data. Istio's Custom Resource Definitions (CRDs), such as `VirtualServices` and `DestinationRules`, are commonly used to manage these configurations.

For the purpose of this setup, we will:
- Deploy a Kafka cluster using the Strimzi operator.
- Configure Istio to manage Kafka pods and enable mTLS.
- Use Kubernetes resources to enforce communication policies.

Below is a sample Pulumi TypeScript program that demonstrates how you could set up Kafka with Strimzi including Istio security configurations in a Kubernetes cluster using Pulumi. The program assumes that you've already installed the Strimzi and Istio operators in your Kubernetes cluster.

```typescript
import * as k8s from "@pulumi/kubernetes";

// Use Strimzi to create a KafkaCluster CRD.
const kafkaCluster = new k8s.apiextensions.CustomResource("kafka-cluster", {
    apiVersion: "kafka.strimzi.io/v1beta2",
    kind: "Kafka",
    metadata: {
        namespace: "kafka",
        name: "my-cluster",
    },
    spec: {
        kafka: {
            version: "2.7.0",
            replicas: 3,
            listeners: [
                {
                    name: "tls",
                    port: 9093,
                    type: "internal",
                    tls: true,
                    authentication: { type: "tls" },
                },
                // Add other listeners as needed.
            ],
            config: {
                "offsets.topic.replication.factor": "3",
                "transaction.state.log.replication.factor": "3",
                "transaction.state.log.min.isr": "2",
                "log.message.format.version": "2.7",
                // Add other configurations as needed.
            },
            storage: {
                type: "jbod",
                volumes: [
                    {
                        id: 0,
                        type: "persistent-claim",
                        size: "100Gi",
                        deleteClaim: false,
                    },
                ],
            },
        },
        zookeeper: {
            replicas: 3,
            storage: {
                type: "persistent-claim",
                size: "100Gi",
                deleteClaim: false,
            },
        },
        entityOperator: {
            topicOperator: {},
            userOperator: {},
        },
    },
});

// Use Istio to set a PeerAuthentication policy to enforce mTLS for the Kafka namespace.
const peerAuthentication = new k8s.apiextensions.CustomResource("kafka-mtls-policy", {
    apiVersion: "security.istio.io/v1beta1",
    kind: "PeerAuthentication",
    metadata: {
        namespace: "kafka",
        name: "default",
    },
    spec: {
        mtls: {
            mode: "STRICT",
        },
    },
}, { dependsOn: [kafkaCluster] });

// Export the Kafka cluster name.
export const kafkaClusterName = kafkaCluster.metadata.name;
```

This program defines two custom resources: `Kafka` and `PeerAuthentication`.
- The `Kafka` custom resource creates a Kafka cluster managed by the Strimzi operator. The configuration specifies the Kafka version, the number of Kafka and Zookeeper replicas, storage requirements, and listeners for communication. In this case, we are using a TLS listener for secure communication within the Kubernetes cluster.
- The `PeerAuthentication` custom resource enforces mTLS for the Kafka cluster's namespace using Istio's security features. This ensures that all the communication between the Kafka brokers themselves and the Kafka clients is encrypted and secure.

Please make sure you have the Strimzi Kafka operator and Istio installed on your Kubernetes cluster to recognize and act upon these custom resources. Without these operators, the above resources will not be functional. You can install them following the instructions on their respective official documentation:
- [Strimzi Kafka Operator](https://strimzi.io/docs/operators/latest/full/deploying.html#deploying-cluster-operator-kubernetes-str)
- [Istio Service Mesh](https://istio.io/latest/docs/setup/install/)

After installing Pulumi and configuring it to access your Kubernetes cluster, you can run `pulumi up` to start creating this configuration in your cluster.