How do I deploy Strimzi Kafka on Kubernetes with Fluent Bit for log processing?
In this guide, we will deploy Strimzi Kafka on a Kubernetes cluster and set up Fluent Bit for log processing. Strimzi provides a way to run Apache Kafka on Kubernetes, while Fluent Bit is a lightweight log processor and forwarder.
We will use Pulumi to define and deploy the necessary Kubernetes resources. The resources include Kafka and Zookeeper clusters managed by Strimzi, and a Fluent Bit DaemonSet to collect and forward logs.
import * as pulumi from "@pulumi/pulumi";
import * as k8s from "@pulumi/kubernetes";
// Create a namespace for Kafka and Fluent Bit
const namespace = new k8s.core.v1.Namespace("logging", {
metadata: {
name: "logging",
},
});
// Deploy the Strimzi Kafka operator
const strimziOperator = new k8s.helm.v3.Chart("strimzi", {
chart: "strimzi-kafka-operator",
version: "0.25.0",
fetchOpts: {
repo: "https://strimzi.io/charts/",
},
namespace: namespace.metadata.name,
});
// Define a Kafka cluster
const kafkaCluster = new k8s.apiextensions.CustomResource("kafkaCluster", {
apiVersion: "kafka.strimzi.io/v1beta2",
kind: "Kafka",
metadata: {
name: "my-cluster",
namespace: namespace.metadata.name,
},
spec: {
kafka: {
version: "2.8.0",
replicas: 3,
listeners: [
{
name: "plain",
port: 9092,
type: "internal",
tls: false,
},
{
name: "tls",
port: 9093,
type: "internal",
tls: true,
},
],
config: {
"offsets.topic.replication.factor": 3,
"transaction.state.log.replication.factor": 3,
"transaction.state.log.min.isr": 2,
"log.message.format.version": "2.8",
},
storage: {
type: "jbod",
volumes: [
{
id: 0,
type: "persistent-claim",
size: "100Gi",
deleteClaim: false,
},
],
},
},
zookeeper: {
replicas: 3,
storage: {
type: "persistent-claim",
size: "100Gi",
deleteClaim: false,
},
},
entityOperator: {
topicOperator: {},
userOperator: {},
},
},
});
// Deploy Fluent Bit as a DaemonSet
const fluentBitConfigMap = new k8s.core.v1.ConfigMap("fluent-bit-config", {
metadata: {
namespace: namespace.metadata.name,
name: "fluent-bit-config",
},
data: {
"fluent-bit.conf": `
[SERVICE]
Flush 1
Log_Level info
[INPUT]
Name tail
Path /var/log/containers/*.log
Parser docker
Tag kube.*
[FILTER]
Name kubernetes
Match kube.*
Kube_URL https://kubernetes.default.svc:443
Merge_Log On
K8S-Logging.Parser On
K8S-Logging.Exclude Off
[OUTPUT]
Name kafka
Match *
Brokers my-cluster-kafka-bootstrap.logging:9092
Topics logs
Format json
Timestamp_Key @timestamp
Timestamp_Format iso8601
`,
},
});
const fluentBitDaemonSet = new k8s.apps.v1.DaemonSet("fluent-bit", {
metadata: {
namespace: namespace.metadata.name,
name: "fluent-bit",
},
spec: {
selector: {
matchLabels: {
"app.kubernetes.io/name": "fluent-bit",
},
},
template: {
metadata: {
labels: {
"app.kubernetes.io/name": "fluent-bit",
},
},
spec: {
containers: [
{
name: "fluent-bit",
image: "fluent/fluent-bit:1.8.3",
volumeMounts: [
{
name: "varlog",
mountPath: "/var/log",
},
{
name: "config",
mountPath: "/fluent-bit/etc/fluent-bit.conf",
subPath: "fluent-bit.conf",
},
],
},
],
volumes: [
{
name: "varlog",
hostPath: {
path: "/var/log",
},
},
{
name: "config",
configMap: {
name: "fluent-bit-config",
},
},
],
},
},
},
});
Key Points
- We created a namespace named
logging
to isolate our Kafka and Fluent Bit resources. - We deployed the Strimzi Kafka operator using a Helm chart and defined a Kafka cluster with Zookeeper.
- We set up Fluent Bit as a DaemonSet to collect logs from Kubernetes nodes and forward them to our Kafka cluster.
Summary
In this guide, we demonstrated how to deploy Strimzi Kafka on a Kubernetes cluster and set up Fluent Bit for log processing using Pulumi. We used a Helm chart to deploy the Strimzi Kafka operator, defined a Kafka cluster, and configured Fluent Bit to collect and forward logs to Kafka.
Deploy this code
Want to deploy this code? Sign up for a free Pulumi account to deploy in a few clicks.
Sign upNew to Pulumi?
Want to deploy this code? Sign up with Pulumi to deploy in a few clicks.
Sign upThank you for your feedback!
If you have a question about how to use Pulumi, reach out in Community Slack.
Open an issue on GitHub to report a problem or suggest an improvement.