Zookeeper for Kubernetes Service Coordination.

Question

Pulumi · Accepted Answer

To set up Zookeeper for service coordination in a Kubernetes cluster with Pulumi, we need to follow a few steps:

1. **Define the Kubernetes Service**: We need a Kubernetes Service that points to the Zookeeper pods. We can configure the type of service based on our requirements, for example as a `ClusterIP` for internal communication or a `NodePort` for external access.

2. **Deploy Zookeeper StatefulSet**: Zookeeper requires stable identity and storage for each of its instances. A StatefulSet in Kubernetes is the perfect way to define this, as it creates a set of Pods with unique, persistent identities and stable persistent storage.

3. **Persistent Volumes**: Zookeeper will need to store data persistently. Kubernetes PersistentVolumes (PVs) can be used to provision storage for these requirements.

4. **Create a ConfigMap or use environment variables**: To set up Zookeeper-specific configurations, we can use a Kubernetes ConfigMap or define them as environment variables.

Here's a Pulumi program that sets up Zookeeper using these components in a Kubernetes cluster:

```python
import pulumi
import pulumi_kubernetes as k8s

# Define the namespace for Zookeeper
zookeeper_namespace = k8s.core.v1.Namespace("zookeeper-namespace",
    metadata={
        "name": "zookeeper",
    })

# Define a headless service for Zookeeper to create a DNS entry
# Headless services are required for StatefulSets so that each pod gets a unique DNS entry
zookeeper_service = k8s.core.v1.Service("zookeeper-service",
    metadata=k8s.meta.v1.ObjectMetaArgs(
        name="zookeeper",
        namespace=zookeeper_namespace.metadata["name"],
        labels={
            "app": "zookeeper",
        },
    ),
    spec=k8s.core.v1.ServiceSpecArgs(
        ports=[k8s.core.v1.ServicePortArgs(
            port=2181,
            name="client",
        )],
        cluster_ip="None",  # This makes the service "headless"
        selector={
            "app": "zookeeper",
        },
    ))

# Define the StatefulSet for Zookeeper
zookeeper_stateful_set = k8s.apps.v1.StatefulSet("zookeeper-statefulset",
    metadata=k8s.meta.v1.ObjectMetaArgs(
        name="zookeeper",
        namespace=zookeeper_namespace.metadata["name"],
    ),
    spec=k8s.apps.v1.StatefulSetSpecArgs(
        selector=k8s.meta.v1.LabelSelectorArgs(
            match_labels={
                "app": "zookeeper",
            },
        ),
        service_name=zookeeper_service.metadata["name"],
        replicas=3,
        template=k8s.core.v1.PodTemplateSpecArgs(
            metadata=k8s.meta.v1.ObjectMetaArgs(
                labels={
                    "app": "zookeeper",
                },
            ),
            spec=k8s.core.v1.PodSpecArgs(
                containers=[
                    k8s.core.v1.ContainerArgs(
                        name="zookeeper",
                        image="zookeeper:3.6.1",
                        ports=[
                            k8s.core.v1.ContainerPortArgs(
                                container_port=2181,
                                name="client",
                            ),
                        ],
                        volume_mounts=[
                            k8s.core.v1.VolumeMountArgs(
                                mount_path="/data",
                                name="data",
                            ),
                        ],
                    ),
                ],
            ),
        ),
        volume_claim_templates=[
            k8s.core.v1.PersistentVolumeClaimArgs(
                metadata=k8s.meta.v1.ObjectMetaArgs(
                    name="data",
                ),
                spec=k8s.core.v1.PersistentVolumeClaimSpecArgs(
                    access_modes=["ReadWriteOnce"],
                    resources=k8s.core.v1.ResourceRequirementsArgs(
                        requests={
                            "storage": "2Gi",
                        },
                    ),
                ),
            ),
        ],
    ))

# Export the Zookeeper Namespace name
pulumi.export('zookeeper_namespace', zookeeper_namespace.metadata["name"])
# Export the Zookeeper Service name
pulumi.export('zookeeper_service', zookeeper_service.metadata["name"])
```

Here is what the program is doing:

- It first creates a `Namespace` for Zookeeper to keep it isolated from other services.
- Next, it creates a `Service` with `cluster_ip` set to `None` to make it a headless service, necessary for the StatefulSet pods to have their unique DNS.
- Then, we define the core of the Zookeeper cluster, the `StatefulSet` which has 3 replicas, meaning 3 Zookeeper pods would be created, each with a stable network identifier and storage.
- The `volume_claim_templates` is a list of `PersistentVolumeClaim` resources that Zookeeper pods use to request physical storage.

With this setup, a Zookeeper service is created with stable identity and persistent storage, suitable for coordination tasks in your Kubernetes environment. You can apply this Pulumi program to deploy the service to your cluster.