Kubernetes Persistent Volumes for Dataset Storage

Question

Pulumi · Accepted Answer

Creating persistent storage in Kubernetes involves defining Persistent Volume (PV) and Persistent Volume Claim (PVC) resources. A PV is a piece of storage in the cluster that has been provisioned by an administrator or dynamically provisioned using Storage Classes. A PVC is a request for storage by a user. It is similar to a pod; pods consume node resources and PVCs consume PV resources.

Here's a breakdown of the process:

1. **Define a Persistent Volume (PV)**: This is where you define a volume that is available for use. It can be hosted on a variety of supported storage backends and configured with specific capabilities (such as size, access modes, etc.).

2. **Create a Persistent Volume Claim (PVC)**: This is a request for storage by a user. PVCs can request specific size and access modes (like a read-only or read-write volume).

3. **Mount the PVC in a Pod**: Once the PVC is bound to a PV, it can be mounted into a pod for use by an application.

For example, let's create a Kubernetes Persistent Volume backed by NFS (Network File System), and a Persistent Volume Claim to claim that storage for use. You'll also write a pod to show how to use the PVC. In this example, we are assuming that you have an existing NFS server with an IP address of `10.10.10.20` and a path `/dataset`.

Let's write the Pulumi program in Python to create the above resources:

```python
import pulumi
import pulumi_kubernetes as k8s

# Create a Kubernetes Persistent Volume
nfs_persistent_volume = k8s.core.v1.PersistentVolume("nfsPersistentVolume",
    metadata=k8s.meta.v1.ObjectMetaArgs(
        name="nfs-pv",
    ),
    spec=k8s.core.v1.PersistentVolumeSpecArgs(
        capacity={"storage": "5Gi"},
        access_modes=["ReadWriteMany"],
        nfs=k8s.core.v1.NFSVolumeSourceArgs(
            server="10.10.10.20",
            path="/dataset"
        )
    )
)

# Create a Persistent Volume Claim
nfs_persistent_volume_claim = k8s.core.v1.PersistentVolumeClaim("nfsPersistentVolumeClaim",
    metadata=k8s.meta.v1.ObjectMetaArgs(
        name="nfs-pvc",
    ),
    spec=k8s.core.v1.PersistentVolumeClaimSpecArgs(
        access_modes=["ReadWriteMany"],
        resources=k8s.core.v1.ResourceRequirementsArgs(
            requests={"storage": "5Gi"}
        )
    )
)

# Create a Pod that uses the PVC
nfs_pod = k8s.core.v1.Pod("nfsPod",
    metadata=k8s.meta.v1.ObjectMetaArgs(
        name="nfs-pod",
    ),
    spec=k8s.core.v1.PodSpecArgs(
        volumes=[k8s.core.v1.VolumeArgs(
            name="nfs-storage",
            persistent_volume_claim=k8s.core.v1.PersistentVolumeClaimVolumeSourceArgs(
                claim_name=nfs_persistent_volume_claim.metadata.name,
            ),
        )],
        containers=[k8s.core.v1.ContainerArgs(
            name="alpine",
            image="alpine:latest",
            command=["sh", "-c"],
            args=["while true; do echo hello; sleep 10;done"],
            volume_mounts=[k8s.core.v1.VolumeMountArgs(
                name="nfs-storage",
                mount_path="/mnt/dataset",
            )]
        )]
    )
)

# Export the name of the PVC
pulumi.export('persistent_volume_claim_name', nfs_persistent_volume_claim.metadata.name)
```

In this example:

- We define a `PersistentVolume` with NFS details where `capacity` is `5Gi` and `access_modes` is set to `ReadWriteMany`, allowing multiple nodes to write simultaneously.
- We create a `PersistentVolumeClaim`, which will look for a `PersistentVolume` that satisfies its requirements like `access_modes` and requested `storage` size.
- We declare a simple pod with a single container that will mount the storage from our Persistent Volume Claim to the path `/mnt/dataset` within the container.

Here, when the `nfs-pod` is running, it will use the NFS-backed Persistent Volume to store any data written to `/mnt/dataset`.

Make sure to have your Pulumi and Kubernetes configurations set up (i.e., context configured to point to your Kubernetes cluster) before running this Pulumi program. After you've set up your configurations, you can run this Pulumi program by executing `pulumi up` in your command line, in the directory where this file is saved. This will provision the resources in your Kubernetes cluster.