Using kubernetes cilium.io with ceph.rook.io

Question

Pulumi · Accepted Answer

To deploy Cilium with Kubernetes for networking and Rook for storage, you'll need to first set up a Kubernetes cluster that Cilium can manage. Then you can deploy Rook Ceph to provide a distributed storage system over this Kubernetes cluster.

Cilium is a networking plugin that enhances the performance and security of container network interfaces. It manages network policies, load balancing, and more at the kernel level using eBPF technology.

Rook, on the other hand, turns distributed storage systems like Ceph into self-managing, self-scaling, and self-healing storage services. It automates the tasks of a storage administrator: deployment, bootstrapping, configuration, provisioning, scaling, upgrading, migration, disaster recovery, monitoring, and resource management.

When using Pulumi to manage such deployments, you'll create the necessary Kubernetes objects as Pulumi resources.

Below is a TypeScript program that outlines the necessary steps to deploy Cilium and Rook Ceph on a Kubernetes cluster using Pulumi. Note that this will not include the entire setup for a production-ready cluster but will give you an idea of how to start.

First, make sure you have Pulumi installed and configured for use with your Kubernetes cluster. Then you can start by writing a `index.ts` file with the following contents:

```typescript
import * as k8s from '@pulumi/kubernetes';

// Name of the namespace where Cilium will be deployed
const ciliumNamespaceName = "kube-system";

// Create a namespace for Cilium if you're using a namespace other than "kube-system"
const ciliumNamespace = new k8s.core.v1.Namespace("cilium-namespace", {
    metadata: { name: ciliumNamespaceName },
});

// Deploy Cilium as a DaemonSet
const ciliumChart = new k8s.helm.v3.Chart("cilium", {
    chart: "cilium",
    version: "1.9.1", // specify the version you desire
    namespace: ciliumNamespace.metadata.name,
    fetchOpts:{
        repo: "https://helm.cilium.io/",
    },
}, { dependsOn: [ciliumNamespace] });

// Namespace where Rook operator and its cluster will be deployed
const rookNamespaceName = "rook-ceph";

// Create a namespace for Rook Ceph
const rookNamespace = new k8s.core.v1.Namespace("rook-namespace", {
    metadata: { name: rookNamespaceName },
});

// Deploy Rook operator
const rookOperator = new k8s.helm.v3.Chart("rook-operator", {
    chart: "rook-ceph",
    version: "1.5.9", // specify the version you desire
    namespace: rookNamespace.metadata.name,
    fetchOpts:{
        repo: "https://charts.rook.io/release",
    },
    values: {
        // specify necessary values here
    },
}, { dependsOn: [rookNamespace] });

// Once you have the Rook operator deployed, you'd typically create a CephCluster custom resource.
// This is a very involved resource and your specific configuration will depend greatly on your own needs and environment.
// Below is a basic example of creating a CephCluster resource with minimal settings.

const cephCluster = new k8s.yaml.ConfigGroup("ceph-cluster", {
    files: "ceph-cluster.yaml", // this should point to your own CephCluster configuration file
}, { dependsOn: [rookOperator] });

// Export the Cilium and Rook resource names
export const ciliumChartName = ciliumChart.metadata.name;
export const rookOperatorName = rookOperator.metadata.name;
```

Before running this program with Pulumi, put your CephCluster custom resources into a file named `ceph-cluster.yaml`. This file should define your Ceph cluster, including how many mons, osds, mgrs, and so on you want, as well as any specific configurations for storage. As the configuration for Ceph clusters can be complex, I recommend referencing the [Rook Ceph Cluster CRD documentation](https://rook.io/docs/rook/v1.5/ceph-cluster-crd.html) for the specifics.

When you deploy this Pulumi program, it will first create the necessary namespaces (if needed), deploy Cilium using its Helm chart, deploy the Rook operator using its Helm chart, and then create a Ceph cluster using the Rook Ceph operator.

This is a powerful combination that uses modern software-defined networking and storage solutions to provide an efficient and scalable environment for your applications.