Deploy the spark-history-server helm chart on Digital Ocean Kubernetes Service

Question

Pulumi · Accepted Answer

To deploy the `spark-history-server` Helm chart on the Digital Ocean Kubernetes Service using Pulumi, you'll need to follow these steps:

1. **Set up a Digital Ocean Kubernetes Cluster**: Use the `digitalocean.KubernetesCluster` resource to create a new Kubernetes cluster in Digital Ocean. You'll specify the region, version, and node pool details in this step.

2. **Configure Kubernetes Provider**: Once your Kubernetes cluster is up, configure Pulumi to use the correct cluster by setting up a Kubernetes provider. This step will tell Pulumi to manage resources in the created Digital Ocean cluster.

3. **Deploy Helm Chart**: Use the `kubernetes.helm.v3.Chart` resource to deploy the `spark-history-server` Helm chart. This resource needs information about the chart, such as the repository where it's located, the chart version, and any customized values for the deployment.

Let's proceed with the Pulumi TypeScript program that encapsulates the steps mentioned above:

```typescript
import * as pulumi from "@pulumi/pulumi";
import * as digitalocean from "@pulumi/digitalocean";
import * as k8s from "@pulumi/kubernetes";

// Create a Digital Ocean Kubernetes cluster
const cluster = new digitalocean.KubernetesCluster("do-cluster", {
    region: "nyc1", // Choose the appropriate region for your cluster
    version: "latest", // Use the latest available version of Kubernetes
    nodePool: {
        name: "worker-pool",
        size: "s-2vcpu-2gb", // Choose the appropriate size for your workload
        nodeCount: 2, // Number of nodes you want in your node pool
    },
});

// Export the cluster's kubeconfig
export const kubeconfig = cluster.kubeConfigs[0].rawConfig;

// Create a Kubernetes provider pointing to the Digital Ocean cluster
const k8sProvider = new k8s.Provider("k8s-provider", {
    kubeconfig: kubeconfig,
});

// Deploy spark-history-server Helm chart using the k8s provider
const sparkHistoryServerChart = new k8s.helm.v3.Chart("spark-history-server-chart", {
    chart: "spark-history-server",
    version: "5.1.0", // Specify the chart version you wish to deploy
    fetchOpts: {
        repo: "https://googlecloudplatform.github.io/spark-on-k8s-operator", // Location of the spark-history-server chart
    },
}, { provider: k8sProvider });

// Export the URL to access the Spark History Server (adjust based on your service/ingress setup if needed)
export const historyServerUrl = sparkHistoryServerChart.getResourceProperty("v1/Service", "spark-history-server", "status").apply(status => status.loadBalancer.ingress[0].hostname);
```

In this program:

- A new Digital Ocean Kubernetes cluster is created using the `digitalocean.KubernetesCluster` resource. Replace `region`, `version`, and `size` with the desired values for your deployment.
- The `kubeconfig` of the created cluster is exported using `cluster.kubeConfigs[0].rawConfig`, which is used by the Kubernetes provider to interact with the cluster.
- We establish a Pulumi Kubernetes provider (`k8s.Provider`) that uses the kubeconfig from the newly created Digital Ocean cluster. This allows Pulumi to manage resources within that cluster.
- The `spark-history-server` Helm chart is deployed using the `k8s.helm.v3.Chart` resource. This is where you specify the Helm chart name (`"spark-history-server"`), the version, and the repository (`fetchOpts`). The chart version `5.1.0` and repo link used here are examples; make sure to verify the correct values from the actual Helm repository containing the `spark-history-server` chart.
- Finally, the program exports the `historyServerUrl`, which should point to the external endpoint to access the Spark History Server. Adjust this code for your specific access configuration, such as Ingress resources.

Remember to adjust the resource specifications to fit your application's needs, including the size of the nodes in the cluster, the number of nodes, and any Helm chart custom values.

You would apply this Pulumi program with your Pulumi CLI, watching it provision your cluster and deploy the chart. At the end of the run, Pulumi will provide you with the value of `historyServerUrl`, which you can use to access your Spark History Server.

To deploy and provision the resources defined in the program, run `pulumi up` from your command line after saving the program to a TypeScript file (`.ts`). Ensure you have Pulumi CLI installed, and you've set up Pulumi to connect to your Digital Ocean account.