Deploy the hadoop-yarn helm chart on Google Kubernetes Engine (GKE)

Question

Pulumi · Accepted Answer

To deploy the Hadoop-YARN Helm chart on Google Kubernetes Engine (GKE) using Pulumi, you'll follow these steps:

1. Create a GKE cluster.
2. Configure the Kubernetes provider to target the newly created GKE cluster.
3. Deploy the Hadoop-YARN Helm chart onto the GKE cluster.

Firstly, ensure you have the necessary Pulumi packages installed:

- `@pulumi/google-native` for creating GKE cluster on Google Cloud.
- `@pulumi/kubernetes` for deploying Helm charts on Kubernetes.
- `@pulumi/pulumi` as this is required by all Pulumi programs.

Here's a program that accomplishes this:

```typescript
import * as gcp from "@pulumi/gcp";
import * as k8s from "@pulumi/kubernetes";
import * as pulumi from "@pulumi/pulumi";

// Step 1: Create a Google Kubernetes Engine cluster
const cluster = new gcp.container.Cluster("hadoop-cluster", {
    initialNodeCount: 2,
    nodeVersion: "latest",
    minMasterVersion: "latest",
    nodeConfig: {
        preemptible: true,
        machineType: "n1-standard-1",
        oauthScopes: [
            "https://www.googleapis.com/auth/compute",
            "https://www.googleapis.com/auth/devstorage.read_only",
            "https://www.googleapis.com/auth/logging.write",
            "https://www.googleapis.com/auth/monitoring",
        ],
    },
});

// Export the Cluster name
export const clusterName = cluster.name;

// Step 2: Configure the Kubernetes provider to use the generated kubeconfig.
const kubeconfig = pulumi.
    all([cluster.name, cluster.endpoint, cluster.masterAuth]).
    apply(([name, endpoint, auth]) => {
        const context = `${gcp.config.project}_${gcp.config.zone}_${name}`;
        return `apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: ${auth.clusterCaCertificate}
    server: https://${endpoint}
  name: ${context}
contexts:
- context:
    cluster: ${context}
    user: ${context}
  name: ${context}
current-context: ${context}
kind: Config
preferences: {}
users:
- name: ${context}
  user:
    auth-provider:
      config:
        cmd-args: config config-helper --format=json
        cmd-path: gcloud
        expiry-key: '{.credential.token_expiry}'
        token-key: '{.credential.access_token}'
      name: gcp
`;
    });

const k8sProvider = new k8s.Provider("k8s-provider", {
    kubeconfig: kubeconfig,
});

// Step 3: Use the Helm chart to deploy Hadoop-YARN.
const hadoopChart = new k8s.helm.v3.Chart("hadoop-yarn", {
    chart: "hadoop",
    version: "3.0.0", // Replace with the actual chart version you want to deploy
    fetchOpts: {
        repo: "https://helm-repo-url", // Replace with the actual Helm repository URL
    },
}, { provider: k8sProvider });

// Optional: Export the Helm chart status.
export const hadoopStatus = hadoopChart.status;
```

This program does the following:

- **Step 1**: Creates a new GKE cluster instance with the specified configuration. Here, we're assuming you want a small cluster with preemptible VMs to keep costs down. You can modify the `machineType` and other parameters as needed for your workload.

- **Step 2**: The kubeconfig is constructed using the cluster's details, which allows Pulumi's Kubernetes provider to communicate with the GKE cluster. It's set up to use Google Cloud authentication.

- **Step 3**: A Helm chart for Hadoop-YARN is deployed to the GKE cluster using the `k8s.helm.v3.Chart` resource from Pulumi's Kubernetes provider. You'll need to replace the `repo` with the actual repository URL where the Hadoop-YARN Helm chart is located and specify the chart version you wish to deploy.

I've also included optional export statements for the cluster name and Helm chart status. These will output useful information upon completion of the `pulumi up` command, which you can use to, for example, access your GKE cluster.

Remember to replace the placeholder values (`https://helm-repo-url`, `3.0.0`) with the actual ones applicable to the Hadoop-YARN Helm chart you're deploying.