Deploy the mlflow-helm-chart helm chart on Google Kubernetes Engine (GKE)

Question

Pulumi · Accepted Answer

To deploy the `mlflow-helm-chart` on a Google Kubernetes Engine (GKE) cluster using Pulumi, we will perform the following steps:

1. Create a new GKE cluster.
2. Deploy the MLflow Helm chart to the created GKE cluster.

To accomplish these tasks, we will use the `@pulumi/gcp` package to create the GKE cluster and the `@pulumi/kubernetes` package to deploy the MLflow chart using Helm.

Here's a step-by-step Pulumi TypeScript program that sets up a GKE cluster and then uses a Helm chart to deploy MLflow:

1. **Setting up a GKE Cluster**: We utilize the `gcp.container.Cluster` to create a new Kubernetes cluster in GCP. We specify the necessary settings such as the zone, initial node count, and node machine type.
2. **Creating a Kubernetes Provider**: Once the GKE cluster is ready, we instantiate a Kubernetes provider that uses the kubeconfig from the created cluster. This provider will be used to interact with the GKE cluster.
3. **Installing MLflow using Helm**: We then deploy the MLflow chart to the GKE cluster using the `kubernetes.helm.v3.Chart` resource. We'll need to specify the chart name and version, and you can also provide custom values if needed.

Let's look at the Pulumi TypeScript code that accomplishes this:

```typescript
import * as pulumi from "@pulumi/pulumi";
import * as gcp from "@pulumi/gcp";
import * as k8s from "@pulumi/kubernetes";

// Step 1: Create the GKE cluster.
const cluster = new gcp.container.Cluster("mlflow-cluster", {
    initialNodeCount: 2,
    nodeConfig: {
        machineType: "n1-standard-1",
        oauthScopes: [
            "https://www.googleapis.com/auth/compute",
            "https://www.googleapis.com/auth/devstorage.read_only",
            "https://www.googleapis.com/auth/logging.write",
            "https://www.googleapis.com/auth/monitoring",
        ],
    },
});

// Step 2: Create a Kubernetes provider instance using the GKE cluster credentials.
const k8sProvider = new k8s.Provider("k8s-provider", {
    kubeconfig: cluster.masterAuth.apply(auth => {
        const context = `${gcp.config.project}_${gcp.config.zone}_${cluster.name}`;
        return `apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: ${auth.clusterCaCertificate}
    server: https://${auth.endpoint}
  name: ${context}
contexts:
- context:
    cluster: ${context}
    user: ${context}
  name: ${context}
current-context: ${context}
kind: Config
preferences: {}
users:
- name: ${context}
  user:
    auth-provider:
      config:
        cmd-args: config config-helper --format=json
        cmd-path: gcloud
        expiry-key: '{.credential.token_expiry}'
        token-key: '{.credential.access_token}'
      name: gcp
`;
    }),
});

// Step 3: Deploy the MLflow helm chart using the Helm Chart resource.
const mlflowChart = new k8s.helm.v3.Chart("mlflow-helm-chart", {
    chart: "mlflow",
    version: "1.0.0", // Specify the chart version you want to deploy
    fetchOpts: {
        repo: "http://your-helm-chart-repository/", // Replace with the actual Helm chart repository URL
    },
}, { provider: k8sProvider });

// Export the cluster's kubeconfig.
export const kubeconfig = cluster.masterAuth.apply(auth => {
    const context = `${gcp.config.project}_${gcp.config.zone}_${cluster.name}`;
    return `apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: ${auth.clusterCaCertificate}
    server: https://${auth.endpoint}
  name: ${context}
contexts:
- context:
    cluster: ${context}
    user: ${context}
  name: ${context}
current-context: ${context}
kind: Config
preferences: {}
users:
- name: ${context}
  user:
    auth-provider:
      config:
        cmd-args: config config-helper --format=json
        cmd-path: gcloud
        expiry-key: '{.credential.token_expiry}'
        token-key: '{.credential.access_token}'
      name: gcp
`;
});
```

Be sure to replace `"http://your-helm-chart-repository/"` with the actual repository URL from which the MLflow Helm chart can be fetched.

This program, when run with `pulumi up`, will provision the resources described above. After deployment, the Kubeconfig can be used to interact with the GKE cluster from your local machine using kubectl or other Kubernetes tools.

For more details on the resources used above, you can refer to the Pulumi documentation:

- GKE Cluster: [gcp.container.Cluster](https://www.pulumi.com/registry/packages/gcp/api-docs/container/cluster/)
- Kubernetes Provider: [k8s.Provider](https://www.pulumi.com/registry/packages/kubernetes/api-docs/provider/)
- Helm Chart: [kubernetes.helm.v3.Chart](https://www.pulumi.com/registry/packages/kubernetes/api-docs/helm/v3/chart/)

Remember to configure your Pulumi GCP project and credentials accordingly before running the above program.