Deploy the kafka-connector helm chart on Google Kubernetes Engine (GKE)

Question

Pulumi · Accepted Answer

To deploy the Kafka Connector Helm chart on Google Kubernetes Engine (GKE) using Pulumi, you'll first need to create a GKE cluster, and then deploy the Helm chart into that cluster. Below you'll find a detailed Pulumi TypeScript program that carries out these steps:

1. **Create a GKE Cluster:** Using the `gcp.container.Cluster` resource, we'll define and create a new GKE cluster. You will be able to modify the configuration to suit your needs, such as setting the number of nodes.

2. **Deploy Kafka Connector Helm Chart:** With the `kubernetes.helm.v3.Chart` resource from Pulumi's Kubernetes package, we'll deploy the Kafka Connector Helm chart to the GKE cluster. You'll need to specify the name of the chart and any configuration settings it requires.

Here's the full program:

```typescript
import * as pulumi from "@pulumi/pulumi";
import * as gcp from "@pulumi/gcp";
import * as k8s from "@pulumi/kubernetes";

// Create a GKE cluster
const cluster = new gcp.container.Cluster("kafka-connector-cluster", {
    initialNodeCount: 1,
    nodeVersion: "latest",
    minMasterVersion: "latest",
    nodeConfig: {
        // Choose the machine type based on your workload requirements
        machineType: "e2-medium",
        oauthScopes: [
            "https://www.googleapis.com/auth/compute",
            "https://www.googleapis.com/auth/devstorage.read_only",
            "https://www.googleapis.com/auth/logging.write",
            "https://www.googleapis.com/auth/monitoring"
        ],
    },
});

// Export the Cluster name
export const clusterName = cluster.name;

// Export the Kubeconfig to access the GKE cluster
export const kubeconfig = pulumi.
    all([cluster.name, cluster.endpoint, cluster.masterAuth]).
    apply(([name, endpoint, masterAuth]) => {
        const context = `${gcp.config.project}_${gcp.config.zone}_${name}`;
        return `apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: ${masterAuth.clusterCaCertificate}
    server: https://${endpoint}
  name: ${context}
contexts:
- context:
    cluster: ${context}
    user: ${context}
  name: ${context}
current-context: ${context}
kind: Config
preferences: {}
users:
- name: ${context}
  user:
    auth-provider:
      config:
        cmd-args: config config-helper --format=json
        cmd-path: gcloud
        expiry-key: '{.credential.token_expiry}'
        token-key: '{.credential.access_token}'
      name: gcp
`;
    });

// Create a Kubernetes provider that uses our cluster from above.
const clusterProvider = new k8s.Provider("kafka-connector-k8s", {
    kubeconfig: kubeconfig,
});

// Deploy Kafka Connector Helm Chart into the GKE cluster
const kafkaConnectorChart = new k8s.helm.v3.Chart("kafka-connector", {
    chart: "kafka-connector",
    // You'll need to specify the repository that contains the kafka-connector chart
    fetchOpts: { 
        repo: "https://my-helm-charts-repo.local/"
    },
    // Configure your Helm chart values here
    values: {
        // Helm chart values go here, for example:
        image: "confluentinc/cp-kafka-connect",
        version: "latest",
    },
}, { provider: clusterProvider });

// Export the Helm chart resources
export const kafkaConnectorResources = kafkaConnectorChart.resources;
```

This program performs the following actions:

- Imports the necessary Pulumi libraries.
- Creates a GKE cluster with an initial node count of one and sets the machine type for the nodes.
- Exports the name of the cluster to be used in the Kubernetes provider configuration.
- Exports a kubeconfig which enables you to interact with your cluster via `kubectl` or other Kubernetes tools.
- Sets up the Kubernetes provider to point to the newly created GKE cluster.
- Deploys the Kafka Connector to the GKE cluster using the Helm chart library.
- Exports the resources created by the Helm chart for reference.

To use this program:

1. Save the code to a file with a `.ts` extension, for instance, `deployKafkaConnector.ts`.
2. Ensure you've installed Pulumi and configured it for use with GCP.
3. Run `pulumi up` in the same directory as your `.ts` file.

Note that you may need to adjust certain values in the code to match your requirements, such as Helm chart repository URL, image names, machine types, and so on.

Also, be aware that deploying cloud resources might incur costs. Please review the pricing information for GKE and other services you are using to avoid unexpected charges.