Deploy the elastic-filebeat helm chart on Google Kubernetes Engine (GKE)

Question

Pulumi · Accepted Answer

To deploy the Elastic Filebeat Helm chart on Google Kubernetes Engine (GKE), you will need to follow these steps:

1. **Set up a GKE Cluster**: First, you'll set up a Kubernetes cluster in GKE where your Filebeat Helm chart will be deployed.
2. **Install and Configure Helm in your Pulumi Program**: You'll need to install the Helm CLI tool and configure it to deploy charts to your GKE cluster.
3. **Deploy the Filebeat Helm Chart**: Deploy the Filebeat chart from the Elastic Helm repository to your GKE cluster using Pulumi's Kubernetes provider.

Below is a Pulumi program written in TypeScript that performs these steps. The program first creates a GKE cluster, then uses the Pulumi Kubernetes provider to deploy the Filebeat Helm chart to the cluster.

Let's start by installing the necessary Pulumi packages for the GCP and Kubernetes providers:

```bash
# Install the Pulumi GCP package
$ npm install @pulumi/gcp

# Install the Pulumi Kubernetes package
$ npm install @pulumi/kubernetes
```

Now, we'll create the Pulumi program that provisions the GKE cluster and deploys the Filebeat Helm chart to it:

```typescript
import * as gcp from "@pulumi/gcp";
import * as k8s from "@pulumi/kubernetes";
import * as pulumi from "@pulumi/pulumi";

// Create a GKE cluster.
const cluster = new gcp.container.Cluster("my-gke-cluster", {
    initialNodeCount: 2,
    nodeConfig: {
        machineType: "n1-standard-1",
        oauthScopes: [
            "https://www.googleapis.com/auth/compute",
            "https://www.googleapis.com/auth/devstorage.read_only",
            "https://www.googleapis.com/auth/logging.write",
            "https://www.googleapis.com/auth/monitoring"
        ],
    },
});

// Export the Cluster name
export const clusterName = cluster.name;

// Export the Kubeconfig to access the cluster using kubectl
export const kubeconfig = pulumi.
    all([ cluster.name, cluster.endpoint, cluster.masterAuth ]).
    apply(([ name, endpoint, masterAuth ]) => {
        const context = `${gcp.config.project}_${gcp.config.zone}_${name}`;
        return `apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: ${masterAuth.clusterCaCertificate}
    server: https://${endpoint}
  name: ${context}
contexts:
- context:
    cluster: ${context}
    user: ${context}
  name: ${context}
current-context: ${context}
kind: Config
preferences: {}
users:
- name: ${context}
  user:
    auth-provider:
      config:
        cmd-args: config config-helper --format=json
        cmd-path: gcloud
        expiry-key: '{.credential.token_expiry}'
        token-key: '{.credential.access_token}'
      name: gcp
`;
    });

// Create a Kubernetes provider instance that uses our cluster from above.
const k8sProvider = new k8s.Provider("my-k8s-provider", {
    kubeconfig: kubeconfig,
});

// Deploy Filebeat Helm chart to the GKE cluster using the Helm Provider.
const filebeatChart = new k8s.helm.v3.Chart("filebeat", {
    chart: "filebeat",
    version: "7.14.0", // specify the version of the chart you wish to deploy
    fetchOpts:{
        repo: "https://helm.elastic.co",
    },
    // Set the necessary Filebeat configuration values here.
    values: {
        // For example, specify which Elasticsearch instance to use:
        // elasticsearchHosts: "http://your-elastic-instance:9200"
    },
}, { provider: k8sProvider });

// Export the Filebeat Chart status
export const filebeatStatus = filebeatChart.status;
```

In the program above:

- We use `@pulumi/gcp` to create a new GKE cluster.
- We construct the kubeconfig needed to communicate with the GKE cluster.
- We create a Pulumi Kubernetes provider that uses this kubeconfig.
- We instantiate a new Helm Chart resource, specifying the `filebeat` Helm chart to be deployed from the Elastic Helm chart repository.
- We export the status of the Filebeat deployment and the kubeconfig, which you can later use with `kubectl` to interact with your GKE cluster.

To run this Pulumi program:

1. Save the code to a file with a `.ts` extension, for instance, `index.ts`.
2. Ensure you are authenticated with GCP and have set up the necessary permissions.
3. Run `pulumi up` to preview and deploy the changes.

After deployment, Filebeat will start shipping logs to the specified output, which is typically an Elasticsearch instance.

Please replace placeholder values like `your-elastic-instance` with the actual values for your ElasticSearch URL and adjust any configurations in the `values` section according to your requirements. Make sure to specify the version of the Filebeat Helm chart that you want to deploy; here it is set to `7.14.0` as an example.