Using kubernetes secrets.hashicorp.com with logstash.k8s.elastic.co

Question

Pulumi · Accepted Answer

In order to achieve the integration of HashiCorp Vault secrets with a Logstash deployment on Kubernetes, we need to perform a series of steps. These steps include setting up a Vault instance with a Kubernetes secrets engine and creating a Kubernetes `Secret` that can be consumed by a Logstash deployment. Below, I'll provide a conceptual outline of the process, followed by a `TypeScript` Pulumi program that will create the necessary Kubernetes and Vault resources.

### Conceptual Outline

1. **Set up Vault**: Vault must be installed and properly configured within your Kubernetes cluster. This involves setting up a secrets engine specifically for Kubernetes that Vault will use to inject secrets.

2. **Configure Kubernetes Authentication Method**: Vault needs to authenticate requests coming from the Kubernetes pods. This is typically done with a service account that has the necessary roles and permissions.

3. **Create Secret Backend Role**: This role defines the policies and permissions that are granted to the Kubernetes service accounts.

4. **Create Kubernetes Secret**: You will need to create a Kubernetes `Secret` resource that includes the necessary data fields (like the passwords or API tokens that Logstash will need).

5. **Logstash Deployment**: You will deploy Logstash within your Kubernetes cluster. In the Logstash configuration, you will reference the Kubernetes Secret created in the previous step to provide the necessary credentials or configuration information to Logstash.

6. **Update Logstash Configuration**: Ensure that your Logstash configuration is designed to pull from the Kubernetes Secret. This can be done within the Logstash pipeline configuration files.

Here is the TypeScript program that will set up the Kubernetes secret, using Pulumi's Kubernetes and Vault providers.

```typescript
import * as kubernetes from "@pulumi/kubernetes";
import * as pulumi from "@pulumi/pulumi";
import * as vault from "@pulumi/vault";

// Assume Vault is already configured and running inside the Kubernetes cluster
// Create a Kubernetes SecretBackend in Vault
const secretBackend = new vault.kubernetes.SecretBackend("vaultK8sSecretBackend", {
    path: "kubernetes",
    // Additional configurations like TTLs could go here
});

// You would define policies and roles linking Vault's config to Kubernetes ServiceAccounts
// Skipping this in the example since it's dependent on your particular Vault installation and policy setup

// Now, creating a Kubernetes Secret that will store the credentials for Logstash
const logstashSecret = new kubernetes.core.v1.Secret("logstashSecret", {
    metadata: {
        name: "logstash-secret",
    },
    // Secret data comes here. In a real-world scenario, you would not hard-code
    // secrets in your Pulumi program, but fetch them from a secure source, or use
    // Pulumi's secret management.
    stringData: {
        "LOGSTASH_PASSWORD": 'supersecret', // Example data
        // Add other secret data as needed
    },
}, { provider: /* your k8s provider here */ });

// Deploy Logstash into the Kubernetes cluster
const logstashDeployment = new kubernetes.apps.v1.Deployment("logstashDeployment", {
    metadata: {
        name: "logstash",
    },
    spec: {
        selector: {
            matchLabels: {
                app: "logstash",
            },
        },
        template: {
            metadata: {
                labels: {
                    app: "logstash",
                },
            },
            spec: {
                containers: [
                    {
                        name: "logstash",
                        image: "logstash:7.9.3", // Use a proper Logstash container image
                        ports: [
                            { containerPort: 5044 },
                        ],
                        envFrom: [
                            {
                                secretRef: {
                                    name: logstashSecret.metadata.name,
                                },
                            },
                        ],
                        // Volume mounts, configs, and other container settings go here
                    },
                ],
            },
        },
    },
}, { provider: /* your k8s provider here */ });
```

In this program:

- The Vault Kubernetes SecretBackend is stubbed in as a placeholder, but won't do anything without Vault being configured correctly.
- We create a Kubernetes secret `logstashSecret` to store credentials; in reality, you would use Vault to inject these secrets securely.
- We deploy a Logstash instance and configure it to consume the Kubernetes secret we created, by referencing the secret in the `envFrom` section.

This setup assumes you have installed Pulumi and configured it for your Kubernetes cluster and Vault installation. Additionally, replace `/* your k8s provider here */` with an actual Kubernetes provider configuration as needed for your cluster. The Pulumi program pragmatically represents the resources you'd deploy to achieve the end state rather than configuring the specific details of your Vault setup, which would require a fully set up Vault provider and associated policies.