Using kubernetes minio.min.io with gateway.envoyproxy.io

Question

Pulumi · Accepted Answer

MinIO is a popular high-performance object storage that is API compatible with Amazon S3 cloud storage service. Envoy is a cloud-native high-performance edge/middle/service proxy. In a Kubernetes environment, MinIO can be deployed to serve as the object storage solution while Envoy can act as the gateway to provide an edge proxy, load balancing, and security features.

Combining MinIO with Envoy in Kubernetes might involve creating a Kubernetes Deployment to deploy MinIO, creating a Service to expose it, and configuring an Envoy proxy to direct external traffic to the MinIO service within the cluster.

We will use the following resources to accomplish this:
- `kubernetes:apps/v1:Deployment`: To deploy MinIO on the Kubernetes cluster.
- `kubernetes:core/v1:Service`: To expose the MinIO deployment within the Kubernetes cluster.
- `kubernetes:core/v1:ConfigMap`: To provide a configuration for Envoy.
- `kubernetes:apps/v1:Deployment`: Another Deployment for Envoy.
- `kubernetes:networking.k8s.io/v1:Ingress`: If you would like Envoy to be exposed outside the cluster. (Assuming Ingress is properly setup in your cluster)

Here is a basic Pulumi program to deploy MinIO and Envoy within a Kubernetes cluster:

```typescript
import * as k8s from "@pulumi/kubernetes";

// Deploy MinIO as a StatefulSet with a persistent volume claim.
const minioLabels = { app: "minio" };
const minio = new k8s.apps.v1.Deployment("minio", {
    spec: {
        selector: { matchLabels: minioLabels },
        replicas: 1,
        template: {
            metadata: { labels: minioLabels },
            spec: {
                containers: [{
                    name: "minio",
                    image: "minio/minio:latest",
                    args: ["server", "/data"], // Start MinIO server with a data directory.
                    ports: [{ containerPort: 9000 }], // Default MinIO API port.
                    env: [
                        // Default access and secret key, change them for production.
                        { name: "MINIO_ACCESS_KEY", value: "minio" },
                        { name: "MINIO_SECRET_KEY", value: "minio123" },
                    ],
                    volumeMounts: [
                        {
                            name: "storage",
                            mountPath: "/data"
                        }
                    ],
                }],
                volumes: [
                    {
                        name: "storage",
                        // Typically you'd use a PVC here.
                        emptyDir: {}
                    }
                ],
            },
        },
    },
});

// Create a Kubernetes Service to expose the MinIO Deployment.
const minioService = new k8s.core.v1.Service("minio-service", {
    metadata: {
        labels: minioLabels,
    },
    spec: {
        ports: [{ port: 9000, targetPort: 9000 }], // Same port as MinIO container.
        selector: minioLabels,
        type: "ClusterIP", // Internal cluster service, use LoadBalancer to expose externally.
    },
});

// Creating a ConfigMap for Envoy configuration.
const envoyConfig = new k8s.core.v1.ConfigMap("envoy-config", {
    data: {
        "envoy.yaml": `
static_resources:
  listeners:
  - address:
      socket_address: { address: 0.0.0.0, port_value: 8080 }
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: minio_service
              domains: ["*"]
              routes:
              - match: { prefix: "/" }
                route: { cluster: "minio_cluster" }
          http_filters:
          - name: envoy.filters.http.router
  clusters:
  - name: minio_cluster
    connect_timeout: 0.25s
    type: STRICT_DNS
    http2_protocol_options: {}
    load_assignment:
      cluster_name: minio_cluster
      endpoints:
      - lb_endpoints:
        - endpoint:
            address: { socket_address: { address: minio-service, port_value: 9000 } }
`
    },
});

// Deploy Envoy using the configuration defined above.
const envoy = new k8s.apps.v1.Deployment("envoy", {
    spec: {
        selector: { matchLabels: { app: "envoy" } },
        replicas: 1,
        template: {
            metadata: { labels: { app: "envoy" } },
            spec: {
                containers: [{
                    name: "envoy",
                    image: "envoyproxy/envoy:v1.18.3",
                    command: ["/usr/local/bin/envoy"],
                    args: [
                        "-c", "/config/envoy.yaml",
                        "--service-cluster", "envoy",
                        "--service-node", "envoy",
                    ],
                    ports: [{ containerPort: 8080, name: "http" }], // Envoy listening port.
                    volumeMounts: [{
                        name: "config",
                        mountPath: "/config",
                    }],
                }],
                volumes: [{
                    name: "config",
                    configMap: {
                        name: envoyConfig.metadata.name,
                    }
                }],
            },
        },
    },
});

// Optionally, create an Ingress to expose Envoy outside of the cluster.
// This assumes an Ingress controller has been setup in the cluster already.
const ingress = new k8s.networking.v1.Ingress("envoy-ingress", {
    metadata: {
        annotations: { "kubernetes.io/ingress.class": "nginx" }, // Assuming you are using nginx ingress controller.
    },
    spec: {
        rules: [{
            http: {
                paths: [{
                    path: "/",
                    pathType: "Prefix",
                    backend: {
                        service: {
                            name: envoy.metadata.name,
                            port: { number: 8080 }, // Match the Envoy container port.
                        },
                    },
                }],
            },
        }],
    },
});
```

This program does the following:
- It creates a Deployment for MinIO, exposing an API on port 9000. Note the `MINIO_ACCESS_KEY` and `MINIO_SECRET_KEY` are set to simple values, which you should change for a production environment.
- A Kubernetes `Service` is created to expose the MinIO Deployment within the cluster.
- An Envoy configuration is defined in a ConfigMap. This configuration tells Envoy to listen on port 8080 and route requests to the MinIO service.
- An Envoy Deployment is created that mounts the configuration from the ConfigMap and starts a container that runs the Envoy proxy.
- Optionally, an Ingress resource is defined to expose Envoy to the outside world via an Ingress controller, routing external traffic to the Envoy service on port 8080.

To use this program:
- Install Pulumi and configure Kubernetes.
- Create a new TypeScript Pulumi project.
- Copy the code into `index.ts`.
- Run `pulumi up` to deploy the MinIO and Envoy.

This will deploy MinIO and Envoy to provide an S3-compatible object storage service in your Kubernetes cluster, accessible via Envoy as a gateway. Remember to set up DNS and potentially a more robust storage solution for MinIO for production use.