Using kubernetes elasticsearch.k8s.elastic.co with projectcalico.org

Question

Pulumi · Accepted Answer

To deploy an Elasticsearch cluster on Kubernetes using elasticsearch.k8s.elastic.co with network policies enforced by Project Calico (projectcalico.org), we would follow these steps:

Set up a Kubernetes cluster: Ensure you have a running Kubernetes cluster. You can use any cloud provider such as AWS, GCP, or Azure, or run it locally using minikube or kind. Pulumi supports creating clusters on various cloud providers if you don't have one set up yet.
Install the Elasticsearch Operator: The Elasticsearch operator will manage the Elasticsearch cluster on Kubernetes. You will need to apply the Elastic CRDs and the operator yaml to your cluster.
Deploy an Elasticsearch Cluster: After the operator is installed, you will define an Elasticsearch resource to specify the configuration of your Elasticsearch cluster.
Configure Calico Network Policies: Project Calico is used to enforce fine-grained network security policies in Kubernetes. You will create network policy resources to control the traffic allowed to and from the Elasticsearch pods.

Below is a program written in TypeScript using Pulumi which shows how you might define the Elasticsearch cluster and a simple network policy using Project Calico. Note that this program assumes you have Pulumi installed and setup with your Kubernetes cluster context.

Please replace the placeholders like <namespace> with the actual values you wish to use.

import * as k8s from '@pulumi/kubernetes';

// Create a provider resource to specify the namespace where the Elasticsearch cluster will be deployed.
const provider = new k8s.Provider('provider', { namespace: '<namespace>' });

// Install Elasticsearch CRDs
const elasticsearchCRDs = new k8s.yaml.ConfigGroup('elasticsearchCRDs', {
    files: 'https://download.elastic.co/downloads/eck/1.0.1/all-in-one.yaml',
}, { provider });

// Create an Elasticsearch cluster instance
const elasticsearchCluster = new k8s.apiextensions.CustomResource('elasticsearchCluster', {
    apiVersion: 'elasticsearch.k8s.elastic.co/v1',
    kind: 'Elasticsearch',
    metadata: {
        name: 'elasticsearch',
    },
    spec: {
        version: '7.9.0',
        nodeSets: [{
            name: 'default',
            count: 1,
            config: {
                node.master: true,
                node.data: true,
                node.ingest: true,
                node.store.allow_mmap: false,
            },
        }],
    },
}, { provider, dependsOn: [elasticsearchCRDs] });

// Create a Project Calico network policy that allows traffic to Elasticsearch pods only from within the same namespace
const calicoNetworkPolicy = new k8s.apiextensions.CustomResource('calicoNetworkPolicy', {
    apiVersion: 'crd.projectcalico.org/v1',
    kind: 'NetworkPolicy',
    metadata: {
        name: 'elasticsearch-policy',
        namespace: '<namespace>',
    },
    spec: {
        selector: "elasticsearch.k8s.elastic.co/cluster-name == 'elasticsearch'",
        ingress: [{
            action: 'Allow',
            from: [{
                namespaceSelector: {
                    matchLabels: {
                        name: '<namespace>',
                    },
                },
            }],
        }],
        egress: [{
            action: 'Allow',
        }],
    },
}, { provider });

// Export the Elasticsearch cluster service URL
export const elasticsearchURL = elasticsearchCluster.status.childResourceKinds.filter(cr => cr.kind === 'Service')[0].status.loadBalancer.ingress[0].hostname;

In this program:

We use the @pulumi/kubernetes package to interact with Kubernetes.
We install the Elasticsearch custom resource definitions (CRDs) using a Pulumi ConfigGroup. This only needs to be done once per cluster.
We create an Elasticsearch custom resource to deploy the cluster, ensuring it depends on the CRDs being installed.
We define a Calico NetworkPolicy to allow ingress traffic to Elasticsearch pods from within the same namespace and allow all egress traffic. The network policy uses a selector to apply policies specifically to our Elasticsearch pods.
We export the URL of the Elasticsearch service, which you can use to interact with Elasticsearch once it's deployed.

Ensure you have the kubectl command line tool configured to communicate with your Kubernetes cluster, as this is what Pulumi interacts with to deploy and manage your Kubernetes resources.

Replace placeholders in the above code with actual values relevant to your setup. The namespace should be where you want to deploy your services, and you may need to adjust the Elasticsearch version and configuration to fit your needs. Moreover, the Calico NetworkPolicy might need to be more or worse restrictive based on your security requirements.

Run the Pulumi program with the following commands:

pulumi up

This command will preview and deploy the changes described by the program onto your Kubernetes cluster. If everything is setup correctly, the resources defined in the program will be created in the order specified, taking into account their dependencies. After the command completes successfully, you will have an Elasticsearch cluster running and configured with basic network policies using Calico.