How to Deploy Search and Analytics With Amazon OpenSearch Service?

Introduction

In this guide, we will deploy a search and analytics solution using Amazon OpenSearch Service with Pulumi in TypeScript. Amazon OpenSearch Service (formerly known as Amazon Elasticsearch Service) is a managed service that makes it easy to deploy, operate, and scale OpenSearch clusters in the AWS Cloud. This solution will help you to set up a scalable search and analytics engine that can handle large volumes of data efficiently.

Key Services Involved

Amazon OpenSearch Service

Amazon OpenSearch Service is a managed service that simplifies the process of deploying, operating, and scaling OpenSearch clusters. It provides built-in integrations with other AWS services and offers features like automated backups, monitoring, and security.

Amazon VPC

Amazon Virtual Private Cloud (VPC) allows you to provision a logically isolated section of the AWS Cloud where you can launch AWS resources in a virtual network that you define. We will create a VPC to host our OpenSearch cluster.

Amazon EC2

Amazon Elastic Compute Cloud (EC2) provides scalable computing capacity in the AWS Cloud. We will use EC2 instances to access and interact with our OpenSearch cluster.

Step-by-Step Explanation

Step 1: Set Up Pulumi Project

First, set up a new Pulumi project in TypeScript. Initialize a new Pulumi project by running the following commands:

mkdir opensearch-project
cd opensearch-project
pulumi new typescript

Step 2: Configure AWS Provider

Next, configure the AWS provider in your Pulumi project. Add the following code to your index.ts file:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

const config = new pulumi.Config();
const region = config.require("aws:region");

const provider = new aws.Provider("aws", {
    region: region,
});

Step 3: Create a VPC

Create a new VPC to host your OpenSearch cluster. Add the following code to your index.ts file:

const vpc = new aws.ec2.Vpc("vpc", {
    cidrBlock: "10.0.0.0/16",
    enableDnsHostnames: true,
    enableDnsSupport: true,
});

const subnet = new aws.ec2.Subnet("subnet", {
    vpcId: vpc.id,
    cidrBlock: "10.0.1.0/24",
    availabilityZone: "${region}a",
});

Step 4: Create an OpenSearch Domain

Create an OpenSearch domain with the necessary configurations. Add the following code to your index.ts file:

const domain = new aws.opensearch.Domain("opensearch-domain", {
    domainName: "my-domain",
    engineVersion: "OpenSearch_1.0",
    clusterConfig: {
        instanceType: "t3.small.search",
        instanceCount: 2,
    },
    ebsOptions: {
        ebsEnabled: true,
        volumeSize: 10,
        volumeType: "gp2",
    },
    vpcOptions: {
        subnetIds: [subnet.id],
    },
    nodeToNodeEncryption: {
        enabled: true,
    },
    encryptionAtRest: {
        enabled: true,
    },
    advancedSecurityOptions: {
        enabled: true,
        internalUserDatabaseEnabled: true,
        masterUserOptions: {
            masterUserName: "master-user",
            masterUserPassword: "MasterUserPassword123!",
        },
    },
});

Step 5: Create EC2 Instances

Create EC2 instances to access and interact with your OpenSearch cluster. Add the following code to your index.ts file:

const ec2Instance = new aws.ec2.Instance("ec2Instance", {
    ami: "ami-0c55b159cbfafe1f0", // Amazon Linux 2 AMI
    instanceType: "t2.micro",
    subnetId: subnet.id,
    vpcSecurityGroupIds: [vpc.defaultSecurityGroupId],
    userData: `#!/bin/bash
yum install -y aws-cli
`,
});

Key Points

• Amazon OpenSearch Service: Managed service for deploying, operating, and scaling OpenSearch clusters.
• Amazon VPC: Provides a logically isolated section of the AWS Cloud to host resources.
• Amazon EC2: Scalable computing capacity to access and interact with the OpenSearch cluster.
• Security: Ensure encryption at rest and node-to-node encryption for data security.
• Scalability: Configure the OpenSearch cluster to handle large volumes of data efficiently.

Conclusion

In this guide, we have successfully deployed a search and analytics solution using Amazon OpenSearch Service with Pulumi in TypeScript. We set up a VPC, created an OpenSearch domain, and launched EC2 instances to interact with the cluster. This solution provides a scalable and secure search and analytics engine that can handle large volumes of data efficiently.

Full Code Example

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

const provider = new aws.Provider("aws", {
    region: "us-west-2",
});

const vpc = new aws.ec2.Vpc("vpc", {
    cidrBlock: "10.0.0.0/16",
    enableDnsHostnames: true,
    enableDnsSupport: true,
});

const subnet = new aws.ec2.Subnet("subnet", {
    vpcId: vpc.id,
    cidrBlock: "10.0.1.0/24",
    availabilityZone: "us-west-2a",
});

const domain = new aws.opensearch.Domain("opensearch-domain", {
    domainName: "my-domain",
    engineVersion: "OpenSearch_1.0",
    clusterConfig: {
        instanceType: "t3.small.search",
        instanceCount: 2,
    },
    ebsOptions: {
        ebsEnabled: true,
        volumeSize: 10,
        volumeType: "gp2",
    },
    vpcOptions: {
        subnetIds: [subnet.id],
    },
    nodeToNodeEncryption: {
        enabled: true,
    },
    encryptAtRest: {
        enabled: true,
    },
    advancedSecurityOptions: {
        enabled: true,
        internalUserDatabaseEnabled: true,
        masterUserOptions: {
            masterUserName: "master-user",
            masterUserPassword: "MasterUserPassword123!",
        },
    },
});

const ec2Instance = new aws.ec2.Instance("ec2Instance", {
    ami: "ami-0c55b159cbfafe1f0", // Amazon Linux 2 AMI
    instanceType: "t2.micro",
    subnetId: subnet.id,
    vpcSecurityGroupIds: [vpc.defaultSecurityGroupId],
    userData: `#!/bin/bash
yum install -y aws-cli
`,
});

export const vpcId = vpc.id;
export const subnetId = subnet.id;
export const domainEndpoint = domain.endpoint;
export const instanceId = ec2Instance.id;