How do I query logs and data in S3 directly using Amazon OpenSearch Service zero-ETL?

In this guide, we’ll set up a simple configuration to query logs and data stored in Amazon S3 directly using Amazon OpenSearch Service without the need for explicit ETL operations. This is accomplished by creating an OpenSearch Service domain and granting it access to an S3 bucket.

We’ll need to establish:

An Amazon S3 bucket to store your logs and data.
An Amazon OpenSearch Service domain configured to access the S3 bucket.

Step-by-Step Setup

Below is the complete setup configuration to achieve our goal.

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

// Create an S3 bucket to store logs and data
const myLogsBucket = new aws.s3.BucketV2("my_logs_bucket", {
    bucket: "my-logs-and-data-bucket",
    acl: "private",
    policy: `  {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Principal": {
          "Service": "opensearchservice.amazonaws.com"
        },
        "Action": "s3:GetObject",
        "Resource": "arn:aws:s3:::my-logs-and-data-bucket/*"
      }
    ]
  }
`,
});
// Create an OpenSearch service domain
const myOpensearchDomain = new aws.opensearch.Domain("my_opensearch_domain", {
    domainName: "my-awesome-opensearch-domain",
    clusterConfig: {
        instanceType: "t2.small.search",
    },
    ebsOptions: {
        ebsEnabled: true,
        volumeSize: 10,
    },
    accessPolicies: `  {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Principal": {
          "AWS": "*"
        },
        "Action": "es:*",
        "Resource": "arn:aws:es:us-west-2:123456789012:domain/my-awesome-opensearch-domain/*"
      }
    ]
  }
`,
});
export const s3BucketName = myLogsBucket.bucket;
export const opensearchDomainEndpoint = myOpensearchDomain.endpoint;
export const opensearchDomainArn = myOpensearchDomain.arn;

Summary

We successfully created an S3 bucket for storing logs and data and an Amazon OpenSearch Service domain with the required permissions to access and query data from the S3 bucket directly. This setup enables zero-ETL querying capability, streamlining your data analysis processes.

Deploy this code

Want to deploy this code? Sign up for a free Pulumi account to deploy in a few clicks.

New to Pulumi?

Want to deploy this code? Sign up with Pulumi to deploy in a few clicks.