Using aws kendra with s3storage

Question

Pulumi · Accepted Answer

Amazon Kendra is an intelligent search service powered by machine learning, and AWS S3 (Simple Storage Service) is a scalable object storage service. One of the core capabilities of Amazon Kendra is to index content from various data sources, including S3 buckets, allowing users to search through their data using natural language queries.

To leverage AWS Kendra with S3, you would typically perform the following steps:

1. **Create an S3 Bucket**: Where your documents to be indexed are stored.
2. **Set up an Amazon Kendra Index**: The index is where the data will be searched from.
3. **Create a Data Source**: You need to connect Kendra to your S3 bucket where your documents are. Kendra will use this to index the documents.
4. **Sync the Data Source**: This is so Kendra can read the documents from S3 and index their content.

Below is a TypeScript program that sets up an Amazon Kendra index and connects it to an S3 data source using Pulumi. This program assumes that you've already set up AWS credentials for Pulumi to use.

Please replace `YOUR_S3_BUCKET_NAME_HERE` with the actual name of the S3 bucket that you have created, and where your documents reside.

```typescript
import * as aws from "@pulumi/aws";

// Create the IAM role that Kendra will use to access S3
const kendraRole = new aws.iam.Role("kendraRole", {
    assumeRolePolicy: `{
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Principal": {"Service": "kendra.amazonaws.com"},
                "Action": "sts:AssumeRole"
            }
        ]
    }`
});

// Attach the required policies to the IAM role
const kendraPolicyAttatchment = new aws.iam.PolicyAttachment("kendraPolicyAttachment", {
    roles: [kendraRole.name],
    policyArn: aws.iam.ManagedPolicies.AmazonKendraFullAccess, // This provides full access to Kendra. Scope down as necessary for production.
});

// Create the Amazon Kendra Index
const kendraIndex = new aws.kendra.Index("kendraIndex", {
    roleArn: kendraRole.arn,
    edition: "DEVELOPER_EDITION", // Kendra offers Developer and Enterprise editions. Select as per need.
});

// Create the data source linking Kendra to S3
const kendraS3DataSource = new aws.kendra.DataSource("kendraS3DataSource", {
    indexId: kendraIndex.id,
    type: "S3",
    roleArn: kendraRole.arn,
    configuration: {
        s3Configuration: {
            bucketName: "YOUR_S3_BUCKET_NAME_HERE",
            // Optionally, you can set up inclusion and exclusion patterns for indexing.
        },
    },
});

// Output the Kendra Index ID
export const kendraIndexId = kendraIndex.id;

// Output the Data Source ID
export const kendraDataSourceId = kendraS3DataSource.id;
```

In this program, we begin by setting up an IAM role that Amazon Kendra can assume to access the contents of an S3 bucket (`kendraRole`). We attach the `AmazonKendraFullAccess` managed policy to it, which may be adjusted to better fit secure access requirements.

Then, we create a Kendra index (`kendraIndex`) which is where the data will be indexed and searched from. For demonstration purposes, we use the 'DEVELOPER_EDITION' of Kendra, which is more cost-effective for a proof of concept or development purposes. The 'ENTERPRISE_EDITION' is available for production workloads.

After that, we set up the data source that connects Kendra to our S3 bucket (`kendraS3DataSource`). The data source configuration points to the S3 bucket where the documents are stored.

Finally, we export the Kendra Index ID and Data Source ID as stack outputs, which can be useful for querying the status or managing these resources outside of Pulumi.

To deploy this Pulumi program, save it to a file named `index.ts`, and then run the following commands:

```shell
pulumi up
```

This will set up Amazon Kendra to index and enable searches on the documents in the specified S3 bucket. Please note, that in a production environment, access and permission scopes should be limited to what is necessary for the intended operations to adhere to the principle of least privilege.