1. Search-Enabled AI Applications using AWS OpenSearch Domain


    To create a search-enabled AI application using AWS OpenSearch (formerly known as Elasticsearch), you will need to set up an OpenSearch Domain, which will serve as the foundation for indexing and searching your application data. Pulumi allows you to define this infrastructure as code, making it easy to deploy and manage.

    Here is a step-by-step guide along with a Pulumi Python program that demonstrates how to create an OpenSearch Domain on AWS.

    Step 1: Create an OpenSearch Domain

    We will start by creating an AWS OpenSearch Domain with encryption at rest, VPC options, and node-to-node encryption for security. The domain will require an access policy to define permissions, such as who can access the domain.

    Step 2: Configure Access Policies

    Access policies are essential for securing your OpenSearch Domain. We will use an AWS Identity and Access Management (IAM) policy to control access to the domain. This policy can be as open or restrictive as needed; for this example, we will give open access to the domain from any AWS resource (note that this is not recommended for production environments).

    Step 3: Export the Domain Endpoint

    After creating the domain, we'll want to export the endpoint URL, which applications will use to connect to the OpenSearch service.

    The Pulumi Program

    Below is the Python program that sets up an AWS OpenSearch Domain with the steps mentioned above. Make sure you have your AWS credentials configured correctly before running the program. You can either set them up via the AWS CLI or the Pulumi configuration.

    import pulumi import pulumi_aws as aws # Create an AWS OpenSearch Domain with encryption, VPC options, and node-to-node encryption enabled. opensearch_domain = aws.opensearch.Domain("ai-search-domain", engine_version="OpenSearch_1.0", # Specify the OpenSearch version to use cluster_config=aws.opensearch.DomainClusterConfigArgs( instance_type="t2.small.search", # Choose the instance type for data nodes in the cluster instance_count=1, # Number of instances. More instances increase reliability. ), ebs_options=aws.opensearch.DomainEbsOptionsArgs( ebs_enabled=True, volume_size=10, # Size in GiB. Adjust according to your needs. ), encrypt_at_rest=aws.opensearch.DomainEncryptAtRestArgs( enabled=True # Enable encryption at rest. ), node_to_node_encryption=aws.opensearch.DomainNodeToNodeEncryptionArgs( enabled=True # Enable node-to-node encryption. ), vpc_options=aws.opensearch.DomainVpcOptionsArgs( # Specify the VPC, subnet, and security group for the OpenSearch domain. # These must be set if you want to place your domain within a VPC. subnet_ids=["subnet-xxxxxxxx"], # Replace with your actual subnet ID security_group_ids=["sg-xxxxxxxx"], # Replace with your actual security group ID ), # Open access policy (not recommended for production). # You should restrict it based on your requirements. access_policies=pulumi.Output.all(opensearch_domain.arn).apply(lambda arn: f""" {{ "Version": "2012-10-17", "Statement": [ {{ "Effect": "Allow", "Principal": "*", "Action": "es:*", "Resource": "{arn}" }} ] }} """), ) # Export the endpoint of the OpenSearch Domain. pulumi.export('domain_endpoint', opensearch_domain.endpoint)

    In this program, we start by importing pulumi and pulumi_aws modules. We then create an instance of aws.opensearch.Domain, which represents an AWS OpenSearch Domain. The engine_version parameter specifies the OpenSearch version we want to create. The cluster_config options such as instance_type and instance_count allow you to specify the computing resources for your domain.

    The ebs_options parameter enables Elastic Block Store (EBS) for your domain and sets the volume size. encrypt_at_rest and node_to_node_encryption ensure that data is encrypted while stored and as it travels between nodes.

    We configure the domain to be inside a VPC for added security and network isolation by setting vpc_options. The subnet_ids and security_group_ids are utilized here to place the domain within your existing VPC.

    The access policy controls who has access to the OpenSearch Domain. Here, we've provided an open access policy—which is not recommended for production—for instructional purposes.

    Finally, we use pulumi.export to make the domain endpoint available outside of Pulumi, so you can use it in your application or other parts of your infrastructure.

    Remember to replace the placeholder subnet-xxxxxxxx and sg-xxxxxxxx with your actual subnet and security group IDs. You'll also want to fine-tune the access policy based on your actual requirements to enforce proper security measures.