Pulumi Blog: Chatbot

Uploading documents to your AI Slackbot in real-time using S3, SQS and Pulumi on AWS

Tyler Mulligan — Mon, 03 Jun 2024 17:21:02 +0100

In the introductory blog post, we learned to Create an AI Slack Bot to Chat with Your Data Using Embedchain, Pulumi on AWS, and continued with Adding data to Pinecone using S3, Embedchain, and Pulumi on AWS for an AI Slack bot.

For reference, here’s what our architecture looked like at the end of the second blog post.

To follow along, clone the project, git clone https://github.com/catmeme/arti.git or view it on GitHub.

S3 Events and SQS

AWS provides a few ways to integrate with AWS S3 Events, for this project, we’re going to focus on the latest and most straight-forward, S3 Event Notifications. Depending on your use case, integrating with EventBridge may make more sense.

While we can integrate S3 events directly with Lambda, what happens when we add 100’s of files at the same time? Potentially, hundreds of Lambdas could be invoked simultaneously, which could become an issue depending on other factors. We’ll solve for this by Using Lambda with Amazon SQS, and reserved concurrency.

Our Pulumi program is updated to add a Queue and integrate it with our new Lambda SQS handler (policies left out for brevity, you can view them in the deploy/pulumi/__main__.py in the project).

# Add a queue
sqs_queue = aws.sqs.Queue(...)

# Add a docker image runtime for our Lambda SQS handler
docker_image_lambda_sqs_handler = docker.Image(...)

# Add Lambda SQS handler
app_sqs_lambda = aws.lambda_.Function(
 resource_name=f"{app_name}-sqs-lambda",
 environment=lambda_environment, # We've added the sqs_queue.url here as well
 image_uri=docker_image_lambda_sqs_handler.repo_digest,
 opts=pulumi.ResourceOptions(depends_on=[docker_image_lambda_sqs_handler]),
 **common_lambda_options,
)

# Add event source mapping
event_source_mapping = aws.lambda_.EventSourceMapping(
 f"{app_name}-lambda-sqs-event-source-mapping",
 event_source_arn=sqs_queue.arn,
 function_name=app_sqs_lambda.name,
 enabled=True,
 batch_size=10,
)

Add the application code responsible for processing S3 events coming from SQS.

def handler(event, context):
 for record in event.get("Records", []):
 s3_event = json.loads(record.get("body", "{}"))
 for s3_record in s3_event.get("Records", []):
 object_key = s3_record.get("s3", {}).get("object", {}).get("key")
 if object_key:
 logger.info("Processed %s in bucket %s.", object_key, app_bucket_name)
 load_data(asset_location=f"{app_bucket_name}/{object_key}")

 return {"statusCode": 200, "body": json.dumps("Successfully processed S3 event(s) from SQS messages.")}

We also have a scheduled event that we set up for ingestion in the last post, which we’re going to disable in favor of real-time events.

That’s pretty much it on the infrastructure side, here’s the updated diagram.

Slack to S3 file upload bridge

So what do we do with this new functionality? How do we expose it to the user? The most obvious approach is to reuse our existing Slack integration.

We’ll start by creating a Slack file upload bridge to S3, the bot watches for file_shared events on Slack and re-uploads files to our S3 bucket.

Using Slack Bolt, we add a new function handle_file_upload with the @app.event("file_shared") decorator, binding it to said event. When that event occurs, it’s processed, and the details of our file are retrieved, a new payload is constructed using requests, and the file gets uploaded to S3, this is our bridge.

@app.event("file_shared")
def handle_file_uploads(event, say, client, logger):
 file_id = event['file']['id']
 channel_id = event['channel_id']

 # Fetch file details using the Slack API
 file_info = client.files_info(token=slack_bot_token, file=file_id)['file']
 file_url = file_info['url_private_download']

 # Download file content from Slack
 headers = {'Authorization': f"Bearer {slack_bot_token}"}
 response = requests.get(file_url, headers=headers, stream=True)

 if response.status_code == 200:
 # Upload the file to S3
 s3_client.upload_fileobj(
 Fileobj=response.raw,
 Bucket=os.getenv("APP_BUCKET_NAME"),
 Key=file_info['name']
 )

 # Fetch recent messages to find the correct timestamp
 messages = client.conversations_history(channel=channel_id, limit=10)['messages']
 for msg in messages:
 if msg.get('files') and any(f['id'] == file_id for f in msg.get('files', [])):
 # Found the message with the correct file
 message_ts = msg['ts']
 client.reactions_add(channel=channel_id, name="white_check_mark", timestamp=message_ts)
 # Respond in a thread to the message containing the file
 say(text=f"File `{file_info['name']}` uploaded to S3 successfully! :tada:", thread_ts=message_ts)
 return

 logger.error("No valid message found for reaction.")

Keep in mind, we’re not doing any validation here yet. We’re blindly copying files from Slack to S3 and having our data loader process them. It’s probably not a good idea to deploy this bot to a public Slack channel without more moderation/validation in place.

Here’s what it looks like when we drag/drop a file into Slack now. On the right, a file was shared with the bot in private message. The bot acknowledged this upload with the checkmark reaction, the job is placed on the queue. Shortly after, on the left, the SQS Lambda handler has finished processing the uploaded file. We’re now able to use the bot to talk about the uploaded files.

Slack interactivity

In minutes, we were able to move from a scheduled task for processing files in a bucket, to real-time file processing integrated with Slack. We have our /arti command we can use to interrogate the documents we just uploaded, but we’re going to have a little more fun.

Thanks to Slack Block Kit Builder, we can build a form that exposes all of the configuration options for the query method in Embedchain’s app, providing the user with more options for interacting with the RAG model.

Our code to utilize the JSON we’ve crafted in the builder looks something like (we’ll call this /smarti):

@app.command("/smarti")
def open_smarti_modal(ack, body, client):
 ack()
 print("Opening SMARTI modal...")
 modal_file_path = Path(config.project_root) / 'config' / 'slack_modals' / 'chat.json'

 with open(modal_file_path, 'r') as json_file:
 modal_view = json.load(json_file)

 client.views_open(
 trigger_id=body["trigger_id"],
 view=modal_view
 )

And the code for handling that submission looks something like:

def handle_smarti_submission(client, logger, body, view):
 values = view["state"]["values"]
 question = values["question"]["input"]["value"]
 model = values["model"]["input"]["selected_option"]["value"]
 temperature = float(values["temperature"]["input"]["value"])
 max_tokens = int(values["max_tokens"]["input"]["value"])
 top_p = float(values["top_p"]["input"]["value"])
 prompt = values["prompt"]["input"]["value"]
 system_prompt = values["system_prompt"]["input"]["value"]
 citation_options = values["citations"]["input"].get("selected_options", [])
 citations = any(option['value'] == 'true' for option in citation_options)

 ai_response = ask_ai(
 question=question,
 model=model,
 temperature=temperature,
 max_tokens=max_tokens,
 top_p=top_p,
 prompt=prompt,
 system_prompt=system_prompt,
 citations=citations
 )
 ...

Submitting that form results in a response such as below. The bot honored our prompts, talking like Shakespeare, a thread is created, and citations were noted, showing where the data was retrieved from.

Lastly, if we want interact with the bot more publicly, then we can use the app_mention event.

@app.event("app_mention")
def handle_app_mention(event, say, client):
 text = event['text']
 bot_user_id = client.auth_test()["user_id"]

 # Strip the mention of the bot to isolate the command and arguments
 command_text = text.split(f'<@{bot_user_id}>')[1].strip() if f'<@{bot_user_id}>' in text else text

 # Parse the command and arguments
 parts = command_text.split(maxsplit=1)
 if not parts:
 say("I need a command to do something useful!")
 return

 # Get the command and the remainder of the message as arguments
 command = parts[0].lower()
 args = parts[1] if len(parts) > 1 else ""

 if command == "ask":
 if args:
 response = ask_ai(question=args)
 say(response)
 else:
 say("Please provide a question to ask.")
 elif command == "list":
 list_data_sources(say)
 else:
 say(f"Sorry, I don't recognize the command `{command}`.")

Resulting in an interaction such as

Of course, this same methodology can be used on any of the events exposed by Slack.

To learn more about extending this bot, see Slack Bolt’s documentation.

Conclusion

In this blog post, we took our existing Slack bot which has a data loading process that loads documents from an S3 bucket into a vector database on a schedule, and we wired it up to behave in real time.

To expose this functionality to the end user, we created a bridge between file uploads to Slack and S3, allowing drag/drop capabilities of files in conversational chat.

Furthering our interactions with our application, we leveraged Slack’s Block Kit to create a custom modal, and interact with events.

As always, we welcome your feedback and contributions in the Pulumi Community Slack, GitHub repository, and Pulumi Community Discussions.

New to Pulumi? Signing up is easy and free. Get started today!

Happy AI building!

Adding data to Pinecone using S3, Embedchain and Pulumi on AWS for an AI Slack bot

Tyler Mulligan — Wed, 10 Apr 2024 17:00:00 +0100

In the introductory blog post, we learned to Create an AI Slack Bot to Chat with Your Data Using Embedchain, Pulumi on AWS. However, we made a few concessions in the application logic to illustrate the broader picture of what we were able to achieve combining these three technologies. Now that we have a solid foundation for deploying our Slack bot and querying our data, lets begin moving from proof-of-concept to production-ready, iteratively.

For reference, here’s what our architecture looked like at the end of the first iteration.

To follow along, clone the project, git clone https://github.com/catmeme/arti.git

Configuration

In our proof-of-concept, we defined our Embedchain config in our Python code. Best practices dictate strict separation of config from code, so the first thing we must address is getting the embedchain configuration out of our code.

A config/config.yaml made from our dict, is much easier to read, maintain and share.

app:
 config:
 id: arti
 collect_metrics: False
 log_level: INFO
llm:
 provider: openai
 config:
 model: 'gpt-3.5-turbo-1106'
 temperature: 0.5
 max_tokens: 1000
 top_p: 1
embedder:
 provider: openai
chunker:
 chunk_size: 2000
 chunk_overlap: 0
 length_function: len
vectordb:
 provider: chroma
 config:
 collection_name: arti-ai
 dir: /tmp/db
 allow_reset: True

Our goal is to replace our big inline dict, and read a YAML file instead, following this configuration specification.

def ask_ai(question):
 """Ask the AI a question."""
 print(f"Processing question: {question}")

 app_config = {
 # hardcoded configuration we're moving out to YAML
 }

 loader = DirectoryLoader(config={"recursive": True})
 app = App.from_config(config=app_config)
 app.add("assets", loader=loader)

 return app.query(question)

To do that, though, we don’t want to implement it directly as we already have a config module, src/arti_ai/config.py. Let’s define a new method to read that YAML file, and pass it to our application at startup.

class Config:
 def __init__(self, config_file="config/config.yaml", secrets_manager_client=None):
 self.secrets_manager_client = secrets_manager_client or boto3.client("secretsmanager")
 self.project_root = Path(__file__).resolve().parents[2]
 self.config_file = self.project_root / config_file
 load_dotenv()

 def get_embedchain_config(self):
 try:
 with open(self.config_file, 'r') as file:
 return yaml.safe_load(file)
 except FileNotFoundError as e:
 raise FileNotFoundError(f"Configuration file not found: {self.config_file}") from e
 except yaml.YAMLError as e:
 raise yaml.YAMLError(f"Error parsing YAML configuration: {self.config_file}") from e

Our app.py function simplifies, becoming more readable and flexible, we’ve even fixed a bug, making our application more portable with config.project_root, now arti can be run from any directory.

def ask_ai(question: str):
 """Ask the AI a question."""
 print(f"Processing question: {question}")

 from embedchain import App
 from embedchain.loaders.directory_loader import DirectoryLoader

 loader = DirectoryLoader(config={"recursive": True})
 app = App.from_config(config=config.get_embedchain_config()) # inline dict has become this
 app.add(f"{config.project_root}/assets", loader=loader)

 return app.query(question)

Now that we’ve separated out our configuration, we can have multiple scenarios preconfigured. In the repository, there are a few examples, including a private ai configuration, so that embedchain can be run 100% locally on your machine using gpt4all. The configuration difference boils down to updating the LLM and Embedder to open-source models.

llm:
 provider: gpt4all
 config:
 model: 'orca-mini-3b-gguf2-q4_0.gguf'
 max_tokens: 1000
 top_p: 1
embedder:
 provider: huggingface
 config:
 model: 'sentence-transformers/all-MiniLM-L6-v2'

Now that we have a dynamic approach to loading configuration from YAML, lets move on to improving the application architecture, which was not built to scale beyond relatively small data sets.

Application architecture

Knowing your limits

The current application architecture uses a single process with the expectation that a response is generated within 29 seconds, before the API Gateway timeout.

Slack also has an expectation, that we must acknowledge (or ack) a request within 3 seconds.

Currently, when the application receives a request, the API Gateway Lambda integration starts the application. The lifecycle of this single process approach is diagrammed below.

If we break this into two processes, Data Processing, and API Handling, we give our API handler more dedicated time within that 29 seconds. Simultaneously, we overcome another limitation, large amounts of data being loaded will take more than 29 seconds, by breaking out data processing into its own process, we can now have up to 15 minutes to load data using Lambda.

We’re going to use the same docker image for both Lambdas by adding an entrypoint script that can take an argument to select the handler, essentially deploying the same application twice as different Lambda Functions.

Our entrypoint.sh

#!/usr/bin/env sh
set -e

. /app/venv/bin/activate

if [ -n "$LAMBDA_HANDLER" ]; then
 /app/venv/bin/python -m awslambdaric $LAMBDA_HANDLER
else
 arti "$@"
fi

Here’s what the two process lifecycles look like separated

We have, however, introduced a new issue, and that is persistence. Our single process approach always loads the data, and warm requests will have the data cached, so it was okay that the Chroma database was ephemeral, it could always be recreated on the fly.

When our corpus is many gigabytes or terabytes, losing and recreating those embeddings for the vector database is a costly endeavor, both in time, and very likely monetarily.

How do we solve for that?

Multi-tier architecture and data persistence

By putting our vector database into another tier, we don’t couple its lifecycle with our application’s. It’s pretty cool that we can have an ephemeral vector database in Lambda for demonstration purposes, but it’s not an optimal configuration for production.

One option, is to deploy Chroma ourselves using a service such as EC2 or ECS, for example, but that incurs costs. For the ease of this demonstration, we’re going to use Pinecone, which Pulumi conveniently has a provider for. Though, there are many alternative options.

Navigate to Pinecone’s homepage, and click “Get started.” After confirming your account, login to Pinecone, and click “API Keys” in the sidebar. Create a new API key for Arti.

After configuring our Pinecone API key in AWS Secrets Manager, we are able to define our Pinecone index in our __main__.py Pulumi program.

import pulumi
import pinecone_pulumi as pinecone

pinecone_index = pinecone.PineconeIndex(
 f"{app_name}-pinecone-index",
 name="main-index",
 dimension=1536,
 metric=pinecone.IndexMetric.COSINE,
 spec=pinecone.PineconeSpecArgs(
 pod=pinecone.PineconePodSpecArgs(
 environment="gcp-starter",
 pod_type="starter",
 replicas=1,
 )
 ),
)

pulumi.export("output", {"value": pinecone_index.host})

Handling data load events

In the proof-of-concept, we are abusing docker to distribute our data sources, this isn’t optimal for a few reasons, the size of the docker image increases, which also increases build times. Do our documents really belong in the container, what about privacy, triggering builds, lead time to get new data added, etc.?

The obvious next place to put our documents is AWS S3 (Simple Storage Service). We don’t want to regress our existing implementation, which works locally as well as deployed to Lambda, so we abstract the concept of loading a path of assets, whether it’s a local path or S3. In the presence of an S3 bucket name, assume we intend to use that loader, otherwise use the directory loader.

def get_loader_and_asset_root():
 """Select and configures the appropriate loader based on AWS S3 bucket presence."""
 app_bucket_name = os.getenv("APP_BUCKET_NAME")
 root_path = os.getenv("ASSETS_ROOT_PATH", "assets")
 asset_location = root_path

 # pylint: disable=import-outside-toplevel
 if app_bucket_name:
 from arti_ai.s3_loader import S3BucketLoader # type: ignore

 loader = S3BucketLoader()
 asset_location = f"{app_bucket_name}/{root_path}"
 else:
 from embedchain.loaders.directory_loader import DirectoryLoader # type: ignore

 loader = DirectoryLoader(config={"recursive": True})

 # pylint: enable=import-outside-toplevel

 return loader, asset_location


def load_data():
 """Load data from data sources."""
 print("Loading data")

 # pylint: disable=import-outside-toplevel
 from embedchain import App

 # pylint: enable=import-outside-toplevel

 app = App.from_config(config=config.get_embedchain_config())

 loader, asset_location = get_loader_and_asset_root()
 response = app.add(asset_location, loader=loader)

 return response

With our data loading logic figured out, we’ll pass the APP_BUCKET_NAME in our Pulumi program to our Lambda Function, so it can retrieve the data.

app_bucket = aws.s3.Bucket(
 f"{app_name}-bucket",
 bucket=f"{app_name}-bucket",
 # ...
)

lambda_environment = pulumi.Output.all(
 app_bucket.id,
).apply(
 lambda args: {
 "variables": {
 # ...
 "APP_BUCKET_NAME": args[0],
 # ...
 }
 }
)

app_event_lambda = aws.lambda_.Function(
 f"{app_name}-lambda",
 architectures=["x86_64"],
 environment=lambda_environment,
 image_uri=docker_image_lambda_event_handler.repo_digest,
 # ...
)

If you’re new to Pulumi, you might be wondering what’s going on with the lambda_environment, why isn’t app_bucket.id referenced directly, and what is pulumi.Output.all? Simply put, this tells Pulumi to wait until the bucket is created before trying to use its name in a string we’re building. This is an important core feature of Pulumi that largely happens by nesting Outputs. To learn more, see Inputs & Outputs.

Triggering the data loader

When deployed, we’re going to need something that doesn’t require a local machine. One method for loading data is to use a scheduled task. AWS EventBridge makes that task simple, our Pulumi resource declarations are below. We define a rule/schedule (pulled from stack configuration), add our target (the event handler Lambda), and setup permissions to invoke the Lambda function.

event_rule_schedule = aws.cloudwatch.EventRule(
 f"{app_name}-event-rule-schedule",
 schedule_expression=data_load_event_rule_schedule,
 state=data_load_event_rule_state,
)

event_rule_target = aws.cloudwatch.EventTarget(
 f"{app_name}-event-target-schedule",
 rule=event_rule_schedule.name,
 arn=app_event_lambda.arn,
)

event_rule_target_permission = aws.lambda_.Permission(
 f"{app_name}-lambda-event-target-permissions-schedule",
 action="lambda:InvokeFunction",
 function=app_event_lambda.name,
 principal="events.amazonaws.com",
 source_arn=event_rule_schedule.arn,
)

Our event lambda handler looks something like

import json

from arti_ai.app import load_data


def handler(event, context):
 print("Received event: " + json.dumps(event, indent=2))
 print("Context AWS request ID: " + json.dumps(context, indent=2))

 load_data()

 return {"statusCode": 200, "body": json.dumps("Event processed successfully!")}

Locally, we can use our command-line interface (CLI), defined in __main__.py

if args.command == "ask":
 handle_ask(args.question)
elif args.command == "load":
 load_data()
elif args.command == "reset":
 reset_data()
else:
 parser.print_help()

Any of our YAML configuration files should work with the following commands:

arti ask "what can you tell me about fruits and vegetables?"
arti load
arti reset
arti -h

You could manage your live Pinecone data this way, whether you should is highly dependent on the use case.

Wrapping up

We reflected on the project that was presented in the original post, and reviewed short-comings with the proof-of-concept, working through improvements that touched upon architecture, application design, and infrastructure. Specifically, we have moved application configuration out of code into YAML, decoupled data-loading from querying, broken out our vector database to another tier, and integrated our application with AWS EventBridge to trigger scheduled tasks.

Our delivery on the concept was expedited by utilizing services that aligned with our improvements, and Pulumi helped us achieve our architecture changes using familiar patterns in just a few lines of code.

In the next post in this series, we’ll continue to make iterative improvements to both our applications and infrastructure, and start getting deeper into RAG.

As always, we welcome your feedback and contributions in the Pulumi Community Slack, GitHub repository, and Pulumi Community Discussions.

New to Pulumi? Signing up is easy and free. Get started today!

Happy AI building!

Build an AI Slack Bot on AWS Using Embedchain & Pulumi

Tyler Mulligan — Mon, 18 Mar 2024 17:21:02 +0100

The integration of artificial intelligence (AI) to improve user experiences is gaining popularity in today’s world. One fascinating application of AI is the creation of chatbots, which can engage users in conversation and provide helpful information or services.

In this blog post, we’ll explore the process of building an AI-powered Slack bot using Embedchain, a Retrieval-Augmented Generation (RAG) framework powered by LangChain. Additionally, we’ll deploy our bot on AWS using Pulumi, a modern infrastructure as code (IaC) platform.

What are LLMs and RAG chatbots?

Large Language Models (LLMs) are sophisticated AI models trained on vast amounts of text data, enabling them to generate human-like text responses. Retrieval-Augmented Generated (RAG) is a cutting-edge approach to chatbot development that combines the capabilities of LLMs with information retrieval techniques. RAG chatbots can generate responses based on both the input query and relevant information retrieved from a knowledge base, resulting in more contextually relevant and informative interactions.

What’s an Embedchain?

Embedchain is a RAG framework powered by LangChain. It simplifies the process of creating RAG applications by providing high-level abstractions that reduce the complexity of AI technologies. With Embedchain, developers can focus on defining conversational flows and configuring the bot’s behavior using YAML configuration files, rather than dealing with low-level implementation details.

Embedchain supports configuration for various data sources, LLMs, vector databases, embedding models, and evaluation.

Creating a Slack bot

Start by cloning the project:

git clone https://github.com/catmeme/arti.git
cd arti

The files within the src/arti_ai directory can be summarized as:

The fourth file isn’t strictly necessary, but it illustrates a design pattern. The main application, app.py, is imported by slack_app but also __main__.py. This makes it easy to add additional user interfaces, such as APIs, or other chat services with minimal code.

The AI application

Let’s start with “the brain”, our Embedchain application. Below is a simple example (using explicit configuration for clarity). This code expects that OPENAI_API_KEY is populated in your environment. export OPENAI_API_KEY=<your key> to experiment with the code below.

from embedchain import App

app_config = {
 "app": {
 "config": {
 "collect_metrics": False,
 "log_level": "INFO",
 }
 },
 "llm": {
 "provider": "openai",
 "config": {
 "model": "gpt-3.5-turbo-1106",
 "temperature": 0.5,
 "max_tokens": 1000,
 "top_p": 1,
 },
 },
 "embedder": {"provider": "openai"},
 "chunker": {"chunk_size": 2000, "chunk_overlap": 0, "length_function": "len"},
 "vectordb": {
 "provider": "chroma", # we're writing to a vectordb in memory in lambda
 "config": {"collection_name": "arti-ai", "dir": "/tmp/db", "allow_reset": True},
 },
}

app = App.from_config(config=app_config)
app.add("https://www.pulumi.com/docs/", data_type="docs_site")
app.query("What is Pulumi?")

See how easy it was for us to configure our application, an LLM, an embedding model, chunker and vector database? Then we added a datasource, which is abstracting away more complex logic for us. The simple line app.add("https://www.pulumi.com/docs/", "docs_site") ends up crawling the website, chunking the documents, generating embeddings and inserting them into our vector database.

The app_config reference is the Embedchain configuration schema.

The Slack application

Now that our brain is built, we need a way for our users to interact with it. Slack Bolt is an official library for building Slack bots.

Follow their instructions to create a new Slack app.

OAuth & permissions

chat:write
channels:read
commands
im:read
im:write
users:read
users:write

Event subscriptions

message.channels
message.im

Code breakdown

The example below demonstrates two commands. knock knock is a basic ping/pong test that you expect to respond immediately. It’s not doing any complex logic, so that’s easy. What’s going on with this /arti command then? Slack requires your bot respond in 3 seconds or less. In our serverless architecture utilizing AWS API Gateway and Lambda, we’re operating on thin margins when we have cold starts. This enables us to respond quickly to Slack, but start our longer running process.

It is worth noting that in this architecture we’re still operating under 29 seconds for API Gateway’s response timeout. This can be overcome in a number of ways, but that’s out of scope for this demonstration.

from slack_bolt import App

from arti_ai.app import ask_ai, config

slack_bot_token, slack_bot_signing_secret = config.get_slack_credentials()
app = App(process_before_response=True, token=slack_bot_token, signing_secret=slack_bot_signing_secret)


def process_request(respond, body):
 """Process the request and respond to the user."""
 question = body["text"]
 response = ask_ai(question)
 respond(f"Q: _{question}_ A: {response}")


def respond_to_slack_within_3_seconds(ack):
 """Respond to the user within 3 seconds."""
 ack("Thinking...")


app.command("/arti")(ack=respond_to_slack_within_3_seconds, lazy=[process_request])


@app.message("knock knock")
def ask_who(_message, say):
 """Ask who's there."""
 say("_Who's there?_")

Configuration

The configuration module in this application normalizes retrieval of secrets from environment variables or AWS Secrets Manager. You may have noticed the Embedchain application configuration is hardcoded into the app.py. This isn’t considered best practice, and may be a fun first step in making this project yours. Otherwise, stay tuned for future posts in this series.

Deploying on AWS with Pulumi

Once our bot is ready, we’ll deploy it on AWS using Pulumi. Pulumi allows us to define our cloud infrastructure using familiar programming languages like TypeScript or Python, making it easy to manage and scale our deployment.

It’s important to note that in this selected architecture, we’re working within a few limitations. Lambda’s only writable directory is /tmp and it’s limited to 500MB, and we need to use a Docker container to overcome the 250MB filesize limitation of our application, which we’ll be bundling our data with.

The Pulumi program is broken up into the following sections:

Config: Get configuration from stack configuration, environment, and set up resource tagging
Networking: Set up VPC, VPC endpoints, security groups
S3: Bucket for later use
Application: Docker, Lambda, policies
API: API Gateway
Output: Stack outputs

Resources are contained within the VPC, policies and security groups exercise principle of least privilege, and logging is enabled. lambda_api_handler.py is responsible for handling API Gateway requests and imports the Slack app.

Prerequisites

Create the following keys in Secrets Manager. These names are configurable in deploy/pulumi/Pulumi.<stack>.yaml. Slack and Pinecone are only necessary if they are configured for use.

Secret Name	Schema
/catmeme/cloudplatform/sandbox/arti/access-token/openai	`{ "apiKey": "" }`
/catmeme/cloudplatform/sandbox/arti/access-token/slack	`{ "apiKey": "", "signingSecret": "" }`

Makefile

Makefiles offer a powerful, flexible, and efficient way to manage project tasks, contributing to smoother development processes, better team collaboration, and more reliable product builds. See the project README.md for more information.

cp -R /path/to/my/assets ./assets
cp sample.env .env
python3 -m venv venv
. venv/bin/activate
make develop

Ensure you have a matching AWS profile name to the one in the stack. The Makefile assumes <environment>-deployment. This script will attempt to copy the deploy/pulumi/Pulumi.catmeme.arti.sandbox.us-east-1.yaml as a base for configuration of the stack you deploy.

make deploy DEPLOY_ENVIRONMENT=dev

At this point, you should be able to use /arti to interact with your data via Slack.

Conclusion

In this blog post, we’ve explored the process of building an AI-powered Slack bot using Embedchain and deploying it on AWS with Pulumi. By harnessing the capabilities of LLMs and RAG chatbots, we can create intelligent conversational agents that enhance user interactions and provide valuable insights. With Embedchain’s high-level abstractions and Pulumi’s infrastructure as code platform, developers can streamline the development and deployment of AI-powered applications, unlocking new possibilities for innovation in the digital space.

As always, we welcome your feedback and contributions in the Pulumi Community Slack, GitHub repository, and Pulumi Community Discussions.

New to Pulumi? Signing up is easy and free. Get started today!

Happy AI building!

Deploy an AI/ML Chatbot Frontend on Vercel with Pulumi

Adora Nwodo — Wed, 13 Sep 2023 00:00:00 +0000

The process of taking an idea and turning it into reality has been nothing short of extraordinary since we started innovating with Artificial Intelligence. With this technology, machines learn about and communicate with people, while also helping us in ways we never could have imagined only a few years ago. If you’ve been following along, you might recall that the real AI challenge is cloud, not code where we used Python and Pulumi to a chatbot API (named katwalk).

But our AI adventure hasn’t ended there; it’s only just begun.

Now, imagine this: A Vue.js chatbot frontend, seamlessly integrating human interaction and the AI chatbot API. This blog post is part of the series on AI Infrastructure as Python. We’d delve into the creation of a captivating Vue.js chatbot frontend, and with Pulumi, we’ll deploy it to Vercel.

Figure 1. Architecture diagram

The complexity of building end-to-end AI/ML applications rests not only in the algorithms and models, but also in the several layers of infrastructure required to bring them to life. Consider this: You have a Vue.js chatbot frontend, a chatbot API powered by LLaMA models, a Vercel cloud hosting environment for the frontend, and a Runpod environment for the API. But here’s the catch: you have to weave all of these pieces together seamlessly.

Individual scripts or manual settings to set up and manage each component can soon create a tangled web of inefficiencies and potential problems. We can use Pulumi here to deploy everything. From the frontend to the backend, Pulumi orchestrates the programmatic generation and coordination of every critical infrastructure element.

With Pulumi, developer productivity increases, scalability is a breeze, and the complex infrastructure pieces fit together easily. Now that you can see why this is important, let’s get started.

Before we get into the details of using Pulumi to deploy a Vue.js frontend app to Vercel, let’s start with some quick terminology/context:

Katwalk Frontend is a demo Vue.js application that integrates with the Katwalk Server to produce a function web interface for you to communicate with your chatbot.
Vercel is a modern web development cloud platform and hosting provider. It specialises in making web application deployment simple and efficient.

How to Integrate Your Chatbot Frontend and Deploy to Vercel

If you are ready to integrate your chatbot and deploy it to Vercel along with us, then let’s double check a few requirements before we continue. While Pulumi supports many languages, we chose Python for this project, as Python is a familiar language in the AI industry and this is part of the #PulumiMLOps challenge.

Requirements

Pulumi CLI
Pulumi account and access token
Python3
Git CLI
Endpoint of Katwalk Server API. Follow the steps in this article to deploy the API and get your endpoint.
Vercel account and access token

Now that you have everything needed to get started, let’s hook up our frontend and deploy the frontend to Vercel using Pulumi.

1. Clone the repo and setup your project

The first thing we need to do is clone the repository from GitHub and setup the Pulumi Python environment. The Katwalk Frontend code is also in the repo for the Katwalk Server so this post will do both deployments.

# Clone the repository and cd to the infra directory
git clone https://github.com/pulumiverse/katwalk && cd katwalk/pulumi

# Create and initialize the python virtual env
python3 -m venv venv && source venv/bin/activate

# Install python dependencies
python -m pip install -r requirements.txt

Next, let’s setup our state and secrets store in Pulumi Cloud.

# You can store Pulumi state in multiple ways, here we use Pulumi Cloud
# Other state backends include Azure storage, Amazon S3, local file, and more
pulumi login

# Initialize your stack
# Here we name the stack "dev"
pulumi stack init --stack dev

3. Configure deployment credentials

Before the deployment, you will need to set the config values for your Vercel token, the repository name, and the repository type. The Vercel token is supposed to be set as a secret. If you open the Pulumi.${stackName}.yaml file, you can see the encrypted secret in this plain text YAML configuration.

# The name of the git repository. It should have this format {account-name}/{repository-name}
pulumi config set gitRepoName <repo_name>

# The git provider of the repository. Must be either `github`, `gitlab`, or `bitbucket`.
pulumi config set gitRepoType <github|gitlab|bitbucket>

# Configure vercel project name. Your vercel website will be `https://{vcl_project_name}.vercel.app`
pulumi config set vclProjectName <vcl_project_name>

# Configure vercel token as secret
pulumi config set --secret vclToken <vcl_token>

4. Deploy to Vercel

Now, you’re set up to deploy your Vue app to Vercel.

# Finally, run `pulumi up` to deploy!
pulumi up

5. Interact with your chatbot

Once your web app is running, you can visit the website at https://{vcl_project_name}.vercel.app, send prompts and see the response in real-time!

Figure 2. A prompt and response screenshot from our Katwalk Frontend

6. Tear down resources

Now that you’ve tested out this code and you can see the chatbot in action, remember to deprovision your deployment by running:

pulumi destroy

Understanding the Pulumi Code for Vercel Deployment

If you’d like to learn about what the Pulumi code is doing, then stick around. In this section, we’ll dive into the Python Pulumi code which is responsible for deploying the Vue.js project to Vercel using Pulumi. Let’s break down the code step by step to understand how it works.

1. Import the necessary modules

import pulumi
import pulumiverse_vercel as vercel

The code begins by importing the required modules. pulumi is the primary Pulumi Python library, while pulumiverse_vercel is a Pulumi provider for Vercel. These modules provide the necessary functionality to define and manage our web app infrastructure.

2. Configuration settings

config = pulumi.Config()
vcl_project_name = config.require("vclProjectName")
vcl_token = config.require_secret("vclToken")
git_repo_name = config.require("gitRepoName")
git_repo_type = config.require("gitRepoType")

Here, we retrieve configuration settings using Pulumi’s configuration management. Configuration values like vclProjectName, vclToken, gitRepoName, and gitRepoType are essential for setting up the Vercel deployment. These were the values that you provided using the pulumi config set command before deploying the stack.

3. Vercel provider

provider = vercel.Provider("vercel-provider",
 api_token = vcl_token
)

This part initializes the Vercel provider with the Vercel API token. The provider is used to authenticate and interact with the Vercel platform programmatically.

4. Creating a Vercel project

project = vercel.Project("vercel-project",
 name = vcl_project_name,
 framework = "vue",
 git_repository = vercel.ProjectGitRepositoryArgs(
 repo = git_repo_name,
 type = git_repo_type
 ),
 root_directory = "src/app/katwalk-frontend",
 opts = pulumi.ResourceOptions(
 provider = provider
 )
)

Here, we define a Vercel project vcl_project_name. The git_repository parameter specifies the Git repository details, including git_repo_name and git_repo_type. We also associate this project with the Vercel provider using opts.

5. Setting Environment Variables

environment = vercel.ProjectEnvironmentVariable("vercel-env",
 project_id = project.id,
 key = "VUE_APP_BACKEND_DNS",
 value = runpod_id_fqdn,
 targets = ["production"],
 opts = pulumi.ResourceOptions(
 provider = provider
 )
)

This section sets an environment variable named VUE_APP_BACKEND_DNS for the Vercel project. The value of this variable is actually the DNS name that is generated by the backend server endpoint code. In the code snippet, the environment variable is also linked to the project.id and specifies that it should be available only in the production environment.

6. Deploying the Project

deployment = vercel.Deployment("vercel-deployment",
 project_id = project.id,
 production = True,
 ref = "main",
 opts = pulumi.ResourceOptions(
 provider = provider
 )
)

Finally, we deploy the Vercel project with the vercel.Deployment resource. This deployment is configured to use the main branch and is marked as a production deployment.

As you’ve seen above, we can integrate Pulumi into our deployments when building AI apps with Python.

If you’ve joined us on this incredible journey to connect the dots between the API and the Vue.js chatbot frontend using Pulumi and Python, we’re eager to hear your story. How did it work out for you? Did you find the process as exhilarating and empowering as we did? What are you most excited about in this AI-powered future?

Our journey doesn’t end here; it’s merely a waypoint on the ever-evolving road of innovation. We invite you to share your thoughts, ideas, and suggestions for future installments of the Pulumi Python + MLOps series. Together, we’ll continue pushing the boundaries of what’s possible in the world of AI development.

Join us in the Pulumi Community Slack or in the Pulumiverse Katwalk GitHub repository to decide what #aiChallenge we tackle next!

Pulumi Blog: Chatbot

Uploading documents to your AI Slackbot in real-time using S3, SQS and Pulumi on AWS

S3 Events and SQS

Slack to S3 file upload bridge

Slack interactivity

Conclusion

Adding data to Pinecone using S3, Embedchain and Pulumi on AWS for an AI Slack bot

Configuration

Application architecture

Knowing your limits

Multi-tier architecture and data persistence

Handling data load events

Triggering the data loader

Wrapping up

Build an AI Slack Bot on AWS Using Embedchain & Pulumi

What are LLMs and RAG chatbots?

What’s an Embedchain?

Creating a Slack bot

The AI application

The Slack application

OAuth & permissions

Event subscriptions

Code breakdown

Configuration

Deploying on AWS with Pulumi

Prerequisites

Makefile

Conclusion

Deploy an AI/ML Chatbot Frontend on Vercel with Pulumi

How to Integrate Your Chatbot Frontend and Deploy to Vercel

Requirements

1. Clone the repo and setup your project

2. Login to Pulumi Cloud and initialize stack

3. Configure deployment credentials

4. Deploy to Vercel

5. Interact with your chatbot

6. Tear down resources

Understanding the Pulumi Code for Vercel Deployment

1. Import the necessary modules

2. Configuration settings

3. Vercel provider

4. Creating a Vercel project

5. Setting Environment Variables

6. Deploying the Project