Pulumi Blog: Kat Morgan

The Pulumi 'Push to start' GitOps Experience

Kat Morgan — Tue, 23 Jan 2024 19:41:13 +0000

As a skeptic of “quick starts” myself, I approach most marketing promises with a measure of cautious excitement. If the great and powerful algorithm, friends, or a peer brought your attention here, then I invite you to take this one seriously.

Pulumi, with its full support of many general-purpose programming languages, can appear like a chore to get started with. The feeling can haunt seasoned developers as much as practitioners new to infrastructure code.

However, I’ll show you that finding the proverbial easy street is easier than you might believe. The pulumi new developer story just gets sweeter when combined with a few other nice-to-have conveniences.

This is a “bring your own GitHub login” event, so refresh that login. You will want to grab a Pulumi Cloud login as well. With credentials in hand, you are set with everything that you need to start. Let’s go!

Hint: all of the steps in this demo are ready to run yourself from the Pulumi Devcontainer using just your web browser and a Github account. Even better, they are real activities that Pulumians use every day to orchestrate the cloud too!

New technology learning advice

When it comes to ice cream, sometimes just selecting your flavor can be too much. Similarly, when choosing where to start with cloud tooling the onset of “choice paralysis” can be chronic. Pulumi’s powerful toolset is both a worthy endeavor to learn and a rewarding challenge to master. There is no better time to learn than the present!

“If you are short on time and have a bike, do you stop to hop on the bike, or choose to walk because there is no time to stop and get on?”

Freezing, while great for ice cream, is not quite as satisfying when building the next cloud solution and putting it off for next week or next year can be about as useful as walking fast and pushing a perfectly good bike because there is no time to stop and get on.

How can we overcome the learning curve? Picking up and mastering a new tool can be easy if we choose to start small.

Tech learning hacks:

🏆 Do: Start
🔨 Use: Helpful Tools
🥅 Make: Achievable Goals
🎉 Win: Celebrate Successes

Recognizing achievable goals can be tricky when you are learning something new. Always over-estimate the complexity of each step. If you overestimate how complicated a task might be, the biggest risk is accomplishing the goal faster than you think! Either way, the learning journey should be enjoyable and rewarding by following this rule.

Helpful tools are more abundant than ever. I recommend Pulumi as one of the most helpful tools but that is my bias showing through. Consider how intuitive, durable, and well-integrated any tool selection you make will be. Today we’re using multiple features and services of both the Pulumi and Pulumi Cloud offering, as well as demonstrating the benefits of a tightly integrated Git, Github, Devcontainers, and VSCode workflow to make light work of a traditionally considerably complicated achievement.

The resulting developer stack left me feeling like I would be successful automating anything. Be sure to share your experience with us in the Pulumi Community Slack or find us on Twitter/X @PulumiCorp too!

A DevOps journey

The best solutions are often buried in the details. The only way to reach them is to dig.

In true DevOps fashion, development for this blog post featured a roller coaster of chronic scope creep, pushing deadlines, discovering details that scrapped the original plan, and during the final finish line approach too! The worry and doubt was palpable.

Have we taken on too much? Did project ambitions exceed the achievable? The solution emerged while hacking on a side quest with a friend. A single vision came into focus, tying everything together into a simple “Push to Start” Pulumi developer workflow.

In this story, it took two tries to achieve the goal. Often it can take several tries. With that backstory let’s begin this next learning adventure together.

DevOps day 0

Let’s track our DevOps journey in typical “DevOps Day” style. Because we’re counting like computers, of course, we start with zero. In other words, we need a goal, and we will be starting from scratch.

Day zero is going to be your design, planning, and prep phase. Sometimes this includes various stages of research and other times it is a matter of just getting started with hands-on investigation.

Every journey includes a beginning and an end. To demonstrate the Pulumi Push to Start developer workflow, we need a project to develop! For this adventure let’s use the Pulumi Kubernetes Provider from the Pulumi Providers registry to deploy an app on Kubernetes. This provider is one of our most popular providers, as you might guess.

In this case, we can practice with a new project from the Community Engineering team. The GitHub pulumi/devcontainer repository is a GitHub template repository containing all the fundamentals of a new Pulumi infrastructure as code GitOps project. Let’s begin with a quick tour of the GitOps repo boilerplate.

“What belongs in a GitOps git repository?”

I have seen this question raised several times during the past year. Most recently it was brought up during a great customer conversation at AWS re:Invent.

Filtering through the conflicting opinions and advice about proper GitOps repo hygiene, a few fundamentals remain consistent.

Ops Environment
- Configuration
- Dependencies
- Artifacts
Documentation
- Who it is for
- What it does
- How to Operate
- Why it is important
Actions
- Testing: Continuous Integration
- Deployment: Continuous Deployment
- Safety: Version Controls

You may have assembled projects piece by piece before. In the pulumi/devcontainer repository, we provide a turnkey “known good” starter kit which will even improve over time. As a community resource, we welcome bugs and contributions!

And now for the feature presentation, let’s get to dev’ing!

What’s in a DevOps day 0?

Day 0 usually includes aspects of planning and preparing. For this exercise, we have just a few requirements to get started. Let’s knock these out and get to the fun stuff!

Prerequisites

Now, even the fanciest push-to-start car still requires a fob, so let’s cover those bases super quick.

Web browser
GitHub account
10 minutes (approx.)

Next, let’s review the tools we will depend on for developing and deploying our project.

Commandline Utilities:

*Dependencies supplied in the Pulumi devcontainer ghcr.io/pulumi/devcontainer

Tool	Description
A Programming Language	The code part of IaC
Pulumi CLI	Infrastructure as Code Tools
Pulumi ESC	Secrets and Configuration Store
Kubectl	Kubernetes CLI
Direnv	Automated Environment Variables
KinD	Local Kubernetes Infrastructure
Github CLI	Github & Git Repository Binaries
curl, jq, act, awk, etc	Other useful tools

Cloud-native … cloud developer environment?

Besides our command line tools and the programming language of your choice, we need an editor to write our code with. Many practitioners are fluent with Vim or Emacs; others turn to editors like VSCode, IntelliJ, and others. We are going to choose the GitHub Codespaces VSCode IDE for its deep integration with GitHub, and its convenience of requiring nothing from the user other than a web browser to use.

If you are already familiar with GitHub Codespaces, then you might know what to expect next. If not, you are in for a treat, and yes that is right, we are developing for the cloud, in the cloud, with the cloud.

Alright, let’s jump in and get started by launching a new GitHub Codespaces.

👇 Click this button to begin.👇

If you prefer a more informed approach you can launch the same codespaces by navigating to the Pulumi Devcontainer Repository or you can learn more about Codespaces at this Github Codespaces Link.

Clicking the ‘Open in GitHub Codespaces’ button should take you directly into a new Codespaces session.

Note: This demonstration is deeply integrated into the Web based Codespaces environment offered by Github. Alternatively, VSCode also supports running Devcontainers with Docker locally allowing you the benefit of Pulumi Devcontainer on your own hardware.

With a new codespaces session launched you should see the VSCode IDE in your browser. If you are not already familiar with VSCode, you can learn more about it in the VSCode Docs.

Congratulations! Your developer environment is ready for some Pulumi action.

For our final Day 0 step, we still need to decide what app to deploy with our Infrastructure-as-Code (IaC) program.

Focusing on something fun, let’s deploy Minecraft as the app for our Kubernetes exercise. It’s a fun way to get started with Pulumi, a great way to show off your cloud skills to friends, and maybe even share with your favorite young future technologists! After all, Open Source DevOps is for everyone!

DevOps day 1

We have an environment! We have a plan! Now we need to code … or do we?

Don’t just take my word for it. The big brains of Pulumi’s AI Engineering team have delivered a brilliant Pulumi AI. Your Pulumi IaC developer sidekick makes learning and developing Pulumi just that much easier.

Q: What can Pulumi AI do? A: Today’s AI is all experimental. As an industry, we are only just beginning to discover the possibilities, but you are invited to try it yourself with the Pulumi Neo WebUI or with the pulumi new cli as demonstrated below.

`~$ pulumi new --ai`

Pulumi Neo is available via WebUI much like the ChatGPT console. Besides the familiar browser-based interface, Pulumi AI is also built into the pulumi new CLI too! Now you can prompt Pulumi AI interactively from the convenience of the command line tools you will already be using, conveniently configured in your new Codespaces IDE.

Let’s try it! (Reminder: Perform all these steps from within the Codespaces IDE.)

# First Login to Pulumi Cloud for a few features we will use in upcoming GitOps content.
pulumi login

Step 2. Create a new program

# Let's create a new pulumi with the Pulumi CLI
pulumi new

Step 3. YOLO for fun and profit!

Let’s go all in on this one and try Pulumi AI. We can always enhance the prompt with additional dialog back and forth during the pulumi new chat if it doesn’t meet our expectations right away.

It is good to keep in mind that even as advanced as AI is today, the practice is still very much in its infancy. We will save more deep dives into the differences Traditional Ops (Imperative Ops), GitOps (Automated Declarative Ops), and AI Ops (Probability Augmented GitOps) in another post, for now let’s see how close the Pulumi AI can take us.

Woohoo! We selected the AI option on CLI, let’s follow the user prompts and feed the AI a simple prompt of what we want our Pulumi project to create.

Let’s try something like this:

Write a pulumi infrastructure as code program which creates the namespce 'minecraft' and using the pulumi_kubernetes.helm.v3.Release to deploy the https://itzg.github.io/minecraft-server-charts 'minecraft-server' helm chart, sets EULA to true, deploys onto Kubernetes in the minecraft namespace."

How did we do? Let’s see what the AI came up with.

It’s looking pretty good for a simple deployment. In DevOps, nothing works till the CI testing turns green. Since fortune favors the bold let’s proceed. For my developer workflow I ran kind create cluster in the Codespaces terminal. You can do the same or add your own kubeconfig to deploy on your choice of cluster.

Of course, proceed through the remaining prompts from the pulumi new interactive CLI.

I chose Python for this project, but you can choose any of the supported languages. We can see that pulumi new creates a new Python virtual environment for us programmatically. In addition to the boilerplate supporting our new project, Pulumi AI wrote the first iteration of our Python Minecraft Deployment IaC! The next steps should be printed on the screen. Assuming your Kubernetes cluster kubeconfig is ready to roll we just have one step left.

Hint, this is where we get the name for our annual “Pulumi Up conference.

Follow the link for the full playlist of 2023 Pulumi Up sessions

pulumi up -y

DevOps day 2

Finally, it is day two. This is the beginning of a choose-your-own-adventure wonderland and the opportunities go almost as far as your imagination. This is where we commit our code and plan our next change requests and enhancements.

We can commit this code to git. Build a Github Actions Workflow to automate the CI testing of our IaC with every git pr or commit. Once you have a production-ready IaC, you can add another to deploy to production clusters. If you run make act you will see how comfortable testing new GitHub actions workflows can be in the convenience of your new enhanced IDE workflow. You can see the dev container workflow running yourself right away by executing make act as a starting point.

To conclude, Pulumi is not a single tool. Pulumi is a complete cloud developer ecosystem. Tools, SDKs, and cloud services come together forming the most accelerated way to create your own cloud success legends.

Conclusion

Did it deploy?

Let’s check on the status of our deployment.

# Check for your new Minecraft pod
kubectl get po

With any luck, our Pulumi Neo gave you a working program right out of the gate. For references on Kubernetes Pulumi package resources to expand your infrastructure from here head to the Pulumi Registry Kubernetes Provider page for more.

Sharing Pulumi with people is a genuine passion found among Pulumians all around the globe but we all know evidence reigns king among engineers who, like any good practitioner, expect proof. Evolving your developer workflow with Pulumi gets even better than just easy, it’s remarkably powerful and ready for you to adopt in your daily workflows right away too.

Follow PulumiTV for upcoming examples built on this solution and to see how Pulumi ESC brings safety to your secrets and more.

What’s in the recipe?

Because no hero’s journey is complete without a gruesome account of what came before, I have included many of the manual steps required to reach this state one command at a time for the ~~morbidly~~ curious as well.

click ‘details’ to expand

Git Code Repository

Now create a git code repository to version control this project infrastructure code in Github.

# Authenticate with Github before proceeding.
gh auth login --web --git-protocol https --scopes "repo,gist,read:packages,admin:org,delete_repo,codespace"

# Create new Git Repository
gh repo create workshop --public \
 --gitignore Python --license apache-2.0 \
 --description "pulumi iac developer workflow workshop"

# Initialize the git repository
git clone --recurse-submodules https://github.com/${GITHUB_USER}/workshop .

# Configure `git` cli to use the `gh` cli for authentication with Github
gh auth setup-git

# Configure the username and email associated with your Github account.
git config --global user.email ${GIT_COMMITTER_EMAIL}
git config --global user.name ${GITHUB_USER}

# List files being tracked by git and their current status.
git status

Direnv: Automatic Environment Variables

Various non-secret environment variables may be worth maintaining in code locally as well. There are many ways to do this, but here we are going to use Direnv to automatically load environment variables from a .envrc file.

# Add environment variables useful during development to .envrc
cat <<EOF >> .envrc
export NO_COLOR=true
export PULUMI_HOME=\${HOME}/.pulumi
export PULUMI_SKIP_UPDATE_CHECK=true
export PULUMI_SKIP_CONFIRMATIONS=true
export KUBECONFIG=\${PWD}/.kube/config
source venv/bin/activate 2>/dev/null || true
EOF

Now you can enable direnv on this directory.

# Enable direnv in this directory.
direnv allow

Note:

Env variables shown are for educational purposes only and should be used with care. Read more about available Pulumi Environment variables here: https://www.pulumi.com/docs/cli/environment-variables

Pulumi Devcontainer

Now, to make sure we have all of our CLI dependencies, let’s grab the Pulumi Devcontainer.

# Add the Pulumi Devcontainer git submodule
git submodule add https://github.com/pulumi/devcontainer .devcontainer
git submodule update --init --recursive .devcontainer

# Copy the devcontainer config of your choice into the top level of your project
# so it will auto select by default when you open Codespaces in the future.
cp -f .devcontainer/devcontainer.json .devcontainer.json

# Rebuild the codespaces using the newly added Pulumi Devcontainer
gh codespace rebuild --codespace ${CODESPACE_NAME}

Pulumi Cloud

Let’s login to Pulumi Cloud and initialize a new ESC environment to store our environment variables, secrets, and configuration which should be preserved and distributed securely from your Pulumi Cloud account.

# Login to Pulumi Cloud
pulumi login

# Create a new ESC Environment (optionally use another already existing environment)
pulumi env init workshop

KinD: Kubernetes-in-docker

We are going to use Kubernetes to demonstrate our Pulumi IaC. Let’s go ahead and create a new cluster now.

# Create the directory for your new kind kubeconfig
mkdir .kube

# Create a Kind Cluster
kind create cluster

# Let's make sure our kubeconfig works and pods are starting up
kubectl get pods -A

# Load Kubeconfig into Pulumi ESC as an encrypted secret for safe keeping.
# This is an example use case for storing kubeconfig which would not usually be practical
# for disposable kind kubernetes clusters like this however the approach is practical in many
# other scenarios which you may come across.
pulumi env set workshop secrets.kubeconfig.kind --secret "$(jq . -R -s < $KUBECONFIG)"
pulumi env set workshop files.KUBECONFIG --plaintext \${secrets.kubeconfig.kind}

In our environment, we maintain our secrets including API personal access tokens, kubeconfigs, and such all with Pulumi’s Environments, Secrets, and Configuration cloud service.

# Load the newly updated Pulumi ESC Environment in the local shell
eval $(pulumi env open workshop --format shell)

Now let’s create the deployment code!

If fortune favors the bold, let’s be bold on this next step and let AI write our sample code on the fly!

As this is a “hello world” style demonstration intended to showcase the developer workflow more than any specific cloud technologies, let’s deploy something fun like a Minecraft server! Be sure to watch the video demonstration of this exercise on Pulumi’s PulumiTV YouTube channel for an extra easter egg bonus step!

# Write a new Pulumi Python IaC program to deploy Minecraft on Kubernetes
pulumi new \
 --ai "Write a program using pulumi kubernetes helm v3 Release to deploy the itzg/minecraft-server helm chart on Kubernetes, and include a pulumi output to show the helm release status." \
 --description "A pulumi infrastructure as code (iac) program for deploying and serving minecraft on kubernetes" \
 --name "minecraft-on-kubernetes" \
 --language python \
 --stack "workshop" \
 --force \
 --dir .

Now let’s see if our new Infrastructure as Code Pulumi Python Codebase will deploy!

# Deploy your new Pulumi IaC on Kubernetes
pulumi stack select workshop
pulumi up

Cleanup

pulumi destroy -y --skip-preview
pulumi stack rm workshop
kind delete cluster --name kind
gh repo delete ${GITHUB_USER}/workshop

LangChain for DevOps: Learn LLM & GenAI for Dev, Sec & Ops

Sara Huddleston — Thu, 26 Oct 2023 00:00:00 +0000

The emergence of DevOps revolutionized software development. Now, with AI powered tools like LangChain, these transformations are being accelerated. Unsurprisingly, our distinguished speaker at the launch of Pulumi’s in-person AI Talks, Patrick Debois, who coined the term “DevOps,” has recently tuned into LLM and GenAI Ops using the Langchain framework.

This article unwraps the topics addressed during our “Generative AI, Apps, and DevOps | AI/ML Talks” event in Seattle, focusing on Langchain and GenAI for DevOps. You’ll be able to watch Patrick Debois teach LangChain to learn LLMs and GenAI, with code examples and lessons that are easily understood by traditional software engineering.

What is LangChain used for?

LangChain is an open-source Python framework designed to empower developers to construct robust generative AI applications. It facilitates the integration of advanced AI models such as OpenAI’s GPT, Google’s Gemini, and Meta’s LLaMA. This versatile tool finds application in developing retrieval-augmented chatbots and other personalized assistants, utilizing technologies like ChatGPT. Furthermore, it enables tasks such as question-answering (GQA), summarization, code comprehension, API interactions, data cleansing, and much more.

Why is LangChain becoming popular?

The landscape of AI development is rapidly evolving, and LangChain is packed with incredible features for building LLM applications and tools. Developers with a wide range of expertise use LangChain for tasks such as managing interactions with language models, establishing seamless connections between diverse components, and integrating external resources like APIs and databases effortlessly.

The LangChain platform hosts a diverse collection of APIs seamlessly integrated into applications. This enables developers to incorporate advanced language processing functionalities without the need for laborious construction from scratch.

LangChain Accelerating DevOps

Patrick Debois presenting the workshop Dev, Sec & Ops meet LangChain

LangChain can streamline the integration of large language models and data sources. By helping bring in AI capabilities, DevOps can revolutionize software development life cycles by making it easier for teams to handle challenges while optimizing resource allocation and usage through their cutting-edge, AI-powered solutions. This makes the automation of manual processes easier and the development of new resources far more accessible with faster utilization rates by leveraging artificial intelligence. Significantly important as “Developer Velocity and Productivity” become business performance metrics across various industries.

The features included in LangChain include chain interface memory functions for scalability within apps, control over dataflow, as storage – making code quality more reliable, plus seamless machine learning models integration, which allows them to build custom automated AI programs quickly. Thus providing users with an advanced but easy way of creating sophisticated solutions by combining their knowledge of coding and machine training models together.

How to Use LangChain to Learn LLMs and GenAI for Dev(Sec)Ops

Patrick teaches in the way he would like to be taught. The workshop mainly uses the Langchain framework and basic Python knowledge. OpenAI will also need to be installed. You’ll learn essential skills in multiple use cases, applying Langchain for LLM application Development.

You can see the breakdown of the lessons per role below, and find the code examples in the Learning LLMs and GenAI for DevSecOps repo.

Developer

Calling a simple LLM using OpenAI
Looking at debugging in Langchain
Chatting with OpenAI as a model
Using prompt templates
Use Docloader to read your local files and prepare them for the LLM
Explain the calculation and use of embeddings
Understand how splitting and chunking are important
Loading embeddings and documents in a vector database
Use a chain for Questions and Answers to implement the RAG pattern (Retrieval Augmented Generation)
Show the use of OpenAI documentation to have the llm generate calls to find real-time information
Implement an Agent and provide it with tools to get more real-time information

Operations

Find out how many tokens you are using and the cost
How to cache your calls to an LLM using exact matching or embeddings
How to cache the calculation of embeddings and run the calculation locally
Run your own local LLM (using Ollama)
Track your calls and log them to a file (using a callback handler)
Impose output structure (as JSON) and have the LLM retry if it’s not correct

Security

Explain the OWASP top 10 for LLMS
Show how simple prompt injection works and some mitigation strategies
How to detect prompt injection using a 3rd party model from Hugging Face
Detect project injection by using a prompt
Check the answer LLMs provide and reflect if it is ok
Use a Hugging Face model to detect if an LLM output was toxic
Show a simple prompt for asking the LLM’s opinion on Kubernetes and Trivy vulnerabilities

Real-World Applications of LangChain AI in DevOps

Vector similarity search in a RAG application. Credit to LangChain and Neo4J team

LangChain AI has practical applications in DevOps, particularly related to question-answering using documents as context, extraction, evaluation, and natural language processing tasks, all powered by AI tools. An interesting scenario is implementing a knowledge graph based RAG application with LangChain to support the DevOps team.

ChatGPT based on LLMs enabled with these AI tools is one of the foremost implementations at present. Tracking capabilities through LangChain enables developers to determine which prompts are more effective, thus supporting what artificial intelligence offers when integrated into development operations.

Monitoring and Anomaly Detection with LangChain AI

Monitoring LLM performance with LangChain and LangKit. Credit to WhyLabs

LangChain AI provides an efficient approach to DevOps through continuous monitoring, and anomaly detection. LangChain’s integration in DevOps allows for intelligent monitoring of various system parameters, application performance metrics, and log files. It can process large volumes of data generated by applications and infrastructure components. Through natural language processing (NLP) techniques, LangChain can interpret logs, error messages, and system alerts in real-time.

LangChain’s AI-driven anomaly detection capabilities are pivotal in identifying deviations from expected behavior in the monitored data. By analyzing historical data patterns, LangChain can learn the normal behavior of the system and its components. When the system encounters an unusual event or behavior, LangChain can promptly detect it as an anomaly. These anomalies could be security threats, performance bottlenecks, code quality, or any irregularities within the system.

Keep in mind that monitoring LLM performance is important to ensure you can rely on the model’s accuracy and relevance in real-world applications. Our customer WhyLabs has a great article showing the significance of monitoring LLMs performance with LangChain and how to get started with monitoring.

Future: AI, Apps, and DevOps

Platform engineering, DevOps, or MLOps may be the next steps in the AI-driven software development future. AI tools are the gateway to giving developers access to big language models that allow them to create highly sophisticated solutions.

Accelerating Development with AI-Powered Tools

AI-powered tools are revolutionizing the development process, enabling DevOps teams to access intelligent recommendations and improve code quality or obtain suggestions on new ways to accomplish goals. An example is the usage of Pulumi Neo.

Tools like LangChain, Amazon CodeGuru, and Pulumi AI provide actionable insights for efficient resource utilization while enforcing policy and compliance best practices. An IaC that leverages AI empowers teams to automate and more effectively manage infrastructure.

Developers can reap multiple benefits from higher quality assessment of code lines, instantaneous detection of policy and security issues, useful cues on how to resolve problems, and improving code quality, productivity, and time management.

AI has greatly impacted optimization during the review stage, providing developers with better insights into performance issues, code effectiveness, and resource utilization without any extra effort.

Security and Vulnerability Management with AI

The marriage of AI, machine learning, and DevOps/DevSecOps practices revolutionizes security and vulnerability management in software engineering. AI-enabled tools technologies not only strengthen the defenses of software systems but also enable organizations to stay one step ahead in ensuring successful deployments by removing manual processes through automation, using single sources of truth, enforcing least-privileged access through role-based access controls, and automated remediation to correct configuration violations like auto-tagging, removing Internet access, and enabling storage encryption.

DevSecOps can foster collaboration between developers, security, and operations and still enhance security and compliance best practices using Artificial Intelligence (AI) and Machine Learning techniques.

Leveraging Machine Learning in DevOps

Applying Machine Learning in DevOps is a game-changer. Not only does it automate and enhance various processes but also brings an intelligent, predictive, and adaptive dimension to development workflows. By embracing these technologies, development teams can achieve higher efficiency, improve code quality, optimize performance, and seamlessly integrate with artificial intelligence, thereby shaping the future of software engineering.

ML models can evaluate past outcomes from compilation/builds plus performance metrics within an operation so that developers may leverage these insights when writing code accordingly with machine learning algorithms, allowing them to identify vulnerabilities or issues faster through root cause analysis while ameliorating system security against outside attacks like hackers or DDOS activity thus boosting overall system productivity thanks to AI implementation into existing processes.

Implementing AI models into DevOps: Key Considerations

When considering AI, it’s essential to assess data quality, integration complexity, cost savings, and ethical/legal considerations:

Ensuring the data used for training these AI tools are precise, exhaustive, and up-to-date will help gain more reliable results.
An important aspect of the implementation process is integrating existing DevOps instruments and guaranteeing scalability, depending on the company’s necessities.
Ensuring that all laws applicable when working with such technology.

Conclusion

AI and ML are rapidly becoming a necessity, which can greatly enhance and benefit Development, Security, and Operation teams. At first glance, it may appear daunting or out of reach, but tools like Langchain and Pulumi can make it easier to get started. Follow the Pulumi Python + AI/ML series to upskill and get ahead:

Frequently Asked Questions

Who is Patrick Debois?

Patrick Debois is a highly accomplished technologist whose expertise spans the domains of Development (Dev), Security (Sec), and Operations (Ops). Acknowledged as a respected confidant within the developer, security, and operations communities, Patrick is presently deeply engrossed in the realm of Artificial Intelligence and Machine Learning, continually pushing the boundaries of his technical acumen.

He was the organizer of the inaugural DevOpsDays in 2009. He is credited with coining “DevOps” and co-authoring the renowned DevOps Handbook. In the past, Patrick has collaborated with esteemed technology organizations such as Atlassian and Snyk. Currently, he wears dual hats as the Vice President of Engineering and a Distinguished Engineer at Showpad.

Is LangChain Free?

Yes, LangChain is an open-source framework, free to use and modify.

How Does LangChain Work?

LangChain is a meticulously designed open-source framework tailored to simplify the creation of applications powered by large language models (LLMs). First, you will need to have a language model or you can use a public language model or train your own.

Once you have a language model ready, you can start building applications. The framework offers an array of tools and APIs, simplifying the process of connecting language models to diverse data sources, interacting with their environment, and constructing intricate applications.

LangChain operates by linking together various components, referred to as links, to establish a workflow. Each link in the chain performs a task and is connected in a sequence, enabling the output of one link to serve as the input for the subsequent one. This sequential linking allows the chain to execute intricate tasks by combining simple tasks seamlessly.

See LangChain’s get started docs for more information.

Does ChatGPT use LangChain?

ChatGPT utilizes LangChain to create custom artificial intelligence (AI). This allows for an individualized experience.

Deploy AI Models on Amazon SageMaker using Pulumi Python IaC

Kat Morgan — Mon, 11 Sep 2023 00:00:00 +0000

Running models from Hugging Face on Amazon SageMaker is a popular deployment option for AI/ML services. While the SageMaker console allows for provisioning these cloud resources, this deployment pattern is labor intensive to document and vulnerable to human errors when reproducing as a regular operations practice. Infrastructure as Code (IaC) offers a reliable and easy to duplicate deployment practice. By developing this IaC with Pulumi, practitioners can choose to write their infrastructure code in Python and seamlessly develop both AI application code and IaC code in the same language.

In this short tutorial we will deploy a publicly available Meta AI LlaMa 2 based model from Hugging Face, on Amazon SageMaker. Then we will test it with a natural language prompt using a short Python script.

Pulumi makes AI/ML Easy?

We will use the sagemaker-aws-python Pulumi Template to bootstrap our Python SageMaker IaC. Templates allow you to quickly bootstrap new Pulumi projects from a working scaffold which you can then customize further to your needs. Out of the box, this template provisions Amazon IAM Roles to assign SageMaker privileges, CloudWatch alarms to alert in case of latency or error spikes for the endpoint, and of course a Meta AI LlaMa 2 based LLM (NousResearch/Llama-2-7b-chat-hf) hosted on Hugging Face.

Requirements

Instructions

Let’s begin by logging into Pulumi Cloud:

# There are many ways to store Pulumi state, here we use Pulumi Cloud
# Other state backends include S3, local file, and more
pulumi login

If you’re unsure of which Pulumi state backend you’re using, you can check by running pulumi whoami (or pulumi whoami -v for more info).

2. Prepare a new Pulumi project

Here we create our new Pulumi project directory and populate it from the sagemaker-aws-python Pulumi template

# Create a new directory & change directories into it
mkdir newSageMaker && cd newSageMaker

# Start your project from the sagemaker-aws-python template
# Follow along with the prompts to create your new project and initialize a stack
pulumi new sagemaker-aws-python

While creating a new project from the sagemaker-aws-python Pulumi template, you will be prompted for a project name, description, stack name, and Amazon Web Service Region. You can proceed with defaults, or supply your own.

After responding to all prompts, the pulumi new command will proceed to setup a Python Virtual Environment (venv) and download all dependencies into the venv.

3. Deploy your model as a new SageMaker endpoint

This step may take between 10 and 20 minutes while Amazon builds your infrastructure and deploys the configured model. You can follow along in the console as resources are provisioned, or open the link displayed in terminal to view the deployment status and other stack information in Pulumi Cloud.

pulumi up

4. Try your new SageMaker endpoint

Once your stack has finished deploying, use this rudimentary Python snippet to test the deployed SageMaker endpoint.

NOTE: Notice that we are using us-east-1 in this script. Be sure to change the region in Python to match the region you deployed the SageMaker endpoint into.

First, save the following python snippet as test.py:

import json, boto3, argparse

def main(endpoint_name):
 client = boto3.client('SageMaker-runtime', region_name='us-east-1')
 payload = json.dumps({"inputs": "In 3 words, name the biggest mountain on earth?"})
 response = client.invoke_endpoint(EndpointName=endpoint_name, ContentType="application/json", Body=payload)
 print("Response:", json.loads(response['Body'].read().decode()))

if __name__ == "__main__":
 parser = argparse.ArgumentParser()
 parser.add_argument("endpoint_name")
 main(parser.parse_args().endpoint_name)

Once you have the test.py script created, source the Python virtual environment named venv (created automatically by Pulumi) and run the script using the name of your new endpoint, taken directly from the Pulumi stack output:

# Source the venv
# On Linux & MacOS
source venv/bin/activate

# Execute test.py
python3 test.py $(pulumi stack output EndpointName)

5. Cleanup all resources

Finally, when you’re finished with testing your Hugging Face model on SageMaker, you can easily clean up un-used resources with one easy command.

pulumi destroy

Conclusion

To recap, in a few commands, we created a new Pulumi Python project from a ready-to-roll template, deployed an LLM endpoint on Amazon SageMaker, and tested it with a short Python script to generate a response from our model!

AI and ML is rapidly becoming a necessity with every new day. It may appear daunting or out of reach at first glance, but with the power of IaC written as Pulumi Python programs, getting started has never been easier.

If you followed along then tell us how it worked out for you! We would love to know what you are looking forward to, or if you have ideas for future installments of the Pulumi Python #PulumiMLOps series!

Join us on Twitter, and on the Pulumi Community Slack to decide what #PulumiMLOps we take on next!

The Real AI Challenge is Cloud, not Code!

Kat Morgan — Wed, 30 Aug 2023 00:00:00 +0000

The AI industry is stealing the show as tech’s goldrush of the ’20s. Just looking at ChatGPT’s record setting user growth, and rapid 3rd party integration by top brands, it is not surprising the hype suggests this is the beginning of a major digital transformation.

However, using AI/ML in your own products has some major challenges and obstacles. Below is a diagram of the end to end workflow of building and using an AI model: preparing the data, training a model, fine-tuning a model, hosting and running a model, building a backend service to serve the model, and building the user interface that interacts with the model. Most AI engineers are only involved in a few steps of the process. However, there is one challenge that is common across the entire workflow: creating and managing the cloud infrastructure is hard.

Figure 1. A sneak peak at the map of ML development and product path

Training a model requires spinning up hyper scale GPU clusters. Preparing and consuming the training or fine-tuning data requires managing data warehouses and buckets of object storage. Building an AI backend service requires packaging the model, deploying and scaling the model across compute infrastructure, creating and managing the network infrastructure like load balancing, and managing the vector databases. Building great frontend user experiences requires provisioning and managing CDNs and web application services. Suffice to say: The most difficult challenge that we face with AI today is a problem of cloud orchestration.

Most of this cloud orchestration today is done manually or with complex scripts. This is usually fraught with errors and simply doesn’t scale. Too much time is spent making cloud infrastructure work instead of focusing on actual AI/ML tasks. Infrastructure as code (IaC) is a key component to solving the cloud orchestration challenge present in AI/ML. By applying the rigor and precision of software development practices to cloud operations, IaC offers AI/ML professionals an efficient, reliable, and predictable way to develop at the highest velocity. Pulumi is an open source infrastructure as code platform that allows engineers to use any programming languages, including Python, to provision and manage AI infrastructure.

This blog post is the first installment of a series on AI Infrastructure as Python. We will start this journey with building an AI backend service for a chatbot. In this post, we will deploy a LLM model behind an API backend running on compute infrastructure in the cloud. This entire infrastructure pipeline will be deployed using IaC written in Python. The series will follow this AI backend service through its evolution into a robust and resilient cloud native service.

The most difficult challenge that we face with AI today is a problem of cloud orchestration

Before we get into the details of using Pulumi to deploy a private LlaMa 2 API chatbot service—in 30 minutes or less, of course—let’s start with some quick terminology/context:

Katwalk Server is a demo AI application written to host your choice of LLM either locally or in the cloud, and serve it as an OpenAI API compatible service. As a practical, hands-on introduction to IaC for the MLOps space, this project aims to demystify the concepts and showcase the benefits Pulumi can bring to your AI/ML projects.
Large Language Models or LLMs (like ChatGPT) are machine learning models trained to generate human-like conversational text which have already become essential tools for businesses and individuals, transforming every day tasks and how we interact with technology.
LlaMa is an LLM created by Meta, available for free for research and commercial use.

Figure 2. Or just let ChatGPT describe itself

Deploy a private LlaMa 2 API chatbot service … in 30 minutes or less!

If you are ready to try the #MLOpsChallenge along with us, then let’s double check a few requirements before starting the clock on our 30 minute chatbot challenge. While Pulumi supports many languages, we chose Python for this project, as Python is a familiar language in the AI industry.

Requirements

Choose one of the following deployment platforms. (You can actually choose more than one, if you’d like!)

Now, for the real fun, let’s start the timer and deploy our own chatbot! Note that the instructions below are for Runpod; instructions for the other two deployment platforms are found in the GitHub repository.

1. Prepare your Pulumi IaC directory

First we need to clone our IaC repository from GitHub and prepare our Python environment.

# Clone the repository and cd to the pulumi iac directory
git clone https://github.com/pulumiverse/katwalk && cd katwalk/pulumi

# Create and initialize the python virtual env
python3 -m venv venv && source venv/bin/activate

# Install python dependencies
python3 -m pip install -r requirements.txt

Next, let’s setup our state and secrets store in Pulumi Cloud.

# There are many ways to store Pulumi state, here we use Pulumi Cloud
# Other state backends include S3, local file, and more
pulumi login

# Initialize your stack
# Here we name the stack "dev"
pulumi stack init --stack dev

3. Configure required credentials

Configure your Huggingface credentials. If you open the Pulumi.${stackName}.yaml file, you can see the encrypted secrets in this plain text YAML configuration.

# Set Huggingface.co username
pulumi config set hfUsername <huggingface_username>

# Configure token as secret
pulumi config set --secret hfToken <huggingface_api_token>

# Set the model that you want to download
pulumi config set hfModel "meta-llama/Llama-2-7b-chat-hf"

4. Deploy in the cloud on Runpod.io

While there are three providers already supported in this codebase, this post illustrates deploying to Runpod.io, a GPU and ML dedicated cloud catering to AI workloads and development tasks.

# This enables the IaC to deploy Katwalk Server
pulumi config set deploy True

# Set the deploy runtime to runpod to deploy on the Runpod cloud
pulumi config set runtime runpod

# Configure your Runpod API Token as a secure Pulumi secret
# API token available at https://www.runpod.io/console/user/settings
pulumi config set --secret runpodToken <your_runpod_api_token>

# Finally, run `pulumi up` to deploy!
pulumi up

# When done, you can 'destroy' the stack to deprovision your deployment
pulumi destroy

There’s currently a bug in the code that does not return the fully-qualified domain name (FQDN) from Runpod for your chatbot instance. To get the FQDN (referenced later in this post as RUNPOD_FQDN), you’ll need to log into Runpod and retrieve that information.

5. Build the Katwalk container

While there are many other ways to orchestrate ML services in the cloud and locally, in this project we are taking ownership of the full application and infrastructure story. In fact, all three supported providers use the same container image to serve the LlaMa LLM API Service, and you can build it yourself just like this:

# Set image build to True
pulumi config set imageBuild True

# Let's build this without pushing to a registry for now
pulumi config set imagePush False

# Give your katwalk image a project name
pulumi config set registryProject $(whoami)

# Now build the image!
pulumi up

6. Talk to your chatbot!

Once your Katwalk LlaMa server is running, you can interact with the API by sending prompts as a JSON payload. In future posts we will orchestrate a chat frontend for our chatbot but for now we’ll use a curl command. You can also substitute an API developer tool such as Postman or Insomnia!

# send a prompt with curl
curl -s -X 'POST' 'http://${RUNPOD_FQDN}/v1/chat' -H 'accept: application/json' -H 'Content-Type: application/json' -d '{"prompt": "Write a short story about a robot."}'
{
 "text": [
 "\nOnce upon a time, there was a little robot named R2. R2 was a friendly and curious robot who lived in a big city. One day, R2 decided to go on an adventure. He set out to explore the city and learn about all the different things he could see and do.\nAs R2 explored the city, he met all kinds of people. Some were kind and welcoming, while others were scared or suspicious of him. Despite this, R2 continued to be friendly and curious, always asking questions and trying to learn more about the world around him.\nOne day, while R2 was exploring a busy market, he saw a group of people gathered around a little girl who was"
 ]
}

Figure 3. A prompt and response from meta-llama/Llama-2-7b-chat-hf hosted on Katwalk Server

In a world where “data is the new black gold” and tokens are money, deploying a chatbot in 30 minutes is not just interesting, it is a novel capability quickly becoming a necessity.

Platform engineering, DevOps, or MLOps may be the next hardest step in building the AI future. Pulumi offers a reliable path to higher velocity AI App development and MLOps. Together with Python-based IaC, it empowers the MLOps community to build equity in common AI platform code, and share IaC in a familiar language.

If you followed along then tell us how it worked out for you! We would love to know what you are looking forward to, or if you have ideas for future installments of the Pulumi Python + MLOps series!

Join us in the Pulumi Community Slack or in the Pulumiverse Community Katwalk Github repository to decide what #aiChallenge we tackle next!

Figure 4. Midjourney generated image of an imagined machine learning research engineer working among cables, computers, keyboards, and screens, in the glow of electronic lights. Image was generated from a prompt ~80% produced by ChatGPT based on this blog post content and a few superficial details about the author

Pulumi Blog: Kat Morgan

The Pulumi 'Push to start' GitOps Experience

Index

New technology learning advice

Tech learning hacks:

A DevOps journey

DevOps day 0

What’s in a DevOps day 0?

Prerequisites

Commandline Utilities:

Cloud-native … cloud developer environment?

DevOps day 1

~$ pulumi new --ai

Step 1. Login to Pulumi Cloud

Step 2. Create a new program

Step 3. YOLO for fun and profit!

DevOps day 2

Conclusion

Did it deploy?

What’s in the recipe?

Git Code Repository

Direnv: Automatic Environment Variables

Pulumi Devcontainer

Pulumi Cloud

KinD: Kubernetes-in-docker

Now let’s create the deployment code!

Cleanup

LangChain for DevOps: Learn LLM & GenAI for Dev, Sec & Ops

What is LangChain used for?

Why is LangChain becoming popular?

LangChain Accelerating DevOps

How to Use LangChain to Learn LLMs and GenAI for Dev(Sec)Ops

Developer

Operations

Security

Real-World Applications of LangChain AI in DevOps

Monitoring and Anomaly Detection with LangChain AI

Future: AI, Apps, and DevOps

Accelerating Development with AI-Powered Tools

Security and Vulnerability Management with AI

Leveraging Machine Learning in DevOps

Implementing AI models into DevOps: Key Considerations

Conclusion

Frequently Asked Questions

Who is Patrick Debois?

Is LangChain Free?

How Does LangChain Work?

Does ChatGPT use LangChain?

Deploy AI Models on Amazon SageMaker using Pulumi Python IaC

Pulumi makes AI/ML Easy?

Requirements

Instructions

1. Login to Pulumi Cloud and initialize stack

2. Prepare a new Pulumi project

3. Deploy your model as a new SageMaker endpoint

4. Try your new SageMaker endpoint

5. Cleanup all resources

Conclusion

The Real AI Challenge is Cloud, not Code!

The most difficult challenge that we face with AI today is a problem of cloud orchestration

Deploy a private LlaMa 2 API chatbot service … in 30 minutes or less!

Requirements

1. Prepare your Pulumi IaC directory

2. Login to Pulumi Cloud and initialize stack

3. Configure required credentials

4. Deploy in the cloud on Runpod.io

5. Build the Katwalk container

6. Talk to your chatbot!

`~$ pulumi new --ai`