Pulumi Blog: Ml

How AI is Transforming DevOps: AI Talks for DevOps Insights

Sara Huddleston — Mon, 13 Nov 2023 22:05:34 +0000

The integration of artificial intelligence (AI) with DevOps signals a new era in software development. DevOps possesses unique characteristics and needs that make it exceptionally compatible with AI augmentation. Given that code fundamentally relies on language, and large language models (LLMs) serve as the core of GPT functionality, these models are particularly well-suited for tasks such as code generation. This article unwraps the topics addressed during our “AI: Friends or Foe | AI Talks for DevOps” event in San Francisco.

How will AI Impact the Future of DevOps

In the Fireside chat, Luke Hoban expressed his enthusiasm for incorporating AI into DevOps, emphasizing its potential utilization by both DevOps professionals and the development team. In contrast, Nana Janashia, TechWorld with Nana, adopted a more cautious stance, drawing attention to potential risks, particularly regarding reliability.

What is DevOps in AI

When considering the adoption of AI and AI engineering, such as prompt engineering and other specific disciplines, AI engineers rely heavily on the cloud. Many AI-based systems are built around cloud-native tools, practices, and managed services.

The extensive use of cloud-native primitives means that AI is fundamentally a cloud adoption story. Consequently, established DevOps and infrastructure as code tooling and best practices play an important role in AI engineering, in what is being done, and in leveraging its use.

What is AI in DevOps

Integrating AI in DevOps entails employing machine learning (ML) and other artificial intelligence technologies to automate and optimize various software development and delivery aspects, including tasks such as automating testing and deployment processes and refining resource management.

The incorporation of AI in DevOps can result in:

Enhanced speed
Improved resource management
Increased reliability throughout the software development lifecycle

Consequently, organizations may experience swifter deployments, reduced errors, and heightened overall productivity. By substituting manual processes with automation and AI-powered solutions, DevOps teams can elevate product quality and more efficiently oversee their systems.

How AI is Currently Used in DevOps

Luke Hoban at the San Francisco Pulumi Group meetup discussing the usage of AI, during the AI Talks for DevOps

DevOps significantly impacts all stages of the development and operations lifecycle. It has helped break silos and optimize to increase developer velocity and productivity, creating a positive impact on the ops side.

AI assists DevOps teams to refine their operations by detecting inefficiencies and triggering warnings as soon as issues appear. This technology also enables real-time observation of systems and applications, permitting prompt actions from operations personnel and minimizing downtime.

AI-driven automation can also help the software development process with code generation. The operations teams from both departments quickly pinpoint any issues in their workflows while speeding up decision-making processes.

In the long term, the AI may use cloud credentials to create and autonomously automate deployment and management and perform DevOps tasks on behalf of the DevOps or platform engineering team. In the near term, it is used to assist humans, from DevOps professionals to developers.

“In the near term, AI usage in DevOps is about how do you keep the human in the loop, how do you use it as a learning tool, as a code generation tool, and for auditing to provide actionable notifications, all of which are still very human-dependent. There is so much more it can be used for. Currently, it’s a tool serving humans. But it is fascinating to imagine it in the future when it can be trusted to be an autonomous agent.” - Luke Hoban, CTO of Pulumi and co-creator of TypeScript

So, how does this technology bring about these advancements? Two main ways are by streamlining development tasks and improving monitoring and security protocols.

Streamlining Development Workflows

AI has the potential to significantly change the software development life cycle by automating routine processes, using resources efficiently, and offering solutions for improved code. DevOps teams can leverage AI-driven tools to increase effectiveness when managing their assets while also improving code quality, leading to faster project deliveries.

For instance, AI can identify any slowdowns during progress and recommend improvements, analyze root causes, and help make judicious decisions regarding resource allocation, ultimately resulting in efficient delivery of software products.

Enhancing Monitoring and Security

The usage of AI significantly boosts DevOps monitoring and security. With the assistance of AI tools, operations teams can use machine learning algorithms to examine a massive amount of data to protect their applications from potential risks and enhance reliability. These algorithmic solutions provide valuable advice for development personnel, which helps strengthen protection measures while optimizing performance simultaneously.

How Can a DevOps Team Take Advantage of Artificial Intelligence

A DevOps team can leverage AI in several ways to enhance efficiency, automation, and overall productivity. Here are some key strategies for a DevOps team to leverage AI:

Code Generation and Learning

AI can be a powerful tool to make the development process much smoother. It can help engineers focus on what matters instead of searching for documentation on websites, developer forums, or search engines for answers to help solve specific problems. The perfect code snippet is often hard to find or may not even exist.

Generative AI, powered by LLMs, can help developers and DevOps engineers quickly discover, learn, and, for example, use new cloud infrastructure APIs.

Pulumi AI response to Run metabase in Azure, using TypeScript

You can see it in practice using Pulumi AI. You can use natural language to ask for a specific cloud infrastructure goal and obtain a corresponding Pulumi program in your preferred programming language. Moreover, Pulumi AI facilitates the iterative refinement of your cloud infrastructure, incorporating new features, enhancements to security and performance, resolution of correctness issues, and clarification of requirements. Try Pulumi Neo.

Intelligent Resource Management

Platform and developer teams are spinning up resources across many cloud accounts within multiple cloud providers. It can be almost impossible to aggregate and search your infrastructure across these accounts since every cloud and every account is (by design!) a walled garden.

Often, to answer key questions about their business, such as “Which team has the largest cloud footprint?”, teams need to combine data across various clouds, cloud accounts, and SaaS tools. This prevents organizations from getting quick answers to operational questions and barriers to building aggregations to derive insights. AI can provide actionable insights over infrastructure with natural language processing (NLP) queries.

“I can get developers using IaC immediately with Pulumi Deployments and its GitHub integration, while Pulumi Insights [Resource Search] makes it really easy to find idle developer environments that need to be shut down, which reduces our cloud costs” - Santosh Dornal, Head of Software Test & DevOps, Alkira.

When using Pulumi Cloud, you can leverage AI Assist. You can provide a plain English (Spanish, French, or Japanese) request to express queries where you might not know the exact syntax, type tokens, or package names. AI Assist makes it easier by providing answers to queries like:

show me all s3 buckets not tagged in production
show azure and azure native security groups
show all AWS VPCs

Remediation Tool: Continuous and Automatic Compliance

Defining and managing security rules, criteria, and conditions are key to enabling security at scale. Policy as Code, or Compliance as Code, has the ability to verify and spot problems before deploying the infrastructure. When policies are written with code, you can apply software development practices such as testing, automated deployment, and version control.

AI can be valuable not just in anomaly detection but also in remediation, which doesn’t just warn or error but can change the infrastructure directly to remediate compliance gaps. With Pulumi, Remediation policies can be configured and applied across any subset of stacks within an organization.

Consider the unfortunate incident at GitLab, marked by a disastrous sequence of events resulting in the loss of segments of production databases and substantial user data.

In this scenario, AI could have swiftly pinpointed the root cause and proposed a rapid solution, potentially mitigating the extent of damage before it escalated.

While the benefits of writing policies as code are evident for DevOps engineers, the organizational benefits are even more significant.

Enhance Collaboration and Communication

Nowadays, many professionals with DevOps in their titles are held accountable for the multiple aspects of Dev + Ops. At its core, DevOps is a cultural movement that removes barriers among teams and promotes collaboration and shared responsibility between dev and ops.

DevOps teams have seen great benefits from AI-driven tools, particularly in improving transparency, encouraging feedback, providing quick answers to common queries, and automating routine tasks. Many major companies, like Microsoft, use AI to build efficient workflows that make better decisions, leading to increased business outcomes. This includes smoother application delivery, a prime function of DevOps initiatives.

Improved Product Delivery

Companies incorporating AI into their DevOps strategy reap the rewards of accelerated product delivery. This has been made possible through improved workflows, automated testing, and strengthened security measures, leading to greater efficiency in software development by allowing development and operations teams more scope for dealing with complex tasks using machine learning models. They can make better decisions, resulting in faster product delivery and enhanced business outcomes.

By streamlining existing systems within this context, customers can benefit from superior products delivered faster than before while companies provide and reinforce their business value.

Planning and Cost Optimization

Implementing AI algorithms or AI tools that analyze usage patterns can assist in predicting future resource needs and optimize cloud infrastructure costs. This also ties in with intelligent resource management mentioned before, which can enhance cost efficiency, optimize resource allocation, and prevent unnecessary expenses.

Top 5 Skills for DevOps Engineers in the AI Era

Adora Nwodo with Nana discussing skills that DevOps engineers will need to learn to make AI work

DevOps engineers must open new skills, mainly in cloud services, data, and AI/ML.

Programming Languages: Although DevOps don’t usually need to know general programming languages in-depth, they should know what is most used by application teams. It’s a nice-to-have to find errors in CI/CD pipelines and fix them themselves as needed, but it is a must-have if the goal is to work MLOps.
Infrastructure as Code: Automation is important for DevOps and critical for MLOps. IaC, using familiar languages, has been used since the early days of MLOps and is a key component to solving the cloud orchestration challenge present in AI/ML. Dive into more detail in The Real AI Challenge is Cloud, not Code!
Continuous Integration and Delivery: CI/CD is mandatory knowledge, and it must be understood in detail to predict data preparation code, AI/ML code, CI/CD code, and infrastructure-related code.
Custom containers: As a DevOps professional, you are expected to understand Docker and/or Kubernetes concepts and best practices.
Orchestration: This is the holy grail of Ops and your key to unlocking AI.

How to Overcome AI for DevOps Challenges

The integration of AI in DevOps brings forth several challenges that organizations must navigate to leverage the full potential of these technologies. Here are some key challenges and potential strategies to overcome them:

1. Skills and Expertise

Challenge: AI & ML expertise is often scarce within DevOps teams, leading to a skill gap in implementing and managing AI-driven tools.

Overcoming Strategy: Organizations will need to hire professionals with AI and ML expertise, or existing team members will need training in machine learning, becoming proficient in MLOps combined to DevOps principles. Collaboration between DevOps and data science teams can also bridge the knowledge gap.

2. Integration Complexity

Challenge: Integrating AI seamlessly into existing DevOps workflows can be complex, given the diverse technologies and processes involved.

Overcoming Strategy: Prioritize a phased integration approach, gradually introducing AI capabilities into specific stages of the DevOps pipeline. This allows teams to adapt without disrupting established processes.

3. Scalability

Challenge: As AI applications within DevOps expand, ensuring scalability becomes crucial to accommodate growing workloads.

Overcoming Strategy: Design AI solutions with scalability in mind, utilizing cloud resources and containerization. Regularly assess performance and adjust infrastructure to meet evolving demands.

4. Data Quality and Availability

Challenge: AI models rely heavily on high-quality and readily available data. Inconsistent or inadequate data can lead to inaccurate predictions or suboptimal performance.

Overcoming Strategy: Prioritize data quality and establish robust data pipelines. Conduct regular audits to ensure data accuracy and implement strategies for handling missing or incomplete data.

5. Cultural Resistance

Challenge: Resistance to change or skepticism about the benefits of AI within DevOps and in software development can impede successful integration.

Overcoming Strategy: Foster a culture of innovation and continuous improvement. Provide clear communication about the benefits of AI, involve team members in the decision-making process, and showcase success stories to build confidence.

Navigating these challenges requires a thoughtful and strategic approach, focusing on collaboration, skill development, and the gradual incorporation of AI into existing DevOps practices. Regular evaluation and adaptation are essential to ensure a successful and sustainable integration of AI in the DevOps environment.

Conclusion

AI in DevOps can transform software development by making workflows more efficient, monitoring and security stronger, and overall effectiveness increased. Organizations that embrace AI/ML aim for better collaboration between teams, automating and intelligently managing infrastructure, reducing time-to-market, and increasing developer productivity and cost savings due to streamlining processes within their development lifecycle. AI and ML knowledge will become necessary skills expected to be applied to enhance and benefit the development, security, and operation teams as a whole. Follow the Pulumi Python + AI/ML series to upskill and get ahead:

Frequently Asked Questions

Can AI replace DevOps?

DevOps has not been taken over by AI yet. Rather, its use has made a difference in code quality and accelerated certain processes. Improvements to the quality of coding have been seen as one of the main benefits brought on through integration with artificial intelligence.

What is an example of AI in DevOps?

AI can be utilized to make DevOps processes more efficient by creating test cases and pipelines, assessing infrastructure configurations for security concerns, and deploying applications. This technology includes machine learning algorithms, natural language processing systems, computer vision capabilities, chatbot programs, and virtual assistants. All are designed with one goal in mind – optimization of operations within development teams.

What is the future of DevOps?

Operations teams collaborating with developers to create quality products quickly and effectively is the goal for DevOps moving forward. The vision of this process involves both sides working together, resulting in an improved system and increased efficiency.

LangChain for DevOps: Learn LLM & GenAI for Dev, Sec & Ops

Sara Huddleston — Thu, 26 Oct 2023 00:00:00 +0000

The emergence of DevOps revolutionized software development. Now, with AI powered tools like LangChain, these transformations are being accelerated. Unsurprisingly, our distinguished speaker at the launch of Pulumi’s in-person AI Talks, Patrick Debois, who coined the term “DevOps,” has recently tuned into LLM and GenAI Ops using the Langchain framework.

This article unwraps the topics addressed during our “Generative AI, Apps, and DevOps | AI/ML Talks” event in Seattle, focusing on Langchain and GenAI for DevOps. You’ll be able to watch Patrick Debois teach LangChain to learn LLMs and GenAI, with code examples and lessons that are easily understood by traditional software engineering.

What is LangChain used for?

LangChain is an open-source Python framework designed to empower developers to construct robust generative AI applications. It facilitates the integration of advanced AI models such as OpenAI’s GPT, Google’s Gemini, and Meta’s LLaMA. This versatile tool finds application in developing retrieval-augmented chatbots and other personalized assistants, utilizing technologies like ChatGPT. Furthermore, it enables tasks such as question-answering (GQA), summarization, code comprehension, API interactions, data cleansing, and much more.

Why is LangChain becoming popular?

The landscape of AI development is rapidly evolving, and LangChain is packed with incredible features for building LLM applications and tools. Developers with a wide range of expertise use LangChain for tasks such as managing interactions with language models, establishing seamless connections between diverse components, and integrating external resources like APIs and databases effortlessly.

The LangChain platform hosts a diverse collection of APIs seamlessly integrated into applications. This enables developers to incorporate advanced language processing functionalities without the need for laborious construction from scratch.

LangChain Accelerating DevOps

Patrick Debois presenting the workshop Dev, Sec & Ops meet LangChain

LangChain can streamline the integration of large language models and data sources. By helping bring in AI capabilities, DevOps can revolutionize software development life cycles by making it easier for teams to handle challenges while optimizing resource allocation and usage through their cutting-edge, AI-powered solutions. This makes the automation of manual processes easier and the development of new resources far more accessible with faster utilization rates by leveraging artificial intelligence. Significantly important as “Developer Velocity and Productivity” become business performance metrics across various industries.

The features included in LangChain include chain interface memory functions for scalability within apps, control over dataflow, as storage – making code quality more reliable, plus seamless machine learning models integration, which allows them to build custom automated AI programs quickly. Thus providing users with an advanced but easy way of creating sophisticated solutions by combining their knowledge of coding and machine training models together.

How to Use LangChain to Learn LLMs and GenAI for Dev(Sec)Ops

Patrick teaches in the way he would like to be taught. The workshop mainly uses the Langchain framework and basic Python knowledge. OpenAI will also need to be installed. You’ll learn essential skills in multiple use cases, applying Langchain for LLM application Development.

You can see the breakdown of the lessons per role below, and find the code examples in the Learning LLMs and GenAI for DevSecOps repo.

Developer

Calling a simple LLM using OpenAI
Looking at debugging in Langchain
Chatting with OpenAI as a model
Using prompt templates
Use Docloader to read your local files and prepare them for the LLM
Explain the calculation and use of embeddings
Understand how splitting and chunking are important
Loading embeddings and documents in a vector database
Use a chain for Questions and Answers to implement the RAG pattern (Retrieval Augmented Generation)
Show the use of OpenAI documentation to have the llm generate calls to find real-time information
Implement an Agent and provide it with tools to get more real-time information

Operations

Find out how many tokens you are using and the cost
How to cache your calls to an LLM using exact matching or embeddings
How to cache the calculation of embeddings and run the calculation locally
Run your own local LLM (using Ollama)
Track your calls and log them to a file (using a callback handler)
Impose output structure (as JSON) and have the LLM retry if it’s not correct

Security

Explain the OWASP top 10 for LLMS
Show how simple prompt injection works and some mitigation strategies
How to detect prompt injection using a 3rd party model from Hugging Face
Detect project injection by using a prompt
Check the answer LLMs provide and reflect if it is ok
Use a Hugging Face model to detect if an LLM output was toxic
Show a simple prompt for asking the LLM’s opinion on Kubernetes and Trivy vulnerabilities

Real-World Applications of LangChain AI in DevOps

Vector similarity search in a RAG application. Credit to LangChain and Neo4J team

LangChain AI has practical applications in DevOps, particularly related to question-answering using documents as context, extraction, evaluation, and natural language processing tasks, all powered by AI tools. An interesting scenario is implementing a knowledge graph based RAG application with LangChain to support the DevOps team.

ChatGPT based on LLMs enabled with these AI tools is one of the foremost implementations at present. Tracking capabilities through LangChain enables developers to determine which prompts are more effective, thus supporting what artificial intelligence offers when integrated into development operations.

Monitoring and Anomaly Detection with LangChain AI

Monitoring LLM performance with LangChain and LangKit. Credit to WhyLabs

LangChain AI provides an efficient approach to DevOps through continuous monitoring, and anomaly detection. LangChain’s integration in DevOps allows for intelligent monitoring of various system parameters, application performance metrics, and log files. It can process large volumes of data generated by applications and infrastructure components. Through natural language processing (NLP) techniques, LangChain can interpret logs, error messages, and system alerts in real-time.

LangChain’s AI-driven anomaly detection capabilities are pivotal in identifying deviations from expected behavior in the monitored data. By analyzing historical data patterns, LangChain can learn the normal behavior of the system and its components. When the system encounters an unusual event or behavior, LangChain can promptly detect it as an anomaly. These anomalies could be security threats, performance bottlenecks, code quality, or any irregularities within the system.

Keep in mind that monitoring LLM performance is important to ensure you can rely on the model’s accuracy and relevance in real-world applications. Our customer WhyLabs has a great article showing the significance of monitoring LLMs performance with LangChain and how to get started with monitoring.

Future: AI, Apps, and DevOps

Platform engineering, DevOps, or MLOps may be the next steps in the AI-driven software development future. AI tools are the gateway to giving developers access to big language models that allow them to create highly sophisticated solutions.

Accelerating Development with AI-Powered Tools

AI-powered tools are revolutionizing the development process, enabling DevOps teams to access intelligent recommendations and improve code quality or obtain suggestions on new ways to accomplish goals. An example is the usage of Pulumi Neo.

Tools like LangChain, Amazon CodeGuru, and Pulumi AI provide actionable insights for efficient resource utilization while enforcing policy and compliance best practices. An IaC that leverages AI empowers teams to automate and more effectively manage infrastructure.

Developers can reap multiple benefits from higher quality assessment of code lines, instantaneous detection of policy and security issues, useful cues on how to resolve problems, and improving code quality, productivity, and time management.

AI has greatly impacted optimization during the review stage, providing developers with better insights into performance issues, code effectiveness, and resource utilization without any extra effort.

Security and Vulnerability Management with AI

The marriage of AI, machine learning, and DevOps/DevSecOps practices revolutionizes security and vulnerability management in software engineering. AI-enabled tools technologies not only strengthen the defenses of software systems but also enable organizations to stay one step ahead in ensuring successful deployments by removing manual processes through automation, using single sources of truth, enforcing least-privileged access through role-based access controls, and automated remediation to correct configuration violations like auto-tagging, removing Internet access, and enabling storage encryption.

DevSecOps can foster collaboration between developers, security, and operations and still enhance security and compliance best practices using Artificial Intelligence (AI) and Machine Learning techniques.

Leveraging Machine Learning in DevOps

Applying Machine Learning in DevOps is a game-changer. Not only does it automate and enhance various processes but also brings an intelligent, predictive, and adaptive dimension to development workflows. By embracing these technologies, development teams can achieve higher efficiency, improve code quality, optimize performance, and seamlessly integrate with artificial intelligence, thereby shaping the future of software engineering.

ML models can evaluate past outcomes from compilation/builds plus performance metrics within an operation so that developers may leverage these insights when writing code accordingly with machine learning algorithms, allowing them to identify vulnerabilities or issues faster through root cause analysis while ameliorating system security against outside attacks like hackers or DDOS activity thus boosting overall system productivity thanks to AI implementation into existing processes.

Implementing AI models into DevOps: Key Considerations

When considering AI, it’s essential to assess data quality, integration complexity, cost savings, and ethical/legal considerations:

Ensuring the data used for training these AI tools are precise, exhaustive, and up-to-date will help gain more reliable results.
An important aspect of the implementation process is integrating existing DevOps instruments and guaranteeing scalability, depending on the company’s necessities.
Ensuring that all laws applicable when working with such technology.

Conclusion

AI and ML are rapidly becoming a necessity, which can greatly enhance and benefit Development, Security, and Operation teams. At first glance, it may appear daunting or out of reach, but tools like Langchain and Pulumi can make it easier to get started. Follow the Pulumi Python + AI/ML series to upskill and get ahead:

Frequently Asked Questions

Who is Patrick Debois?

Patrick Debois is a highly accomplished technologist whose expertise spans the domains of Development (Dev), Security (Sec), and Operations (Ops). Acknowledged as a respected confidant within the developer, security, and operations communities, Patrick is presently deeply engrossed in the realm of Artificial Intelligence and Machine Learning, continually pushing the boundaries of his technical acumen.

He was the organizer of the inaugural DevOpsDays in 2009. He is credited with coining “DevOps” and co-authoring the renowned DevOps Handbook. In the past, Patrick has collaborated with esteemed technology organizations such as Atlassian and Snyk. Currently, he wears dual hats as the Vice President of Engineering and a Distinguished Engineer at Showpad.

Is LangChain Free?

Yes, LangChain is an open-source framework, free to use and modify.

How Does LangChain Work?

LangChain is a meticulously designed open-source framework tailored to simplify the creation of applications powered by large language models (LLMs). First, you will need to have a language model or you can use a public language model or train your own.

Once you have a language model ready, you can start building applications. The framework offers an array of tools and APIs, simplifying the process of connecting language models to diverse data sources, interacting with their environment, and constructing intricate applications.

LangChain operates by linking together various components, referred to as links, to establish a workflow. Each link in the chain performs a task and is connected in a sequence, enabling the output of one link to serve as the input for the subsequent one. This sequential linking allows the chain to execute intricate tasks by combining simple tasks seamlessly.

See LangChain’s get started docs for more information.

Does ChatGPT use LangChain?

ChatGPT utilizes LangChain to create custom artificial intelligence (AI). This allows for an individualized experience.

Deploy an AI/ML Chatbot Frontend on Vercel with Pulumi

Adora Nwodo — Wed, 13 Sep 2023 00:00:00 +0000

The process of taking an idea and turning it into reality has been nothing short of extraordinary since we started innovating with Artificial Intelligence. With this technology, machines learn about and communicate with people, while also helping us in ways we never could have imagined only a few years ago. If you’ve been following along, you might recall that the real AI challenge is cloud, not code where we used Python and Pulumi to a chatbot API (named katwalk).

But our AI adventure hasn’t ended there; it’s only just begun.

Now, imagine this: A Vue.js chatbot frontend, seamlessly integrating human interaction and the AI chatbot API. This blog post is part of the series on AI Infrastructure as Python. We’d delve into the creation of a captivating Vue.js chatbot frontend, and with Pulumi, we’ll deploy it to Vercel.

Figure 1. Architecture diagram

The complexity of building end-to-end AI/ML applications rests not only in the algorithms and models, but also in the several layers of infrastructure required to bring them to life. Consider this: You have a Vue.js chatbot frontend, a chatbot API powered by LLaMA models, a Vercel cloud hosting environment for the frontend, and a Runpod environment for the API. But here’s the catch: you have to weave all of these pieces together seamlessly.

Individual scripts or manual settings to set up and manage each component can soon create a tangled web of inefficiencies and potential problems. We can use Pulumi here to deploy everything. From the frontend to the backend, Pulumi orchestrates the programmatic generation and coordination of every critical infrastructure element.

With Pulumi, developer productivity increases, scalability is a breeze, and the complex infrastructure pieces fit together easily. Now that you can see why this is important, let’s get started.

Before we get into the details of using Pulumi to deploy a Vue.js frontend app to Vercel, let’s start with some quick terminology/context:

Katwalk Frontend is a demo Vue.js application that integrates with the Katwalk Server to produce a function web interface for you to communicate with your chatbot.
Vercel is a modern web development cloud platform and hosting provider. It specialises in making web application deployment simple and efficient.

How to Integrate Your Chatbot Frontend and Deploy to Vercel

If you are ready to integrate your chatbot and deploy it to Vercel along with us, then let’s double check a few requirements before we continue. While Pulumi supports many languages, we chose Python for this project, as Python is a familiar language in the AI industry and this is part of the #PulumiMLOps challenge.

Requirements

Pulumi CLI
Pulumi account and access token
Python3
Git CLI
Endpoint of Katwalk Server API. Follow the steps in this article to deploy the API and get your endpoint.
Vercel account and access token

Now that you have everything needed to get started, let’s hook up our frontend and deploy the frontend to Vercel using Pulumi.

1. Clone the repo and setup your project

The first thing we need to do is clone the repository from GitHub and setup the Pulumi Python environment. The Katwalk Frontend code is also in the repo for the Katwalk Server so this post will do both deployments.

# Clone the repository and cd to the infra directory
git clone https://github.com/pulumiverse/katwalk && cd katwalk/pulumi

# Create and initialize the python virtual env
python3 -m venv venv && source venv/bin/activate

# Install python dependencies
python -m pip install -r requirements.txt

Next, let’s setup our state and secrets store in Pulumi Cloud.

# You can store Pulumi state in multiple ways, here we use Pulumi Cloud
# Other state backends include Azure storage, Amazon S3, local file, and more
pulumi login

# Initialize your stack
# Here we name the stack "dev"
pulumi stack init --stack dev

3. Configure deployment credentials

Before the deployment, you will need to set the config values for your Vercel token, the repository name, and the repository type. The Vercel token is supposed to be set as a secret. If you open the Pulumi.${stackName}.yaml file, you can see the encrypted secret in this plain text YAML configuration.

# The name of the git repository. It should have this format {account-name}/{repository-name}
pulumi config set gitRepoName <repo_name>

# The git provider of the repository. Must be either `github`, `gitlab`, or `bitbucket`.
pulumi config set gitRepoType <github|gitlab|bitbucket>

# Configure vercel project name. Your vercel website will be `https://{vcl_project_name}.vercel.app`
pulumi config set vclProjectName <vcl_project_name>

# Configure vercel token as secret
pulumi config set --secret vclToken <vcl_token>

4. Deploy to Vercel

Now, you’re set up to deploy your Vue app to Vercel.

# Finally, run `pulumi up` to deploy!
pulumi up

5. Interact with your chatbot

Once your web app is running, you can visit the website at https://{vcl_project_name}.vercel.app, send prompts and see the response in real-time!

Figure 2. A prompt and response screenshot from our Katwalk Frontend

6. Tear down resources

Now that you’ve tested out this code and you can see the chatbot in action, remember to deprovision your deployment by running:

pulumi destroy

Understanding the Pulumi Code for Vercel Deployment

If you’d like to learn about what the Pulumi code is doing, then stick around. In this section, we’ll dive into the Python Pulumi code which is responsible for deploying the Vue.js project to Vercel using Pulumi. Let’s break down the code step by step to understand how it works.

1. Import the necessary modules

import pulumi
import pulumiverse_vercel as vercel

The code begins by importing the required modules. pulumi is the primary Pulumi Python library, while pulumiverse_vercel is a Pulumi provider for Vercel. These modules provide the necessary functionality to define and manage our web app infrastructure.

2. Configuration settings

config = pulumi.Config()
vcl_project_name = config.require("vclProjectName")
vcl_token = config.require_secret("vclToken")
git_repo_name = config.require("gitRepoName")
git_repo_type = config.require("gitRepoType")

Here, we retrieve configuration settings using Pulumi’s configuration management. Configuration values like vclProjectName, vclToken, gitRepoName, and gitRepoType are essential for setting up the Vercel deployment. These were the values that you provided using the pulumi config set command before deploying the stack.

3. Vercel provider

provider = vercel.Provider("vercel-provider",
 api_token = vcl_token
)

This part initializes the Vercel provider with the Vercel API token. The provider is used to authenticate and interact with the Vercel platform programmatically.

4. Creating a Vercel project

project = vercel.Project("vercel-project",
 name = vcl_project_name,
 framework = "vue",
 git_repository = vercel.ProjectGitRepositoryArgs(
 repo = git_repo_name,
 type = git_repo_type
 ),
 root_directory = "src/app/katwalk-frontend",
 opts = pulumi.ResourceOptions(
 provider = provider
 )
)

Here, we define a Vercel project vcl_project_name. The git_repository parameter specifies the Git repository details, including git_repo_name and git_repo_type. We also associate this project with the Vercel provider using opts.

5. Setting Environment Variables

environment = vercel.ProjectEnvironmentVariable("vercel-env",
 project_id = project.id,
 key = "VUE_APP_BACKEND_DNS",
 value = runpod_id_fqdn,
 targets = ["production"],
 opts = pulumi.ResourceOptions(
 provider = provider
 )
)

This section sets an environment variable named VUE_APP_BACKEND_DNS for the Vercel project. The value of this variable is actually the DNS name that is generated by the backend server endpoint code. In the code snippet, the environment variable is also linked to the project.id and specifies that it should be available only in the production environment.

6. Deploying the Project

deployment = vercel.Deployment("vercel-deployment",
 project_id = project.id,
 production = True,
 ref = "main",
 opts = pulumi.ResourceOptions(
 provider = provider
 )
)

Finally, we deploy the Vercel project with the vercel.Deployment resource. This deployment is configured to use the main branch and is marked as a production deployment.

As you’ve seen above, we can integrate Pulumi into our deployments when building AI apps with Python.

If you’ve joined us on this incredible journey to connect the dots between the API and the Vue.js chatbot frontend using Pulumi and Python, we’re eager to hear your story. How did it work out for you? Did you find the process as exhilarating and empowering as we did? What are you most excited about in this AI-powered future?

Our journey doesn’t end here; it’s merely a waypoint on the ever-evolving road of innovation. We invite you to share your thoughts, ideas, and suggestions for future installments of the Pulumi Python + MLOps series. Together, we’ll continue pushing the boundaries of what’s possible in the world of AI development.

Join us in the Pulumi Community Slack or in the Pulumiverse Katwalk GitHub repository to decide what #aiChallenge we tackle next!

Deploy AI Models on Amazon SageMaker using Pulumi Python IaC

Kat Morgan — Mon, 11 Sep 2023 00:00:00 +0000

Running models from Hugging Face on Amazon SageMaker is a popular deployment option for AI/ML services. While the SageMaker console allows for provisioning these cloud resources, this deployment pattern is labor intensive to document and vulnerable to human errors when reproducing as a regular operations practice. Infrastructure as Code (IaC) offers a reliable and easy to duplicate deployment practice. By developing this IaC with Pulumi, practitioners can choose to write their infrastructure code in Python and seamlessly develop both AI application code and IaC code in the same language.

In this short tutorial we will deploy a publicly available Meta AI LlaMa 2 based model from Hugging Face, on Amazon SageMaker. Then we will test it with a natural language prompt using a short Python script.

Pulumi makes AI/ML Easy?

We will use the sagemaker-aws-python Pulumi Template to bootstrap our Python SageMaker IaC. Templates allow you to quickly bootstrap new Pulumi projects from a working scaffold which you can then customize further to your needs. Out of the box, this template provisions Amazon IAM Roles to assign SageMaker privileges, CloudWatch alarms to alert in case of latency or error spikes for the endpoint, and of course a Meta AI LlaMa 2 based LLM (NousResearch/Llama-2-7b-chat-hf) hosted on Hugging Face.

Requirements

Instructions

Let’s begin by logging into Pulumi Cloud:

# There are many ways to store Pulumi state, here we use Pulumi Cloud
# Other state backends include S3, local file, and more
pulumi login

If you’re unsure of which Pulumi state backend you’re using, you can check by running pulumi whoami (or pulumi whoami -v for more info).

2. Prepare a new Pulumi project

Here we create our new Pulumi project directory and populate it from the sagemaker-aws-python Pulumi template

# Create a new directory & change directories into it
mkdir newSageMaker && cd newSageMaker

# Start your project from the sagemaker-aws-python template
# Follow along with the prompts to create your new project and initialize a stack
pulumi new sagemaker-aws-python

While creating a new project from the sagemaker-aws-python Pulumi template, you will be prompted for a project name, description, stack name, and Amazon Web Service Region. You can proceed with defaults, or supply your own.

After responding to all prompts, the pulumi new command will proceed to setup a Python Virtual Environment (venv) and download all dependencies into the venv.

3. Deploy your model as a new SageMaker endpoint

This step may take between 10 and 20 minutes while Amazon builds your infrastructure and deploys the configured model. You can follow along in the console as resources are provisioned, or open the link displayed in terminal to view the deployment status and other stack information in Pulumi Cloud.

pulumi up

4. Try your new SageMaker endpoint

Once your stack has finished deploying, use this rudimentary Python snippet to test the deployed SageMaker endpoint.

NOTE: Notice that we are using us-east-1 in this script. Be sure to change the region in Python to match the region you deployed the SageMaker endpoint into.

First, save the following python snippet as test.py:

import json, boto3, argparse

def main(endpoint_name):
 client = boto3.client('SageMaker-runtime', region_name='us-east-1')
 payload = json.dumps({"inputs": "In 3 words, name the biggest mountain on earth?"})
 response = client.invoke_endpoint(EndpointName=endpoint_name, ContentType="application/json", Body=payload)
 print("Response:", json.loads(response['Body'].read().decode()))

if __name__ == "__main__":
 parser = argparse.ArgumentParser()
 parser.add_argument("endpoint_name")
 main(parser.parse_args().endpoint_name)

Once you have the test.py script created, source the Python virtual environment named venv (created automatically by Pulumi) and run the script using the name of your new endpoint, taken directly from the Pulumi stack output:

# Source the venv
# On Linux & MacOS
source venv/bin/activate

# Execute test.py
python3 test.py $(pulumi stack output EndpointName)

5. Cleanup all resources

Finally, when you’re finished with testing your Hugging Face model on SageMaker, you can easily clean up un-used resources with one easy command.

pulumi destroy

Conclusion

To recap, in a few commands, we created a new Pulumi Python project from a ready-to-roll template, deployed an LLM endpoint on Amazon SageMaker, and tested it with a short Python script to generate a response from our model!

AI and ML is rapidly becoming a necessity with every new day. It may appear daunting or out of reach at first glance, but with the power of IaC written as Pulumi Python programs, getting started has never been easier.

If you followed along then tell us how it worked out for you! We would love to know what you are looking forward to, or if you have ideas for future installments of the Pulumi Python #PulumiMLOps series!

Join us on Twitter, and on the Pulumi Community Slack to decide what #PulumiMLOps we take on next!

The Real AI Challenge is Cloud, not Code!

Kat Morgan — Wed, 30 Aug 2023 00:00:00 +0000

The AI industry is stealing the show as tech’s goldrush of the ’20s. Just looking at ChatGPT’s record setting user growth, and rapid 3rd party integration by top brands, it is not surprising the hype suggests this is the beginning of a major digital transformation.

However, using AI/ML in your own products has some major challenges and obstacles. Below is a diagram of the end to end workflow of building and using an AI model: preparing the data, training a model, fine-tuning a model, hosting and running a model, building a backend service to serve the model, and building the user interface that interacts with the model. Most AI engineers are only involved in a few steps of the process. However, there is one challenge that is common across the entire workflow: creating and managing the cloud infrastructure is hard.

Figure 1. A sneak peak at the map of ML development and product path

Training a model requires spinning up hyper scale GPU clusters. Preparing and consuming the training or fine-tuning data requires managing data warehouses and buckets of object storage. Building an AI backend service requires packaging the model, deploying and scaling the model across compute infrastructure, creating and managing the network infrastructure like load balancing, and managing the vector databases. Building great frontend user experiences requires provisioning and managing CDNs and web application services. Suffice to say: The most difficult challenge that we face with AI today is a problem of cloud orchestration.

Most of this cloud orchestration today is done manually or with complex scripts. This is usually fraught with errors and simply doesn’t scale. Too much time is spent making cloud infrastructure work instead of focusing on actual AI/ML tasks. Infrastructure as code (IaC) is a key component to solving the cloud orchestration challenge present in AI/ML. By applying the rigor and precision of software development practices to cloud operations, IaC offers AI/ML professionals an efficient, reliable, and predictable way to develop at the highest velocity. Pulumi is an open source infrastructure as code platform that allows engineers to use any programming languages, including Python, to provision and manage AI infrastructure.

This blog post is the first installment of a series on AI Infrastructure as Python. We will start this journey with building an AI backend service for a chatbot. In this post, we will deploy a LLM model behind an API backend running on compute infrastructure in the cloud. This entire infrastructure pipeline will be deployed using IaC written in Python. The series will follow this AI backend service through its evolution into a robust and resilient cloud native service.

The most difficult challenge that we face with AI today is a problem of cloud orchestration

Before we get into the details of using Pulumi to deploy a private LlaMa 2 API chatbot service—in 30 minutes or less, of course—let’s start with some quick terminology/context:

Katwalk Server is a demo AI application written to host your choice of LLM either locally or in the cloud, and serve it as an OpenAI API compatible service. As a practical, hands-on introduction to IaC for the MLOps space, this project aims to demystify the concepts and showcase the benefits Pulumi can bring to your AI/ML projects.
Large Language Models or LLMs (like ChatGPT) are machine learning models trained to generate human-like conversational text which have already become essential tools for businesses and individuals, transforming every day tasks and how we interact with technology.
LlaMa is an LLM created by Meta, available for free for research and commercial use.

Figure 2. Or just let ChatGPT describe itself

Deploy a private LlaMa 2 API chatbot service … in 30 minutes or less!

If you are ready to try the #MLOpsChallenge along with us, then let’s double check a few requirements before starting the clock on our 30 minute chatbot challenge. While Pulumi supports many languages, we chose Python for this project, as Python is a familiar language in the AI industry.

Requirements

Choose one of the following deployment platforms. (You can actually choose more than one, if you’d like!)

Now, for the real fun, let’s start the timer and deploy our own chatbot! Note that the instructions below are for Runpod; instructions for the other two deployment platforms are found in the GitHub repository.

1. Prepare your Pulumi IaC directory

First we need to clone our IaC repository from GitHub and prepare our Python environment.

# Clone the repository and cd to the pulumi iac directory
git clone https://github.com/pulumiverse/katwalk && cd katwalk/pulumi

# Create and initialize the python virtual env
python3 -m venv venv && source venv/bin/activate

# Install python dependencies
python3 -m pip install -r requirements.txt

Next, let’s setup our state and secrets store in Pulumi Cloud.

# There are many ways to store Pulumi state, here we use Pulumi Cloud
# Other state backends include S3, local file, and more
pulumi login

# Initialize your stack
# Here we name the stack "dev"
pulumi stack init --stack dev

3. Configure required credentials

Configure your Huggingface credentials. If you open the Pulumi.${stackName}.yaml file, you can see the encrypted secrets in this plain text YAML configuration.

# Set Huggingface.co username
pulumi config set hfUsername <huggingface_username>

# Configure token as secret
pulumi config set --secret hfToken <huggingface_api_token>

# Set the model that you want to download
pulumi config set hfModel "meta-llama/Llama-2-7b-chat-hf"

4. Deploy in the cloud on Runpod.io

While there are three providers already supported in this codebase, this post illustrates deploying to Runpod.io, a GPU and ML dedicated cloud catering to AI workloads and development tasks.

# This enables the IaC to deploy Katwalk Server
pulumi config set deploy True

# Set the deploy runtime to runpod to deploy on the Runpod cloud
pulumi config set runtime runpod

# Configure your Runpod API Token as a secure Pulumi secret
# API token available at https://www.runpod.io/console/user/settings
pulumi config set --secret runpodToken <your_runpod_api_token>

# Finally, run `pulumi up` to deploy!
pulumi up

# When done, you can 'destroy' the stack to deprovision your deployment
pulumi destroy

There’s currently a bug in the code that does not return the fully-qualified domain name (FQDN) from Runpod for your chatbot instance. To get the FQDN (referenced later in this post as RUNPOD_FQDN), you’ll need to log into Runpod and retrieve that information.

5. Build the Katwalk container

While there are many other ways to orchestrate ML services in the cloud and locally, in this project we are taking ownership of the full application and infrastructure story. In fact, all three supported providers use the same container image to serve the LlaMa LLM API Service, and you can build it yourself just like this:

# Set image build to True
pulumi config set imageBuild True

# Let's build this without pushing to a registry for now
pulumi config set imagePush False

# Give your katwalk image a project name
pulumi config set registryProject $(whoami)

# Now build the image!
pulumi up

6. Talk to your chatbot!

Once your Katwalk LlaMa server is running, you can interact with the API by sending prompts as a JSON payload. In future posts we will orchestrate a chat frontend for our chatbot but for now we’ll use a curl command. You can also substitute an API developer tool such as Postman or Insomnia!

# send a prompt with curl
curl -s -X 'POST' 'http://${RUNPOD_FQDN}/v1/chat' -H 'accept: application/json' -H 'Content-Type: application/json' -d '{"prompt": "Write a short story about a robot."}'
{
 "text": [
 "\nOnce upon a time, there was a little robot named R2. R2 was a friendly and curious robot who lived in a big city. One day, R2 decided to go on an adventure. He set out to explore the city and learn about all the different things he could see and do.\nAs R2 explored the city, he met all kinds of people. Some were kind and welcoming, while others were scared or suspicious of him. Despite this, R2 continued to be friendly and curious, always asking questions and trying to learn more about the world around him.\nOne day, while R2 was exploring a busy market, he saw a group of people gathered around a little girl who was"
 ]
}

Figure 3. A prompt and response from meta-llama/Llama-2-7b-chat-hf hosted on Katwalk Server

In a world where “data is the new black gold” and tokens are money, deploying a chatbot in 30 minutes is not just interesting, it is a novel capability quickly becoming a necessity.

Platform engineering, DevOps, or MLOps may be the next hardest step in building the AI future. Pulumi offers a reliable path to higher velocity AI App development and MLOps. Together with Python-based IaC, it empowers the MLOps community to build equity in common AI platform code, and share IaC in a familiar language.

If you followed along then tell us how it worked out for you! We would love to know what you are looking forward to, or if you have ideas for future installments of the Pulumi Python + MLOps series!

Join us in the Pulumi Community Slack or in the Pulumiverse Community Katwalk Github repository to decide what #aiChallenge we tackle next!

Figure 4. Midjourney generated image of an imagined machine learning research engineer working among cables, computers, keyboards, and screens, in the glow of electronic lights. Image was generated from a prompt ~80% produced by ChatGPT based on this blog post content and a few superficial details about the author

Pulumi Blog: Ml

How AI is Transforming DevOps: AI Talks for DevOps Insights

How will AI Impact the Future of DevOps

What is DevOps in AI

What is AI in DevOps

How AI is Currently Used in DevOps

Streamlining Development Workflows

Enhancing Monitoring and Security

How Can a DevOps Team Take Advantage of Artificial Intelligence

Code Generation and Learning

Intelligent Resource Management

Remediation Tool: Continuous and Automatic Compliance

Enhance Collaboration and Communication

Improved Product Delivery

Planning and Cost Optimization

Top 5 Skills for DevOps Engineers in the AI Era

How to Overcome AI for DevOps Challenges

1. Skills and Expertise

2. Integration Complexity

3. Scalability

4. Data Quality and Availability

5. Cultural Resistance

Conclusion

Frequently Asked Questions

Can AI replace DevOps?

What is an example of AI in DevOps?

What is the future of DevOps?

LangChain for DevOps: Learn LLM & GenAI for Dev, Sec & Ops

What is LangChain used for?

Why is LangChain becoming popular?

LangChain Accelerating DevOps

How to Use LangChain to Learn LLMs and GenAI for Dev(Sec)Ops

Developer

Operations

Security

Real-World Applications of LangChain AI in DevOps

Monitoring and Anomaly Detection with LangChain AI

Future: AI, Apps, and DevOps

Accelerating Development with AI-Powered Tools

Security and Vulnerability Management with AI

Leveraging Machine Learning in DevOps

Implementing AI models into DevOps: Key Considerations

Conclusion

Frequently Asked Questions

Who is Patrick Debois?

Is LangChain Free?

How Does LangChain Work?

Does ChatGPT use LangChain?

Deploy an AI/ML Chatbot Frontend on Vercel with Pulumi

How to Integrate Your Chatbot Frontend and Deploy to Vercel

Requirements

1. Clone the repo and setup your project

2. Login to Pulumi Cloud and initialize stack

3. Configure deployment credentials

4. Deploy to Vercel

5. Interact with your chatbot

6. Tear down resources

Understanding the Pulumi Code for Vercel Deployment

1. Import the necessary modules

2. Configuration settings

3. Vercel provider

4. Creating a Vercel project

5. Setting Environment Variables

6. Deploying the Project

Deploy AI Models on Amazon SageMaker using Pulumi Python IaC

Pulumi makes AI/ML Easy?

Requirements

Instructions

1. Login to Pulumi Cloud and initialize stack

2. Prepare a new Pulumi project

3. Deploy your model as a new SageMaker endpoint

4. Try your new SageMaker endpoint

5. Cleanup all resources

Conclusion

The Real AI Challenge is Cloud, not Code!

The most difficult challenge that we face with AI today is a problem of cloud orchestration

Deploy a private LlaMa 2 API chatbot service … in 30 minutes or less!

Requirements

1. Prepare your Pulumi IaC directory

2. Login to Pulumi Cloud and initialize stack