Pulumi Blog: Copilot

Pulumi Updates, Explained: AI-Powered Features in Pulumi CLI

Artur Laksberg — Thu, 22 May 2025 00:00:00 +0000

Note: This post discusses Pulumi Copilot, which Pulumi Neo has replaced. Learn about Neo →

We’re excited to announce the new AI capabilities for Pulumi CLI powered by Pulumi Copilot that translate complex infrastructure changes into clear, human-readable explanations and help resolve deployment errors with actionable guidance. Enable these preview features with the --copilot flag.

At Pulumi, we’re committed to helping you deploy infrastructure efficiently and with minimal friction. Anyone who has worked with cloud infrastructure knows the frustration of sifting through large previews with numerous changes and the difficulty of interpreting cryptic error messages from cloud provider APIs when deployments fail.

There are two key challenges we’ve identified:

Understanding Preview Changes: Reviewing raw update previews can be overwhelming, especially for complex deployments with many resources. It’s difficult to quickly grasp the full scope and impact of pending infrastructure changes.
Interpreting Error Messages: When deployments fail, cloud providers often return error messages that, while comprehensive, bury the core issue within extensive diagnostics, requiring specialized expertise to interpret and address.

Our latest CLI update introduces AI capabilities that tackle both issues. Let’s explore how these new features improve your infrastructure management workflow.

Update Explanation

To address the first challenge above, we’ve introduced a new “explain” menu item in the CLI. When running pulumi preview or pulumi up with the --copilot flag, you’ll see this option in the interactive menu:

In this screenshot, you can see the standard Pulumi update preview showing an AWS VPC with multiple subnets, route tables, and gateways that need to be updated. The interactive menu at the bottom shows the new “explain” option with a sparkle emoji, indicating the AI-powered feature. When selected, this tool analyzes and provides a clear summary of all changes in your pending infrastructure update:

In this example, the AI explanation immediately identifies that this is a “VPC Renaming Operation - Tag Updates Only” and explains that the update is simply changing tags across all components. It clearly states this is a “safe, non-disruptive change that only affects resource labels.”

This explanation serves as a valuable sanity check before deployment. It presents a plain-language summary of exactly what changes will occur, helping you verify that the planned modifications align with your intentions. This visibility is particularly valuable when managing complex infrastructure with many resources, as it helps identify unexpected changes or potential drift between your code and deployed resources. The system also proactively flags potential issues such as security vulnerabilities or destructive updates that might impact your production environment.

Diagnostics Summarization

For the second challenge, when infrastructure deployments fail, cloud providers often return error messages that are technically accurate but difficult to decipher without specialized knowledge. Consider the following scenario where a Go-based Docker image build fails with vendoring issues:

In this example, the standard Go build process failed with a cryptic error about inconsistent vendoring. While the raw output does contain the necessary information, it’s buried within extensive build logs. Our AI-powered Copilot diagnostics immediately identifies the problem and provides a clear, actionable solution: “Docker image update failed due to inconsistent vendoring. Sync vendor with ‘go mod vendor’.”

This clear explanation identifies root causes and provides actionable guidance, transforming dense technical output into concise explanations. It reduces time to resolution by pinpointing both the problem and the exact command needed to fix it, eliminating the need to search through documentation or guess.

Getting Started

The AI-powered features are currently in preview. You can enable them by adding the --copilot flag to your Pulumi CLI commands:

pulumi preview --copilot
pulumi up --copilot

These features require Pulumi CLI version v3.171.0 or greater and the Pulumi Cloud backend. Organization administrators must enable Pulumi Copilot by navigating to Settings > Access Management > Pulumi Copilot in the Pulumi Cloud console.

During this preview period, we’re focused on:

Refining explanations based on user feedback
Adding new capabilities like suggesting fixes for common errors
Improving performance to ensure minimal impact on command execution time

We will continue to monitor usage patterns and incorporate your feedback before enabling these features by default.

We’re excited to see how these AI-powered insights enhance your infrastructure deployment workflow! Please share your experiences with us:

Connect with us in the Pulumi Community Slack
Open an issue on GitHub
Learn more about Pulumi Copilot

Try this today, Get Started with Pulumi for free

Your input will directly influence how we evolve these features from preview to general availability.

A Recipe for a Better AI-based Code Generator

Artur Laksberg — Tue, 07 Jan 2025 20:00:00 -0500

Note: This post discusses Pulumi Copilot, which Pulumi Neo has replaced. Learn about Neo →

When asked about his research process, Anthony Bourdain would describe how he’d blend his formal culinary training with deep dives into local food culture - from market stalls to family recipes. Modern AI code generation follows a similar path: it can’t just rely on what it knows - it must tap into continuously evolving, domain-specific knowledge bases. Just as Bourdain would combine his classical French training with techniques learned from local kitchens, AI code generators blend their built-in knowledge with retrieved code snippets and type definitions to generate code that accurately represents the user’s intent.

This fusion of base knowledge with contextual understanding is especially valuable for Infrastructure as Code (IaC), where rapidly evolving cloud providers and libraries make manual development challenging, traditional debugging cycles impractical, and errors catastrophically expensive.

The role of IaC won’t diminish in the age of AI - if anything, it will become even more central as systems grow increasingly complex and automated. Trustworthy code generators will be a key ingredient in the recipe for modern infrastructure management.

In this post, we share how we developed code generation for Pulumi and what we learned, based on both our production IaC generator powering Pulumi AI and Pulumi Copilot, as well as features and approaches we’re still exploring.

Using RAG for code generation

Pulumi supports over 120 providers, including major cloud providers such as AWS, Azure, Google Cloud, and Kubernetes, as well as many other services and platforms. New providers are continuously added, and existing providers change as their capabilities grow.

Our goal is to generate the most accurate code for every provider - code that is not only correct but also reflects their latest capabilities.

LLMs are great at generating code however they are limited to what they have learned before their knowledge cutoff date. This means that the latest changes in the providers will not be reflected in the generated code. This also could lead to hallucinations when the model struggles to answer a question that requires up to date knowledge.

To help us solve this, we rely on the technique known as the Retrieval Augmented Generation (RAG). RAG helps code generation by integrating information retrieved from external data sources. In Pulumi, we call this data source the Registry - it’s the database we maintain that contains type schema and usage information for every provider.

At a high level, using RAG involves the following steps:

Analyze user’s question.
Look up the pertinent information from the Registry.
Format that information in a system prompt that LLM can understand.
Make the LLM call asking it to generate the code using the additional information in the prompt.

There is lot of fascinating details here, so let’s dig in!

The anatomy of Pulumi Copilot RAG

Before going into the details, let’s consider a simple yet essential insight: an effective dataset for RAG must meet two key requirements:

It must contain the necessary information.
The information must be easily searchable.

Let’s take a sample user query:

“Generate code for S3 Bucket”

To fulfil the request, the system needs to intuit the following:

The cloud provider. This can be determined based on the fact that S3 is a common storage solution in AWS, and possibly the fact that the user’s organization has stated AWS as their preferred cloud provider.
The programming language. This information can again come directly from the organizational preferences, or from the user’s prior conversations.
The information about the type (or types) that must be created - its name and schema, the package it is in, and the capabilities it supports. Some of this information comes from the built-in LLM knowledge, supported and augmented by the Registry.

Putting it all together, we can now expand the original user query into the following prompt that is going to guide the code generation:

“Generate TypeScript code for S3 Bucket, the AWS resource defined in package @pulumi/aws, type aws.s3.Bucket with its schema defined as follows: …

While we had to rely on some guesswork to come up with this prompt, fortunately this process can be iterative - if we don’t get all of it right the first time, we can try again with additional information that will help us refine the results. This is an important point we return to later in the post.

Assessing search quality using recall and precision

To assess how effective the RAG is, we need to first understand the two fundamental concepts used in the information retrieval systems: the recall and the precision. Imagine that you’re looking for apple pie recipes in one of Jamie Oliver’s cookbooks. The book has a recipe for a classic American apple pie, a Dutch apple pie and a modern take on a French apple tart. Due to the book’s narrative approach with the recipes woven into the stories and context, you’ve managed to retrieve only the first two recipes but missed the French apple tart. Having retrieved 2 ouf 3 relevant documents, you have achieved a 67% recall.

Because you were looking for the word “pie”, you also retrieved a recipe for a Shepherd’s pie, which, while delicious, does not qualify as an apple pie. Another document that came up was a fish pie - a classic British dish that does not contain apples or even a pastry crust. Since only 2 of your 4 retrieved documents can be legitimately classified as apple pies, you have achieved a 50% precision.

Now let’s formalize this a bit. Recall measures the ratio of the relevant documents retrieved to the total number of relevant documents in RAG dataset:

$$Recall = \frac{N(Retrieved \cap Relevant)}{N(Relevant)}$$

Where

$N(Retrieved \cap Relevant)$ is the number of documents that are both retrieved and relevant.
$N(Relevant)$ is the total number of relevant documents in the database.

Good recall means that many documents relevant to the query were retrieved.

Precision is the ratio of the relevant documents retrieved to the total number of retrieved documents:

$$Precision = \frac{N(Retrieved \cap Relevant)}{N(Retrieved)}$$

Where $N(Retrieved)$ is the total number of documents that were retrieved.

High precision means that many of the retrieved documents were relevant.

Naturally, we want to maximize both the recall and the precision. It’s been said that high recall is essential to ensure relevant content is available to the code generator while precision is the parameter you want to optimize for to avoid hallucinations.

Practical concerns

Precision and recall are essential in understanding the information retrieval quality, but they are quite hard to measure in practice. Unlike a cookbook, Pulumi registry contains thousands of ever changing documents, and evaluating how many of them are relevant for every user-submitted query is impractical. This makes recall evaluation for live traffic next to impossible. Things are a little easier with precision, where we’re dealing with a small number of documents, but even that metric requires a non-trivial evaluation of relevance, which needs an LLM call or a human judge.

Fortunately, other metrics that often can effectively estimate retrieval quality have been developed. We have found a metric that can predict, with some degree of accuracy, whether the generated code will successfully compile. For this metric, we compare the tokens present in the LLM-produced prompt with the number of tokens present in the generated code. (By token here we understand a compiler token - an identifier such as the name of a class, method or a field and not a traditional LLM token concept), Intuitively, if a token present in the prompt also appears in the generated program, it can be assumed that the token contributed to the generated program. Tokens in the generated program that were not part of the prompt are not necessarily wrong but they are less trusted (they can come from the LLM built-in knowledge or were guessed)

$$prompt \ coverage = \frac{N(\text{Tokens in prompt} \cap \text{Tokens in code})}{N(\text{Tokens in code})}$$

Prompt coverage is a metric we can observe in production, and it’s one of several metrics we use when updating providers to ensure we haven’t regressed the quality of the RAG.

Semantic search with vector embeddings

Semantic search is based on the conceptual similarity of the term you’re looking for with the elements in the data store. For example, searching for “dumplings” can return terms like pierogi and gyoza - words with different spelling but both representing different types of filled dough preparations.

A common way to determine the similarity between the two strings is to first turn these strings into vector embeddings - arrays of floating point values representing the semantic meaning of each string - and then calculate the cosine similarity between the two vectors, which is the cosine of the angle between the vectors. Various methods of producing vector embeddings are fascinating but we will not cover them here in depth.

For Pulumi code generation we are using the OpenAI’s Ada-002 embedding model which at this moment represents a good balance between performance and cost.

Producing vector embeddings from the user query is the standard approach in this situation. However, for Pulumi code generator we added a little twist - to increase the odds of getting more relevant information from the Registry (i.e. to increase the recall) we first make an LLM call to generate a small set of relevant search terms that will produce an array of vector embeddings.

Getting vector embeddings from user query

We then use the array of vector embeddings to retrieve the set of relevant documents from the Registry.

Full text vs semantic search

While semantic search is essential for any modern information retrieval system, we should not forget simple and effective methods for text search already exist. The “S3 Bucket” part of the user search happens to be easily searchable using traditional text search operations (such as SQL LIKE operator).

To be effective, our RAG must be able to handle queries that require semantic understanding (such as “simple storage service in AWS”) as well as the traditional text search to support situations where the user knows exactly what they are looking for. To that end, the industry has adopted an approach known as the “hybrid search”, in which the results of full-text search and semantic search are combined to provide the final result.

For each of these search terms we generate a query that combines the full text search with the semantic search. The resulting documents are then evaluated based on the following criteria:

Dense score: Vector similarity using cosine distance between query embedding and stored embeddings
Sparse score: Text search relevance according to the full-using algorithm (PostgreSQL full-text search in our case)

Both the dense and the sparse scores contribute to the final score that is used to sort the documents. Even though the final score is the average of the two in our current implementation, it would not be correct to say that they have the same weight, or the influence on the end result. This is so because the normalization of the scores must take into account their distribution in the real-world queries and calculating that is quite complex.

Boosting the influence of one score relative to the other is an area we’re actively exploring, and achieving the optimal result depends highly on what parameters we decide to optimize for - high quality code, time to result and so on.

Finally, we apply the rank-based scoring technique in which the results are penalized based on their position in the ranking. This creates separation between results that might have similar initial scores and gives preference to results that appear earlier in the ranking.

Pruning the results

Our Pulumi code generator employs a two-phase document selection strategy. The first phase casts a wide net, gathering all potentially relevant documentation to maximize recall. The second phase applies precision-focused filtering to distill this collection down to the most pertinent documents.

This filtering step serves two purposes. First, it prevents LLM hallucinations that arise from similarly-named types across different providers. Second, it optimizes performance by keeping prompts concise - a critical consideration given that larger prompts increase both latency and computational costs, even when within context window constraints.

Through empirical testing with the Pulumi Registry search, we’ve established these baseline parameters: a maximum of 10 documents per query term selected by relevance score, and a 20K token ceiling for prompts. While these parameters have yielded good results in practice, they are likely not optimal for all scenarios. We continue to iterate on these values through ongoing experimentation.

Prompt generation

We’re now ready to create the system prompt for the code generation LLM call! We’ve already discussed some of the elements needed to build the effective prompt - the original user query, the search terms and the vector embeddings that produce the set of documents that will guide the code generation process.

There is another element that goes into the prompt - a concise set of instructions produced by an LLM call based on the original user’s query. The approach when the output of one prompt is used as input for another is known as “prompt chaining”, and we used it to provide step-by-step instructions to the code generator. For our query, this set of instructions can look as follows:

Create an S3 bucket using Pulumi in TypeScript using the following steps:

Import the necessary Pulumi AWS package.

Define a new S3 bucket resource with basic configuration.

Export the bucket name as an output.

Composition of the system prompt for code generation

Finally, we use the resulting prompt to call the LLM and ask it to generate the code. We’re done!

Self-debugging

Unfortunately, this isn’t always the final step of the process. Despite our best efforts, the code produced by the LLM will not always be correct.

At this point, it’s worth pondering what “correct code” means. The generated program might have the following problems:

It might be syntactically or semantically incorrect, i.e. it might not compile or fail to typecheck.
It might fail at runtime - for example by referring to a non-existing resource, a region and so on.
It might run “successfully” but not do what the user intended - either because the user did not express themselves clearly or because the request was misunderstood.
Finally, the code might run and do exactly what the user wants, but lead to an undesired outcome, for example loss of an asset, or a security vulnerability.

Solutions to many of these problems go well beyond the domain of code generation.

However, the first of these problems can be addressed by the approach known as “self-debugging”.

We observed that many generated TypeScript programs that fail to typecheck contain only a few errors, and asking the LLM to fix these errors often produces a valid program. While this approach is still experimental, we see promising results where getting to a correct program requires only 1-2 iterations of self-debugging. The biggest challenge with this approach is doing it in real-time. The user is staring at the screen waiting for an answer, so latency is a major concern.

Monitoring these typechecking errors in production can also provide valuable insight into the quality of the data used by the RAG and even suggest specific solutions. For example, failure to typecheck a member-access expression is a likely indicator of a missing type schema (a recall problem) or a “wrong” schema brought in by an irrelevant document (a precision problem).

Self-debugging can also be extended to include the pulumi preview command, which is a “dry run” operation before the actual deployment and can detect many real or potential problems such as destructive actions, incorrect configurations that cannot be detected at compile time, dependency conflicts, and policy violations.

Generate infrastructure code with Pulumi Neo

Turn a natural-language request into working infrastructure as code across AWS, Azure, Google Cloud, and Kubernetes, then preview and deploy it with Pulumi Neo.

Learn more

From kitchen to table: testing what works

The landscape of LLM-based code generation is moving fast, and we need to keep learning and adapting as we go. But with all this rapid technological change, it’s crucial to ground our decisions in real numbers. We need to make sure each new advancement actually makes things better, both in our test environments and out in the real world.

The probabilistic nature of LLM-based code generation means we can’t rely solely on pre-production testing. Instead, we adopt multiple layers of quality control working together. Here’s what we’ve learned works best:

Building confidence through testing

Our testing strategy combines several approaches:

Systematic experimentation: We run “what-if” scenarios against various test datasets to understand how the system behaves under different conditions.
Local evaluation pipeline: While local testing has its limitations, it helps catch obvious issues early in development. We evaluate the performance of the code generator and run the code through the typechecker.
Deterministic generation: We set the LLM temperature to 0 to ensure consistent outputs. For code generation, consistency and repeatability matters more than creative variations.

Monitoring quality in production

We track several key metrics to ensure the system performs well:

RAG effectiveness: We measure prompt coverage to evaluate how well our retrieval system performs.
Code quality indicators: We track the percentage of generated code that successfully typechecks for all supported languages along with how this metric responds to system changes.
User feedback: Every “thumbs down” report gets analyzed to identify patterns and potential improvements. This direct user feedback has been invaluable in refining our system.

Wrapping up

Building an effective AI-powered code generator requires carefully balancing multiple concerns: the raw capabilities of LLMs with retrieved contextual knowledge, semantic search with traditional text matching, and thorough testing with real-world scenarios.

Our experience has taught us that success lies in treating code generation like a delicate soufflé - it requires precise measurements, careful monitoring, and constant refinement of technique. The key ingredients are:

A robust RAG system with well-tuned recall and precision.
End-to-end testing and monitoring across development and production.
Self-debugging capabilities to handle common issues like type checking errors.

As we continue to evolve Pulumi’s code generation capabilities, we’re excited about expanding our self-debugging features and further refining our RAG implementation.

We invite you to try these capabilities in your own projects and share your experiences. Your feedback helps us continue improving and advancing the state of AI-assisted infrastructure development.

Announcing the Pulumi Copilot REST API Preview

Artur Laksberg — Thu, 12 Dec 2024 15:15:42 -0500

Note: This post discusses Pulumi Copilot, which Pulumi Neo has replaced. Learn about Neo →

We built Pulumi Copilot to automate a broad spectrum of cloud management activities using the power of LLMs. Since its initial release earlier this year, hundreds of customers have used Pulumi Copilot to understand and manage cloud infrastructure more effectively and securely, and it is only getting better by the day.

Today, we’re excited to announce the availability of the Pulumi Copilot REST API. This new API exposes the full power of Pulumi Copilot, enabling you to integrate infrastructure AI into your own tools, applications, and platforms. While currently in preview, we are eager to get your feedback to ensure it works for anything you can dream up.

Enhancing Copilot Capabilities

With the Copilot REST API, you can extend the Copilot capabilities available in the Pulumi Console in the following ways:

Build Copilot capabilities into your platforms and tools, such as the CLI, development portals, Github actions and so on.
Support multi-user interaction in workplace collaboration platforms such as Slack and Teams.
Automate execution of Copilot queries based on scheduled triggers or events, such as deployment completions.
Access Copilot from mobile clients through platforms like Slack or Teams.

Implementation Guide

Initial Configuration

To begin using the Copilot API, set up the following environment variables:

export PULUMI_COPILOT_URL="https://api.pulumi.com/api/ai/chat/preview"
export PULUMI_ACCESS_TOKEN="pul-..."

Note: You can obtain your PULUMI_ACCESS_TOKEN from the Pulumi Console.

Understanding Cloud Context

All Copilot API interactions require two essential parameters:

Organization context through the orgId field - this is conceptually similar to the organization you select in the dropdown menu on the left-hand side of the Pulumi Console
Resource URL from the Pulumi Console, which must begin with https://app.pulumi.com - think of this as the browser URL you see when navigating the Pulumi Console. This URL helps Copilot understand the context of your query. While you can provide the base URL (https://app.pulumi.com), you can also point to specific resources or updates for more targeted queries. For example, https://app.pulumi.com/myorg/my-demo-project/my-stack/updates/5 would allow you to ask questions about that specific update

These parameters provide the necessary context for queries about specific resources or updates, allowing Copilot to respond with relevant information just as it would in the Console interface.

Making API Requests

Here’s an example of a basic API request:

curl -L "$PULUMI_COPILOT_URL" \
-H "Authorization: token $PULUMI_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
 "query": "Who are the users in my org?",
 "state": {
 "client": {
 "cloudContext": {
 "orgId": "pulumi",
 "url": "https://app.pulumi.com"
 }
 }
 }
}'

Supporting Multi-turn Conversations

The API supports continuous dialogues, in which participants can refer to information shared earlier in the chat. This is supported through conversation IDs that are received with the response:

"conversationId":"369a280c-63f3-4ee6-a13d-c1035a3d05de"

This ID enables follow-up queries that maintain context:

curl -L "$PULUMI_COPILOT_URL" \
-H "Authorization: token $PULUMI_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
 "conversationId":"369a280c-63f3-4ee6-a13d-c1035a3d05de",
 "query": "Who of them are admins?",
 "state": {
 "client": {
 "cloudContext": {
 "orgId": "pulumi",
 "url": "https://app.pulumi.com"
 }
 }
 }
}'

Moving Forward

As we continue to evole our AI efforts, your feedback is essential. We are particularly interested in hearing about your implementation experiences, including features that work well, areas that need improvement, or capabilities you’d like to see added. You can consult the documentation and peruse the samples. Please submit any feedback by opening an issue or by reaching out to us via our Community Slack

We can’t wait to see what you can build with Pulumi Copilot REST API!

AI Engineering Lessons from Building Pulumi Copilot

Artur Laksberg — Thu, 12 Dec 2024 13:56:13 -0500

Note: This post discusses Pulumi Copilot, which Pulumi Neo has replaced. Learn about Neo →

Building AI-powered developer tools comes with unique challenges, and now that we’ve launched our REST API, we want to share some lessons we’ve learned building Pulumi Copilot, an AI assistant for cloud infrastructure.

One of the big challenges was determining what ‘working’ really meant. So when a message landed in our feedback channel after months of rigorous testing - ‘Your tool doesn’t know anything!’ - it caused some mild panic. We’d just made some changes, so we braced for the worst. But our evals were still looking strong, so what was going on?

The user was attempting to force-delete a stack that still contained resources. But when we dug deeper, we found something fascinating: Copilot had confidently suggested a ‘–force’ flag, which would have been a logical solution… except this flag doesn’t exist in Pulumi. Our AI was hallucinating exactly what the user wanted. But this wasn’t just a bug - it was the first of many insights that would reshape how we approach AI-powered tools.

To understand how we got here - and why this “error” actually taught us something valuable about our product - let’s start with the core challenge we faced: balancing traditional software engineering with this new world of prompt engineering.

Engineering for Reality: Prompt Engineering vs Software Engineering

When building LLM-powered applications, it’s tempting to throw every task at the model. Modern LLMs can generate code, format text, and create clickable links. But this approach carries hidden costs.

Working on Copilot taught us a key lesson: let the LLM do what it does best and use good old imperative code for everything else.

Take a seemingly simple feature: listing a user’s Pulumi stacks with clickable links based on data from a backend API. Our first implementation used a complex prompt instructing the LLM to construct URLs in the format app.pulumi.com/org/project/stack. The prompt explained the format, provided examples, and asked the LLM to generate these links from JSON data it had.

It worked - almost all of the time. But we were seeing occasional malformed URLs and more importantly, this was burning input tokens (and money) on a complicated prompt that made AI construct strings that could be deterministically generated.

The solution was straightforward: generate the full links in the backend service and include them directly in the context. The LLM then needs no instructions on how to create them. Simple stuff, but it gave faster responses and perfect URLs at a lower cost.

When you find yourself writing elaborate prompts to handle structured data transformations, stop and ask: Could traditional code do this better? Could this be decomposed so that the LLM does less?

To validate this approach, we tested Copilot ourselves to see what worked.

Copilot in Action: Real-World Dogfooding

The internal testing phase taught us invaluable lessons about how people would actually use the tool. We watched our team try Copilot in their daily work, and three common use cases emerged:

Debugging Deployments: LLMs clearly excel at summarization. One of the first questions our internal users asked was, ‘Why did my latest infrastructure deployment fail?’ Using Copilot to extract a clear natural language explanation requests like these has been a clear win.

Understanding Complex Infrastructure: Copilot helped our engineers gain insights into Pulumi’s own infrastructure. Asking, ‘How many resources are in production?’ ‘What expensive compute is running’ or ‘What version are the EKS clusters in EU?" shows the value of allowing users to express infrastructure questions in natural language.

Generating Code: One of the first queries logged was, ‘I want a static website on AWS behind a CloudFront CDN.’ Another came from a Solutions Engineer, tasked with demonstrating Pulumi’s CrossGuard policy engine to a prospect, asking Copilot to generate policy code.

These early experiences showed the value of Copilot. But they also revealed the need for a systematic approach to handling diverse user queries. This led to the development of what we call skills.

Skillful Slicing: Modular Mastery

As Copilot grew, we broke it into smaller pieces we call skills. Each skill does one specific job. The Insights skill handles queries about resource usage and configuration (“How many S3 buckets do I have?”), the Cloud Skill interacts with the Pulumi Service API to manage infrastructure (“Show me my stacks.”), the Code Skill generates Pulumi code snippets (“Write a Typescript program…”), and the Docs Skill retrieves information from Pulumi documentation (“How do I use update plans?”).

When you ask Copilot something, it figures out what you need and picks the right skill for the job – like a manager deciding which expert to send your question to. This function-calling approach, orchestrated by a component we call the “outer loop,” allows Copilot to access and process information beyond its internal knowledge base.

Fortunately, Pulumi Cloud already exposes a rich API - in fact, this is what powers the Pulumi Console and the Pulumi CLI - so all we had to do is to build a skill that maps the user query to the appropriate Pulumi Cloud REST API. A question like “Show me my stacks” translates into the List Stacks API call. A question like ‘Show me my untagged EC2 instances’ is a bit more complex but it breaks down into clear components - resource type (EC2), filter condition (untagged) - that route to the Insights skill. This mapping helped us handle the many ways users phrase the same technical request.

Refining this routing system revealed another opportunity: streamlining the Debug button workflow.

Debug Dispatch

Originally, when a user clicked ‘Debug with Copilot’, the system would send a text query like “Analyze this update and explain any errors.” Copilot would then:

Determines the user wants to analyze an update
Identifies which API to call
Calls the API
Summarize the result

Having the LLM figure out what the user wants is the right approach in general, but in this case, we already know the user’s intent - they clicked a debug button. So, we can directly call the analysis API to get the results and use the LLM solely for what it does best: summarizing technical output into clear, actionable explanations.

This is another small win for the “Software Engineering over Prompt Engineering” approach. Traditional code handles the predictable parts, while AI focuses on the human-facing explanations.

But while minimizing the LLM workload helped with efficiency, we soon faced an even trickier challenge: the deceptive polish of AI-generated outputs.

The Illusion of Correctness

Large language models excel at generating well-structured, grammatically correct output. They make neat tables, tell good stories, and generally sound confident. That’s what makes them dangerous because this polished presentation can mask underlying flaws in the information itself, creating a false sense of confidence for users.

One of our early testers, Pablo, a data engineer at Pulumi, encountered this firsthand. He posed a query to Pulumi Copilot, asking for a summary of resources within a specific project. The response he received was impeccably formatted, neatly categorizing resources by type and providing counts for each. It looked right, and for us humans sometimes looking right carries a lot of weight.

However, a closer inspection revealed the numbers were way off. Copilot had asked for the wrong data and then summarized it beautifully - but incorrectly. This highlighted our next challenge: how do you systematically test a system that can be confidently wrong while sounding completely right?

Testing the Untestable: Validating LLM Outputs

When testing traditional code, we expect consistent, predictable outputs. With LLMs, even correct answers can vary. Here’s how we tackle this challenge.

Our first approach was simple: keyword checks. For example, when testing the update analysis feature, we checked if the LLM’s response included the word “security” and described the error. This worked for straightforward cases but broke down quickly. Take a question like, “How many Lambdas am I running?” The LLM might give the right numbers but skip the word “running,” failing the test even though the answer was correct.

These early failures revealed the limitations of keyword-based validation and underscored the need for a more nuanced approach. Inspired by platforms like LangSmith and Promptfoo, we began leveraging LLMs themselves as evaluators. For deterministic tasks, simple keyword checks suffice, but for more complex scenarios—like assessing whether a response answers a specific question—we rely on an “LLM Judge.” This approach balances efficiency and flexibility, reserving LLM evaluation for cases where it truly adds value. The test suite now integrates both methods, running against every code change to validate response content, accuracy, and format.

The eval suite keeps getting more robust, which means that when new AI models drop, we can quickly catch any weirdness before it hits production. The generative AI space moves crazy fast and the code changes a lot, but the evals are a safety net - catching hallucinations, maintaining quality, and making sure we don’t ship anything that’ll annoy our users.

So, while hallucinations are now much rarer, what about that one with the --force flag? Yes, it’s “just another bug,” but it taught us something fascinating about these AI errors.

You might also like:

LLMs think like humans (sort of)

The --force hallucination wasn’t totally wrong - it was revealing what users intuitively expect from the CLI, and the LLM accidentally showed us what was missing. Force deletion is a common pattern across developer tools, and the LLM, trained on vast amounts of documentation and code, simply reflects these established conventions.

This has fundamentally changed how we view hallucinations. While the team works constantly to minimize them – and our eval work means they happen way less frequently – some of them are clearly product signals. The LLM, in this light, becomes an unexpected source of user research, drawing on its training across thousands of developer tools and experiences.

This insight is one of the key lessons of building Copilot:

Minimize LLM Usage: Let traditional code handle deterministic tasks, reserve LLMs for natural language work
Decompose into Skills: Break complex tasks into modular units that combine LLM and traditional code appropriately
Test Rigorously: Use multiple validation approaches, including LLMs testing LLMs
Learn from Hallucinations: Sometimes incorrect outputs reveal user expectations
Learn from Users Continuously: User interactions improve our AI systems - from training better skills to catching hallucinations and revealing product opportunities.

These lessons helped shape our latest release: the Pulumi Copilot REST API, now available in preview. You can integrate these same capabilities and skills into your own tools and workflows. Whether you’re building CLI extensions, chat integrations, or automated deployment checks, the API provides a contextual understanding of Copilot. Try it out.

We can’t wait to see what you build!

Introducing Pulumi Copilot: Intelligent Cloud Management

Luke Hoban — Wed, 12 Jun 2024 04:00:00 -0700

Note: This post discusses Pulumi Copilot, which Pulumi Neo has replaced. Learn about Neo →

AI is transforming how users interact with every category of technology today, and cloud infrastructure is no exception. Last year we launched Pulumi AI to combine generative AI with Pulumi’s knowledge of cloud infrastructure, helping users solve complex cloud development problems using Infrastructure-as-Code. Pulumi AI has seen rapid adoption and engagement, with tens of thousands of users leveraging Pulumi AI over the last year.

Today, we’re excited to introduce Pulumi Copilot, a huge leap forward in applying AI to cloud infrastructure management. Pulumi Copilot is a new conversational chat interface integrated throughout Pulumi Cloud, enabling Pulumi Cloud users to quickly accomplish a variety of cloud infrastructure management tasks by leveraging the power of large language models plus the rich capabilities of Pulumi Cloud.

Through Pulumi Copilot, you can explore your cloud infrastructure and gain insights across an incredible breadth of use cases, including:

Pulumi Copilot Key Features

Access any data in Pulumi Cloud

The state of every resource you are managing with Pulumi across any Cloud, any account, and any region. With Pulumi Insights’ Cloud Supergraph support for 160+ cloud providers, this offers an unprecedented breadth of cloud infrastructure data to explore and interrogate with Pulumi Copilot.
Pulumi stacks, projects, updates, deployments, environments, policies, audit logs and more - enabling historical understanding of what happened when, by who, and why across all of your cloud engineering systems managed by Pulumi.

Pulumi IaC Authoring and Deployment

The same great Pulumi AI features for authoring IaC are now available inside Pulumi Copilot as well, enabling you to quickly solve new IaC problems within Pulumi Cloud, and even deploy code directly from Pulumi Copilot.

Access cloud metadata from the clouds themselves

Through the use of new skills, Pulumi Copilot can access cloud metadata in real time in AWS, Azure, Kubernetes, and more, allowing it to join Pulumi’s IaC world view with information about usage, costs, and more – as well as infrastructure not yet under the management of Pulumi.

Pulumi Copilot combines the best generative AI models available in the industry today with Pulumi Copilot “skills” which enable Pulumi Copilot to access the data and actions from within Pulumi Cloud needed to help you with your questions and explorations. Pulumi Copilot also incorporates the context of where the user is in the Pulumi Cloud console to easily answer questions about “this stack”, or “the latest update”, offering an even more natural, conversational and persistent experience across Pulumi Cloud.

Pulumi Copilot is Enterprise-Ready on day one - it adheres to all the same identity and role based access control (RBAC) rules that exist across Pulumi Cloud, ensuring that users only have access to stacks, environments and resources that they have permissions for within Pulumi Cloud. This means that each organization gets AI generated responses based solely on their own organizational data.

Pulumi Copilot is available now in public beta, and is free while in beta for all organizations in Pulumi Cloud. Organization administrators can turn on Pulumi Copilot for their organization by going to Settings > Access Management > Pulumi Copilot within the Pulumi Cloud console.

Try Pulumi Copilot now at https://app.pulumi.com/.

Walkthrough of Pulumi Copilot

Let’s jump in and take Pulumi Copilot for a spin!

We can start off simple - it’s a conversational chat, so we can introduce ourselves. Pulumi Copilot knows who we are from context and we get a light hearted programming related joke.

Perhaps we need to gain access to additional infrastructure within Pulumi Cloud and need to contact an admin, we can ask who the admins are.

We get a hyperlink at the end, enabling us to jump directly to the members page, where we can see all of the details about the members and admins of the organization. Pulumi Copilot helps to not just get information, but then navigate within the Pulumi Cloud console to continue to explore.

Pulumi Copilot has access to everything that Pulumi Cloud manages, including Pulumi ESC environments. We can find all of the environments we have that manage access to GitHub through a natural language query, and then ask a follow up question to get a direct link to where each environment is defined.

If we do a Pulumi update, and it fails, we often get a complex error message, caused by some combination of a coding problem or a cloud provider configuration problem or a transient failure in cloud providers backing service. We can use Pulumi Copilot to ask why an update failed, and get a plain language explanation, leveraging the AI’s existing knowledge of programming languages and cloud providers, combined with access to all of the update logs from within Pulumi Cloud. Notably, we can ask why “this” update failed, taking advantage of the context of the page we are on within Pulumi Cloud to understand which update we are referring to.

Perhaps we received an email from our compliance team telling us there was an issue with the VPC with id vpc-04a116f7075fb7ca5, but we have no idea where that is. Given we have hundreds of AWS accounts and dozens of regions in each of these accounts, finding this resource isn’t simple. We can just ask Pulumi Copilot, and it immediately tells us what AWS account it is in. We can follow this up with a question about which Pulumi project and stack is managing it, and get a link to go directly to the stack to learn more about how it is configured.

All of the capabilities of Pulumi AI for writing Pulumi IaC code are also available in Pulumi Copilot, so we can ask how to write a program for a new use case. But we can also have Pulumi Copilot incorporate information from an existing stack, leveraging multiple Pulumi Copilot skills to help with this task.

If we are thinking about removing some infrastructure we may want to know what other infrastructure potentially depends on it. We can get a list of stacks which depend on our stack, with hyperlinks directly to those stacks so we can explore their dependencies and take action to update them.

If we’re getting started on compliance with a new compliance framework, we can get guidance on areas to focus on for compliance review, ahead of applying more formal compliance tools like Pulumi Crossguard Compliance Ready Policies.

And we can get insights into what infrastructure is exposed to the internet which we may need to look deeper into with a quick question:

And our questions don’t have to be limited to English. Early Spanish-speaking users have found themselves working with Pulumi Copilot in their native language, getting responses automatically in the same language of their question.

All of these examples are just scratching the surface of what Pulumi Copilot can already do today. One of the most exciting things about conversational chat interfaces is their wonderfully wide surface area of potential applications - almost any question you might have related to Cloud Infrastructure managed within Pulumi Cloud is something Pulumi Copilot can help with!

Future of Pulumi Copilot

All of the examples above are things you can do today, but we’re not stopping there. There’s a lot more coming up, and we wanted to give a sneak peek into some of the directions we’re taking Pulumi Copilot based on early customer feedback.

Taking Action

In the initial Beta, Pulumi Copilot is restricted to get data and answer questions, but cannot take action directly, beyond guiding users to a place within the Pulumi Cloud console to take the action themselves. In the near future, we will be extending Pulumi Copilot with the ability to propose actions and then act on them with user approval.

Check out pulumi/cloud-requests#414 for more details.

Pulumi CLI Integration

One of the most common requests we have gotten from early Pulumi Copilot users has been for integration into the Pulumi CLI where many users spend much of their time day-to-day. We have started experimenting with CLI integration to help with diagnosing Pulumi update failures, and will be adding this to the CLI in the near future.

Check out pulumi/cloud-requests#415 for more details.

Pulumi Docs Website Integration

Pulumi Copilot is initially offered as part of Pulumi Cloud. But we are working to extend Pulumi Copilot across the rest of pulumi.com, including integration with the Pulumi Docs website. A new Pulumi Docs skill will allow Pulumi Copilot to help users learn Pulumi concepts and chat with Copilot about these to go deeper on concepts than what is already provided directly within the Pulumi Docs website.

Check out pulumi/cloud-requests#417 for more details.

Cloud CLI Skills

While Pulumi Copilot already has access to all the Cloud Infrastructure you are managing with Pulumi Cloud, there are many cases where it is useful to reach out to AWS directly to ask questions, get operational metrics, and take action. We are already working on extending Pulumi Copilot with the ability to run aws and kubectl CLI commands, using short-lived credentials from one of your Pulumi ESC environments. A sneak peek of that in action - showing how the AWS skill, combined with the ability to get details about resources and ESC environments from within Pulumi Cloud can enable collecting operational metrics from AWS and presenting them for analysis.

Check out pulumi/cloud-requests#416 for more details.

Conclusion

We’re excited to introduce Pulumi Copilot, the first conversational AI for general multi-cloud infrastructure management. Pulumi Copilot is the next big step toward reimagining cloud infrastructure management via AI-based user experiences. We’re just getting started, with many improvements and new capabilities coming soon on top of the core Pulumi Copilot foundation. Pulumi Copilot is brand new and improving quickly in response to user feedback. If you have questions or suggestions:

Get started with Pulumi Copilot in Pulumi Cloud.

Pulumi Blog: Copilot

Pulumi Updates, Explained: AI-Powered Features in Pulumi CLI

Update Explanation

Diagnostics Summarization

Getting Started

Share Your Feedback

A Recipe for a Better AI-based Code Generator

Using RAG for code generation

The anatomy of Pulumi Copilot RAG

Assessing search quality using recall and precision

Practical concerns

Semantic search with vector embeddings

Full text vs semantic search

Pruning the results

Prompt generation

Self-debugging

From kitchen to table: testing what works

Building confidence through testing

Monitoring quality in production

Wrapping up

Announcing the Pulumi Copilot REST API Preview

Enhancing Copilot Capabilities

Implementation Guide

Initial Configuration

Understanding Cloud Context

Making API Requests

Supporting Multi-turn Conversations

Moving Forward

AI Engineering Lessons from Building Pulumi Copilot

Engineering for Reality: Prompt Engineering vs Software Engineering

Copilot in Action: Real-World Dogfooding

Skillful Slicing: Modular Mastery

Debug Dispatch

The Illusion of Correctness

Testing the Untestable: Validating LLM Outputs

LLMs think like humans (sort of)

Introducing Pulumi Copilot: Intelligent Cloud Management

Pulumi Copilot Key Features

Access any data in Pulumi Cloud

Pulumi IaC Authoring and Deployment

Access cloud metadata from the clouds themselves

Walkthrough of Pulumi Copilot

Future of Pulumi Copilot

Taking Action

Pulumi CLI Integration

Pulumi Docs Website Integration

Cloud CLI Skills

Conclusion