Pulumi Blog: Artur Laksberg

Announcing Pulumi Remote MCP Server

Artur Laksberg — Tue, 07 Oct 2025 00:00:00 +0000

We’re excited to announce the Pulumi Remote MCP Server—a hosted service that brings AI-powered infrastructure management to any AI assistant that supports the Model Context Protocol. Connect your favorite AI assistant to https://mcp.ai.pulumi.com/mcp and instantly access your Pulumi Cloud infrastructure, search resources across stacks, and delegate complex automation tasks to Pulumi Neo.

The Evolution of Pulumi MCP

Earlier this year, we launched the Pulumi MCP server as a local npm package that brought AI-assisted infrastructure management to developers’ machines. The adoption and feedback from users and partners has been positive, validating the power of combining AI assistants with infrastructure-as-code.

As the MCP ecosystem has matured and more organizations have adopted the protocol, a clear pattern has emerged: remote MCP servers are becoming the industry standard. Remote servers provide a key advantage—accessibility. One endpoint works everywhere, with no per-machine setup.

Following industry trends and feedback from users and partners, we’re introducing the Remote MCP Server to ease installation and version management. The remote server preserves everything developers love about the local version while adding powerful new capabilities like seamless Pulumi Neo integration.

Note: The local MCP server continues to be available and fully supported for developers who prefer local tooling or need offline capabilities.

Why Remote MCP?

The Pulumi Remote MCP Server runs as a hosted service. Instead of managing local installations, you configure it once and get automatic updates and consistent functionality across all your development environments.

Zero local setup, universal access

Instead of installing npm packages, you simply configure your AI assistant with a single URL: https://mcp.ai.pulumi.com/mcp. That’s it.

No per-machine installations - Works the same on your laptop, desktop, or cloud workstation
No manual updates - New features and improvements roll out automatically to all users
Works with any MCP-compatible AI assistant - Cursor, Claude Code, Windsurf, Claude Desktop, and more

For instructions on how to configure different AI assistants, see Pulumi MCP Server.

Centralized authentication & secrets

Remote MCP also solves a critical security challenge: credential management. Instead of scattering Pulumi Access Tokens across laptops, containers, and scripts, the Remote MCP Server uses OAuth-based authentication with your Pulumi Cloud organization.

When you first connect, a browser window opens where you:

Enter your Pulumi Access Token (which is validated server-side)
Select which organization to access
Return to your AI assistant - now authenticated

Your credentials are stored securely in Pulumi Cloud, not on your individual machine.

What Can You Do With It?

The Remote MCP Server is your AI assistant’s gateway to your entire Pulumi infrastructure. It combines real-time access to your cloud resources with the power of autonomous infrastructure automation through Pulumi Neo.

Discover and query infrastructure

Your AI assistant can instantly explore what you’ve deployed across your entire organization:

List all stacks in your organization
Search for specific resources across all stacks
Find resources by type, name, tags, or any property
Check for policy violations
View organization members and their roles
Identify security gaps, untagged resources, or misconfigured infrastructure

Ask questions like:

“Show me all RDS databases without encryption enabled”
“Which stacks have resources in us-east-1?”
“Find all Lambda functions using deprecated runtimes”

Generate infrastructure code

The MCP server connects directly to the Pulumi Registry, giving your AI assistant access to thousands of cloud resources with complete type information:

Browse available resources
Get detailed resource schemas
Access property documentation, input/output types, and examples

Your AI assistant can:

Look up the exact properties for any cloud resource
Generate type-safe infrastructure code in TypeScript, Python, Go, or any Pulumi language
Include proper configurations, security settings, and best practices
Reference real documentation and examples

This means code generation is more accurate and up-to-date with the latest provider versions.

Autonomous infrastructure with Pulumi Neo

This is where the Remote MCP Server truly shines. For complex infrastructure tasks that require multiple steps, code changes, testing, and pull requests, your AI assistant can delegate directly to Pulumi Neo—Pulumi’s autonomous infrastructure AI agent.

What makes Neo special:

Neo isn’t just an AI that writes code—it’s an AI that ships infrastructure changes autonomously:

Multi-step planning - Neo breaks down complex requests into actionable plans
Code generation at scale - Works across multiple stacks and repositories
Automated testing - Validates changes before creating pull requests
Pull request workflows - Creates PRs with detailed explanations and comments
Continuous execution - Runs in Pulumi Cloud, not consuming your local resources
Human-in-the-loop - Pauses for approval on critical changes

Real-world Neo examples:

Security remediation: “Ask Neo to find all security groups allowing SSH from 0.0.0.0/0 and create a PR restricting them to my office IP”

Neo will:

Search your infrastructure for overly permissive security groups
Create a detailed plan for restricting access
Generate infrastructure code changes
Create a pull request with explanations
Wait for your approval to merge

Runtime migrations: “Ask Neo to migrate all Lambda functions from Python 3.8 to Python 3.12 and create PRs for each affected stack”

Neo handles:

Finding all Lambda functions with Python 3.8
Checking for compatibility issues
Updating runtime configurations
Running tests to ensure functionality
Creating separate PRs per stack for easy review

Policy compliance: “Ask Neo to scan for policy violations and fix them automatically”

Neo will:

Identify violations across all stacks
Generate fixes following your policy rules
Test changes to ensure compliance
Create PRs with clear explanations of what was fixed and why

Cost optimization: “Ask Neo to find idle resources and create a plan to shut them down”

Neo analyzes usage patterns, identifies waste, and proposes infrastructure changes to reduce costs—all autonomously.

The key difference: your AI assistant identifies what needs to be done, and Neo does it—writing code, running tests, creating PRs, and managing the entire workflow in Pulumi Cloud.

Real-World Workflow

Here’s what a typical session looks like:

You: “What stacks do I have with ‘production’ in the name?”

AI Assistant: Uses get-stacks to list: api-production, web-production, data-production

You: “Are there any policy violations in those stacks?”

AI Assistant: Uses get-policy-violations and reports: 3 S3 buckets without encryption, 2 security groups too permissive

You: “Ask Neo to fix those violations and create a PR”

AI Assistant: Uses neo-bridge to launch Neo task, provides link

Neo: Autonomously creates plan, generates fixes, tests changes, creates PR with detailed explanation

You: Review PR, approve, merge

Getting Started

Ready to try it? Check out our documentation for configuration instructions for your AI assistant of choice.

Key points:

Configure once - Add https://mcp.ai.pulumi.com/mcp to your AI assistant’s MCP settings
Authenticate - Browser popup for token entry and org selection (one time)
Start asking - Query your infrastructure, generate code, delegate to Neo

The Remote MCP Server is available now for all Pulumi users. No installation required—just configure and connect.

Learn More

We’re excited to see what you build with AI-assisted infrastructure management. Let us know what you think in our Community Slack!

Pulumi Updates, Explained: AI-Powered Features in Pulumi CLI

Artur Laksberg — Thu, 22 May 2025 00:00:00 +0000

Note: This post discusses Pulumi Copilot, which Pulumi Neo has replaced. Learn about Neo →

We’re excited to announce the new AI capabilities for Pulumi CLI powered by Pulumi Copilot that translate complex infrastructure changes into clear, human-readable explanations and help resolve deployment errors with actionable guidance. Enable these preview features with the --copilot flag.

At Pulumi, we’re committed to helping you deploy infrastructure efficiently and with minimal friction. Anyone who has worked with cloud infrastructure knows the frustration of sifting through large previews with numerous changes and the difficulty of interpreting cryptic error messages from cloud provider APIs when deployments fail.

There are two key challenges we’ve identified:

Understanding Preview Changes: Reviewing raw update previews can be overwhelming, especially for complex deployments with many resources. It’s difficult to quickly grasp the full scope and impact of pending infrastructure changes.
Interpreting Error Messages: When deployments fail, cloud providers often return error messages that, while comprehensive, bury the core issue within extensive diagnostics, requiring specialized expertise to interpret and address.

Our latest CLI update introduces AI capabilities that tackle both issues. Let’s explore how these new features improve your infrastructure management workflow.

Update Explanation

To address the first challenge above, we’ve introduced a new “explain” menu item in the CLI. When running pulumi preview or pulumi up with the --copilot flag, you’ll see this option in the interactive menu:

In this screenshot, you can see the standard Pulumi update preview showing an AWS VPC with multiple subnets, route tables, and gateways that need to be updated. The interactive menu at the bottom shows the new “explain” option with a sparkle emoji, indicating the AI-powered feature. When selected, this tool analyzes and provides a clear summary of all changes in your pending infrastructure update:

In this example, the AI explanation immediately identifies that this is a “VPC Renaming Operation - Tag Updates Only” and explains that the update is simply changing tags across all components. It clearly states this is a “safe, non-disruptive change that only affects resource labels.”

This explanation serves as a valuable sanity check before deployment. It presents a plain-language summary of exactly what changes will occur, helping you verify that the planned modifications align with your intentions. This visibility is particularly valuable when managing complex infrastructure with many resources, as it helps identify unexpected changes or potential drift between your code and deployed resources. The system also proactively flags potential issues such as security vulnerabilities or destructive updates that might impact your production environment.

Diagnostics Summarization

For the second challenge, when infrastructure deployments fail, cloud providers often return error messages that are technically accurate but difficult to decipher without specialized knowledge. Consider the following scenario where a Go-based Docker image build fails with vendoring issues:

In this example, the standard Go build process failed with a cryptic error about inconsistent vendoring. While the raw output does contain the necessary information, it’s buried within extensive build logs. Our AI-powered Copilot diagnostics immediately identifies the problem and provides a clear, actionable solution: “Docker image update failed due to inconsistent vendoring. Sync vendor with ‘go mod vendor’.”

This clear explanation identifies root causes and provides actionable guidance, transforming dense technical output into concise explanations. It reduces time to resolution by pinpointing both the problem and the exact command needed to fix it, eliminating the need to search through documentation or guess.

Getting Started

The AI-powered features are currently in preview. You can enable them by adding the --copilot flag to your Pulumi CLI commands:

pulumi preview --copilot
pulumi up --copilot

These features require Pulumi CLI version v3.171.0 or greater and the Pulumi Cloud backend. Organization administrators must enable Pulumi Copilot by navigating to Settings > Access Management > Pulumi Copilot in the Pulumi Cloud console.

During this preview period, we’re focused on:

Refining explanations based on user feedback
Adding new capabilities like suggesting fixes for common errors
Improving performance to ensure minimal impact on command execution time

We will continue to monitor usage patterns and incorporate your feedback before enabling these features by default.

We’re excited to see how these AI-powered insights enhance your infrastructure deployment workflow! Please share your experiences with us:

Connect with us in the Pulumi Community Slack
Open an issue on GitHub
Learn more about Pulumi Copilot

Try this today, Get Started with Pulumi for free

Your input will directly influence how we evolve these features from preview to general availability.

A Recipe for a Better AI-based Code Generator

Artur Laksberg — Tue, 07 Jan 2025 20:00:00 -0500

Note: This post discusses Pulumi Copilot, which Pulumi Neo has replaced. Learn about Neo →

When asked about his research process, Anthony Bourdain would describe how he’d blend his formal culinary training with deep dives into local food culture - from market stalls to family recipes. Modern AI code generation follows a similar path: it can’t just rely on what it knows - it must tap into continuously evolving, domain-specific knowledge bases. Just as Bourdain would combine his classical French training with techniques learned from local kitchens, AI code generators blend their built-in knowledge with retrieved code snippets and type definitions to generate code that accurately represents the user’s intent.

This fusion of base knowledge with contextual understanding is especially valuable for Infrastructure as Code (IaC), where rapidly evolving cloud providers and libraries make manual development challenging, traditional debugging cycles impractical, and errors catastrophically expensive.

The role of IaC won’t diminish in the age of AI - if anything, it will become even more central as systems grow increasingly complex and automated. Trustworthy code generators will be a key ingredient in the recipe for modern infrastructure management.

In this post, we share how we developed code generation for Pulumi and what we learned, based on both our production IaC generator powering Pulumi AI and Pulumi Copilot, as well as features and approaches we’re still exploring.

Using RAG for code generation

Pulumi supports over 120 providers, including major cloud providers such as AWS, Azure, Google Cloud, and Kubernetes, as well as many other services and platforms. New providers are continuously added, and existing providers change as their capabilities grow.

Our goal is to generate the most accurate code for every provider - code that is not only correct but also reflects their latest capabilities.

LLMs are great at generating code however they are limited to what they have learned before their knowledge cutoff date. This means that the latest changes in the providers will not be reflected in the generated code. This also could lead to hallucinations when the model struggles to answer a question that requires up to date knowledge.

To help us solve this, we rely on the technique known as the Retrieval Augmented Generation (RAG). RAG helps code generation by integrating information retrieved from external data sources. In Pulumi, we call this data source the Registry - it’s the database we maintain that contains type schema and usage information for every provider.

At a high level, using RAG involves the following steps:

Analyze user’s question.
Look up the pertinent information from the Registry.
Format that information in a system prompt that LLM can understand.
Make the LLM call asking it to generate the code using the additional information in the prompt.

There is lot of fascinating details here, so let’s dig in!

The anatomy of Pulumi Copilot RAG

Before going into the details, let’s consider a simple yet essential insight: an effective dataset for RAG must meet two key requirements:

It must contain the necessary information.
The information must be easily searchable.

Let’s take a sample user query:

“Generate code for S3 Bucket”

To fulfil the request, the system needs to intuit the following:

The cloud provider. This can be determined based on the fact that S3 is a common storage solution in AWS, and possibly the fact that the user’s organization has stated AWS as their preferred cloud provider.
The programming language. This information can again come directly from the organizational preferences, or from the user’s prior conversations.
The information about the type (or types) that must be created - its name and schema, the package it is in, and the capabilities it supports. Some of this information comes from the built-in LLM knowledge, supported and augmented by the Registry.

Putting it all together, we can now expand the original user query into the following prompt that is going to guide the code generation:

“Generate TypeScript code for S3 Bucket, the AWS resource defined in package @pulumi/aws, type aws.s3.Bucket with its schema defined as follows: …

While we had to rely on some guesswork to come up with this prompt, fortunately this process can be iterative - if we don’t get all of it right the first time, we can try again with additional information that will help us refine the results. This is an important point we return to later in the post.

Assessing search quality using recall and precision

To assess how effective the RAG is, we need to first understand the two fundamental concepts used in the information retrieval systems: the recall and the precision. Imagine that you’re looking for apple pie recipes in one of Jamie Oliver’s cookbooks. The book has a recipe for a classic American apple pie, a Dutch apple pie and a modern take on a French apple tart. Due to the book’s narrative approach with the recipes woven into the stories and context, you’ve managed to retrieve only the first two recipes but missed the French apple tart. Having retrieved 2 ouf 3 relevant documents, you have achieved a 67% recall.

Because you were looking for the word “pie”, you also retrieved a recipe for a Shepherd’s pie, which, while delicious, does not qualify as an apple pie. Another document that came up was a fish pie - a classic British dish that does not contain apples or even a pastry crust. Since only 2 of your 4 retrieved documents can be legitimately classified as apple pies, you have achieved a 50% precision.

Now let’s formalize this a bit. Recall measures the ratio of the relevant documents retrieved to the total number of relevant documents in RAG dataset:

$$Recall = \frac{N(Retrieved \cap Relevant)}{N(Relevant)}$$

Where

$N(Retrieved \cap Relevant)$ is the number of documents that are both retrieved and relevant.
$N(Relevant)$ is the total number of relevant documents in the database.

Good recall means that many documents relevant to the query were retrieved.

Precision is the ratio of the relevant documents retrieved to the total number of retrieved documents:

$$Precision = \frac{N(Retrieved \cap Relevant)}{N(Retrieved)}$$

Where $N(Retrieved)$ is the total number of documents that were retrieved.

High precision means that many of the retrieved documents were relevant.

Naturally, we want to maximize both the recall and the precision. It’s been said that high recall is essential to ensure relevant content is available to the code generator while precision is the parameter you want to optimize for to avoid hallucinations.

Practical concerns

Precision and recall are essential in understanding the information retrieval quality, but they are quite hard to measure in practice. Unlike a cookbook, Pulumi registry contains thousands of ever changing documents, and evaluating how many of them are relevant for every user-submitted query is impractical. This makes recall evaluation for live traffic next to impossible. Things are a little easier with precision, where we’re dealing with a small number of documents, but even that metric requires a non-trivial evaluation of relevance, which needs an LLM call or a human judge.

Fortunately, other metrics that often can effectively estimate retrieval quality have been developed. We have found a metric that can predict, with some degree of accuracy, whether the generated code will successfully compile. For this metric, we compare the tokens present in the LLM-produced prompt with the number of tokens present in the generated code. (By token here we understand a compiler token - an identifier such as the name of a class, method or a field and not a traditional LLM token concept), Intuitively, if a token present in the prompt also appears in the generated program, it can be assumed that the token contributed to the generated program. Tokens in the generated program that were not part of the prompt are not necessarily wrong but they are less trusted (they can come from the LLM built-in knowledge or were guessed)

$$prompt \ coverage = \frac{N(\text{Tokens in prompt} \cap \text{Tokens in code})}{N(\text{Tokens in code})}$$

Prompt coverage is a metric we can observe in production, and it’s one of several metrics we use when updating providers to ensure we haven’t regressed the quality of the RAG.

Semantic search with vector embeddings

Semantic search is based on the conceptual similarity of the term you’re looking for with the elements in the data store. For example, searching for “dumplings” can return terms like pierogi and gyoza - words with different spelling but both representing different types of filled dough preparations.

A common way to determine the similarity between the two strings is to first turn these strings into vector embeddings - arrays of floating point values representing the semantic meaning of each string - and then calculate the cosine similarity between the two vectors, which is the cosine of the angle between the vectors. Various methods of producing vector embeddings are fascinating but we will not cover them here in depth.

For Pulumi code generation we are using the OpenAI’s Ada-002 embedding model which at this moment represents a good balance between performance and cost.

Producing vector embeddings from the user query is the standard approach in this situation. However, for Pulumi code generator we added a little twist - to increase the odds of getting more relevant information from the Registry (i.e. to increase the recall) we first make an LLM call to generate a small set of relevant search terms that will produce an array of vector embeddings.

Getting vector embeddings from user query

We then use the array of vector embeddings to retrieve the set of relevant documents from the Registry.

Full text vs semantic search

While semantic search is essential for any modern information retrieval system, we should not forget simple and effective methods for text search already exist. The “S3 Bucket” part of the user search happens to be easily searchable using traditional text search operations (such as SQL LIKE operator).

To be effective, our RAG must be able to handle queries that require semantic understanding (such as “simple storage service in AWS”) as well as the traditional text search to support situations where the user knows exactly what they are looking for. To that end, the industry has adopted an approach known as the “hybrid search”, in which the results of full-text search and semantic search are combined to provide the final result.

For each of these search terms we generate a query that combines the full text search with the semantic search. The resulting documents are then evaluated based on the following criteria:

Dense score: Vector similarity using cosine distance between query embedding and stored embeddings
Sparse score: Text search relevance according to the full-using algorithm (PostgreSQL full-text search in our case)

Both the dense and the sparse scores contribute to the final score that is used to sort the documents. Even though the final score is the average of the two in our current implementation, it would not be correct to say that they have the same weight, or the influence on the end result. This is so because the normalization of the scores must take into account their distribution in the real-world queries and calculating that is quite complex.

Boosting the influence of one score relative to the other is an area we’re actively exploring, and achieving the optimal result depends highly on what parameters we decide to optimize for - high quality code, time to result and so on.

Finally, we apply the rank-based scoring technique in which the results are penalized based on their position in the ranking. This creates separation between results that might have similar initial scores and gives preference to results that appear earlier in the ranking.

Pruning the results

Our Pulumi code generator employs a two-phase document selection strategy. The first phase casts a wide net, gathering all potentially relevant documentation to maximize recall. The second phase applies precision-focused filtering to distill this collection down to the most pertinent documents.

This filtering step serves two purposes. First, it prevents LLM hallucinations that arise from similarly-named types across different providers. Second, it optimizes performance by keeping prompts concise - a critical consideration given that larger prompts increase both latency and computational costs, even when within context window constraints.

Through empirical testing with the Pulumi Registry search, we’ve established these baseline parameters: a maximum of 10 documents per query term selected by relevance score, and a 20K token ceiling for prompts. While these parameters have yielded good results in practice, they are likely not optimal for all scenarios. We continue to iterate on these values through ongoing experimentation.

Prompt generation

We’re now ready to create the system prompt for the code generation LLM call! We’ve already discussed some of the elements needed to build the effective prompt - the original user query, the search terms and the vector embeddings that produce the set of documents that will guide the code generation process.

There is another element that goes into the prompt - a concise set of instructions produced by an LLM call based on the original user’s query. The approach when the output of one prompt is used as input for another is known as “prompt chaining”, and we used it to provide step-by-step instructions to the code generator. For our query, this set of instructions can look as follows:

Create an S3 bucket using Pulumi in TypeScript using the following steps:

Import the necessary Pulumi AWS package.

Define a new S3 bucket resource with basic configuration.

Export the bucket name as an output.

Composition of the system prompt for code generation

Finally, we use the resulting prompt to call the LLM and ask it to generate the code. We’re done!

Self-debugging

Unfortunately, this isn’t always the final step of the process. Despite our best efforts, the code produced by the LLM will not always be correct.

At this point, it’s worth pondering what “correct code” means. The generated program might have the following problems:

It might be syntactically or semantically incorrect, i.e. it might not compile or fail to typecheck.
It might fail at runtime - for example by referring to a non-existing resource, a region and so on.
It might run “successfully” but not do what the user intended - either because the user did not express themselves clearly or because the request was misunderstood.
Finally, the code might run and do exactly what the user wants, but lead to an undesired outcome, for example loss of an asset, or a security vulnerability.

Solutions to many of these problems go well beyond the domain of code generation.

However, the first of these problems can be addressed by the approach known as “self-debugging”.

We observed that many generated TypeScript programs that fail to typecheck contain only a few errors, and asking the LLM to fix these errors often produces a valid program. While this approach is still experimental, we see promising results where getting to a correct program requires only 1-2 iterations of self-debugging. The biggest challenge with this approach is doing it in real-time. The user is staring at the screen waiting for an answer, so latency is a major concern.

Monitoring these typechecking errors in production can also provide valuable insight into the quality of the data used by the RAG and even suggest specific solutions. For example, failure to typecheck a member-access expression is a likely indicator of a missing type schema (a recall problem) or a “wrong” schema brought in by an irrelevant document (a precision problem).

Self-debugging can also be extended to include the pulumi preview command, which is a “dry run” operation before the actual deployment and can detect many real or potential problems such as destructive actions, incorrect configurations that cannot be detected at compile time, dependency conflicts, and policy violations.

From kitchen to table: testing what works

The landscape of LLM-based code generation is moving fast, and we need to keep learning and adapting as we go. But with all this rapid technological change, it’s crucial to ground our decisions in real numbers. We need to make sure each new advancement actually makes things better, both in our test environments and out in the real world.

The probabilistic nature of LLM-based code generation means we can’t rely solely on pre-production testing. Instead, we adopt multiple layers of quality control working together. Here’s what we’ve learned works best:

Building confidence through testing

Our testing strategy combines several approaches:

Systematic experimentation: We run “what-if” scenarios against various test datasets to understand how the system behaves under different conditions.
Local evaluation pipeline: While local testing has its limitations, it helps catch obvious issues early in development. We evaluate the performance of the code generator and run the code through the typechecker.
Deterministic generation: We set the LLM temperature to 0 to ensure consistent outputs. For code generation, consistency and repeatability matters more than creative variations.

Monitoring quality in production

We track several key metrics to ensure the system performs well:

RAG effectiveness: We measure prompt coverage to evaluate how well our retrieval system performs.
Code quality indicators: We track the percentage of generated code that successfully typechecks for all supported languages along with how this metric responds to system changes.
User feedback: Every “thumbs down” report gets analyzed to identify patterns and potential improvements. This direct user feedback has been invaluable in refining our system.

Wrapping up

Building an effective AI-powered code generator requires carefully balancing multiple concerns: the raw capabilities of LLMs with retrieved contextual knowledge, semantic search with traditional text matching, and thorough testing with real-world scenarios.

Our experience has taught us that success lies in treating code generation like a delicate soufflé - it requires precise measurements, careful monitoring, and constant refinement of technique. The key ingredients are:

A robust RAG system with well-tuned recall and precision.
End-to-end testing and monitoring across development and production.
Self-debugging capabilities to handle common issues like type checking errors.

As we continue to evolve Pulumi’s code generation capabilities, we’re excited about expanding our self-debugging features and further refining our RAG implementation.

We invite you to try these capabilities in your own projects and share your experiences. Your feedback helps us continue improving and advancing the state of AI-assisted infrastructure development.

Announcing the Pulumi Copilot REST API Preview

Artur Laksberg — Thu, 12 Dec 2024 15:15:42 -0500

Note: This post discusses Pulumi Copilot, which Pulumi Neo has replaced. Learn about Neo →

We built Pulumi Copilot to automate a broad spectrum of cloud management activities using the power of LLMs. Since its initial release earlier this year, hundreds of customers have used Pulumi Copilot to understand and manage cloud infrastructure more effectively and securely, and it is only getting better by the day.

Today, we’re excited to announce the availability of the Pulumi Copilot REST API. This new API exposes the full power of Pulumi Copilot, enabling you to integrate infrastructure AI into your own tools, applications, and platforms. While currently in preview, we are eager to get your feedback to ensure it works for anything you can dream up.

Enhancing Copilot Capabilities

With the Copilot REST API, you can extend the Copilot capabilities available in the Pulumi Console in the following ways:

Build Copilot capabilities into your platforms and tools, such as the CLI, development portals, Github actions and so on.
Support multi-user interaction in workplace collaboration platforms such as Slack and Teams.
Automate execution of Copilot queries based on scheduled triggers or events, such as deployment completions.
Access Copilot from mobile clients through platforms like Slack or Teams.

Implementation Guide

Initial Configuration

To begin using the Copilot API, set up the following environment variables:

export PULUMI_COPILOT_URL="https://api.pulumi.com/api/ai/chat/preview"
export PULUMI_ACCESS_TOKEN="pul-..."

Note: You can obtain your PULUMI_ACCESS_TOKEN from the Pulumi Console.

Understanding Cloud Context

All Copilot API interactions require two essential parameters:

Organization context through the orgId field - this is conceptually similar to the organization you select in the dropdown menu on the left-hand side of the Pulumi Console
Resource URL from the Pulumi Console, which must begin with https://app.pulumi.com - think of this as the browser URL you see when navigating the Pulumi Console. This URL helps Copilot understand the context of your query. While you can provide the base URL (https://app.pulumi.com), you can also point to specific resources or updates for more targeted queries. For example, https://app.pulumi.com/myorg/my-demo-project/my-stack/updates/5 would allow you to ask questions about that specific update

These parameters provide the necessary context for queries about specific resources or updates, allowing Copilot to respond with relevant information just as it would in the Console interface.

Making API Requests

Here’s an example of a basic API request:

curl -L "$PULUMI_COPILOT_URL" \
-H "Authorization: token $PULUMI_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
 "query": "Who are the users in my org?",
 "state": {
 "client": {
 "cloudContext": {
 "orgId": "pulumi",
 "url": "https://app.pulumi.com"
 }
 }
 }
}'

Supporting Multi-turn Conversations

The API supports continuous dialogues, in which participants can refer to information shared earlier in the chat. This is supported through conversation IDs that are received with the response:

"conversationId":"369a280c-63f3-4ee6-a13d-c1035a3d05de"

This ID enables follow-up queries that maintain context:

curl -L "$PULUMI_COPILOT_URL" \
-H "Authorization: token $PULUMI_ACCESS_TOKEN" \
-H "Content-Type: application/json" \
-d '{
 "conversationId":"369a280c-63f3-4ee6-a13d-c1035a3d05de",
 "query": "Who of them are admins?",
 "state": {
 "client": {
 "cloudContext": {
 "orgId": "pulumi",
 "url": "https://app.pulumi.com"
 }
 }
 }
}'

Moving Forward

As we continue to evole our AI efforts, your feedback is essential. We are particularly interested in hearing about your implementation experiences, including features that work well, areas that need improvement, or capabilities you’d like to see added. You can consult the documentation and peruse the samples. Please submit any feedback by opening an issue or by reaching out to us via our Community Slack

We can’t wait to see what you can build with Pulumi Copilot REST API!

AI Engineering Lessons from Building Pulumi Copilot

Artur Laksberg — Thu, 12 Dec 2024 13:56:13 -0500

Note: This post discusses Pulumi Copilot, which Pulumi Neo has replaced. Learn about Neo →

Building AI-powered developer tools comes with unique challenges, and now that we’ve launched our REST API, we want to share some lessons we’ve learned building Pulumi Copilot, an AI assistant for cloud infrastructure.

One of the big challenges was determining what ‘working’ really meant. So when a message landed in our feedback channel after months of rigorous testing - ‘Your tool doesn’t know anything!’ - it caused some mild panic. We’d just made some changes, so we braced for the worst. But our evals were still looking strong, so what was going on?

The user was attempting to force-delete a stack that still contained resources. But when we dug deeper, we found something fascinating: Copilot had confidently suggested a ‘–force’ flag, which would have been a logical solution… except this flag doesn’t exist in Pulumi. Our AI was hallucinating exactly what the user wanted. But this wasn’t just a bug - it was the first of many insights that would reshape how we approach AI-powered tools.

To understand how we got here - and why this “error” actually taught us something valuable about our product - let’s start with the core challenge we faced: balancing traditional software engineering with this new world of prompt engineering.

Engineering for Reality: Prompt Engineering vs Software Engineering

When building LLM-powered applications, it’s tempting to throw every task at the model. Modern LLMs can generate code, format text, and create clickable links. But this approach carries hidden costs.

Working on Copilot taught us a key lesson: let the LLM do what it does best and use good old imperative code for everything else.

Take a seemingly simple feature: listing a user’s Pulumi stacks with clickable links based on data from a backend API. Our first implementation used a complex prompt instructing the LLM to construct URLs in the format app.pulumi.com/org/project/stack. The prompt explained the format, provided examples, and asked the LLM to generate these links from JSON data it had.

It worked - almost all of the time. But we were seeing occasional malformed URLs and more importantly, this was burning input tokens (and money) on a complicated prompt that made AI construct strings that could be deterministically generated.

The solution was straightforward: generate the full links in the backend service and include them directly in the context. The LLM then needs no instructions on how to create them. Simple stuff, but it gave faster responses and perfect URLs at a lower cost.

When you find yourself writing elaborate prompts to handle structured data transformations, stop and ask: Could traditional code do this better? Could this be decomposed so that the LLM does less?

To validate this approach, we tested Copilot ourselves to see what worked.

Copilot in Action: Real-World Dogfooding

The internal testing phase taught us invaluable lessons about how people would actually use the tool. We watched our team try Copilot in their daily work, and three common use cases emerged:

Debugging Deployments: LLMs clearly excel at summarization. One of the first questions our internal users asked was, ‘Why did my latest infrastructure deployment fail?’ Using Copilot to extract a clear natural language explanation requests like these has been a clear win.

Understanding Complex Infrastructure: Copilot helped our engineers gain insights into Pulumi’s own infrastructure. Asking, ‘How many resources are in production?’ ‘What expensive compute is running’ or ‘What version are the EKS clusters in EU?" shows the value of allowing users to express infrastructure questions in natural language.

Generating Code: One of the first queries logged was, ‘I want a static website on AWS behind a CloudFront CDN.’ Another came from a Solutions Engineer, tasked with demonstrating Pulumi’s CrossGuard policy engine to a prospect, asking Copilot to generate policy code.

These early experiences showed the value of Copilot. But they also revealed the need for a systematic approach to handling diverse user queries. This led to the development of what we call skills.

Skillful Slicing: Modular Mastery

As Copilot grew, we broke it into smaller pieces we call skills. Each skill does one specific job. The Insights skill handles queries about resource usage and configuration (“How many S3 buckets do I have?”), the Cloud Skill interacts with the Pulumi Service API to manage infrastructure (“Show me my stacks.”), the Code Skill generates Pulumi code snippets (“Write a Typescript program…”), and the Docs Skill retrieves information from Pulumi documentation (“How do I use update plans?”).

When you ask Copilot something, it figures out what you need and picks the right skill for the job – like a manager deciding which expert to send your question to. This function-calling approach, orchestrated by a component we call the “outer loop,” allows Copilot to access and process information beyond its internal knowledge base.

Fortunately, Pulumi Cloud already exposes a rich API - in fact, this is what powers the Pulumi Console and the Pulumi CLI - so all we had to do is to build a skill that maps the user query to the appropriate Pulumi Cloud REST API. A question like “Show me my stacks” translates into the List Stacks API call. A question like ‘Show me my untagged EC2 instances’ is a bit more complex but it breaks down into clear components - resource type (EC2), filter condition (untagged) - that route to the Insights skill. This mapping helped us handle the many ways users phrase the same technical request.

Refining this routing system revealed another opportunity: streamlining the Debug button workflow.

Debug Dispatch

Originally, when a user clicked ‘Debug with Copilot’, the system would send a text query like “Analyze this update and explain any errors.” Copilot would then:

Determines the user wants to analyze an update
Identifies which API to call
Calls the API
Summarize the result

Having the LLM figure out what the user wants is the right approach in general, but in this case, we already know the user’s intent - they clicked a debug button. So, we can directly call the analysis API to get the results and use the LLM solely for what it does best: summarizing technical output into clear, actionable explanations.

This is another small win for the “Software Engineering over Prompt Engineering” approach. Traditional code handles the predictable parts, while AI focuses on the human-facing explanations.

But while minimizing the LLM workload helped with efficiency, we soon faced an even trickier challenge: the deceptive polish of AI-generated outputs.

The Illusion of Correctness

Large language models excel at generating well-structured, grammatically correct output. They make neat tables, tell good stories, and generally sound confident. That’s what makes them dangerous because this polished presentation can mask underlying flaws in the information itself, creating a false sense of confidence for users.

One of our early testers, Pablo, a data engineer at Pulumi, encountered this firsthand. He posed a query to Pulumi Copilot, asking for a summary of resources within a specific project. The response he received was impeccably formatted, neatly categorizing resources by type and providing counts for each. It looked right, and for us humans sometimes looking right carries a lot of weight.

However, a closer inspection revealed the numbers were way off. Copilot had asked for the wrong data and then summarized it beautifully - but incorrectly. This highlighted our next challenge: how do you systematically test a system that can be confidently wrong while sounding completely right?

Testing the Untestable: Validating LLM Outputs

When testing traditional code, we expect consistent, predictable outputs. With LLMs, even correct answers can vary. Here’s how we tackle this challenge.

Our first approach was simple: keyword checks. For example, when testing the update analysis feature, we checked if the LLM’s response included the word “security” and described the error. This worked for straightforward cases but broke down quickly. Take a question like, “How many Lambdas am I running?” The LLM might give the right numbers but skip the word “running,” failing the test even though the answer was correct.

These early failures revealed the limitations of keyword-based validation and underscored the need for a more nuanced approach. Inspired by platforms like LangSmith and Promptfoo, we began leveraging LLMs themselves as evaluators. For deterministic tasks, simple keyword checks suffice, but for more complex scenarios—like assessing whether a response answers a specific question—we rely on an “LLM Judge.” This approach balances efficiency and flexibility, reserving LLM evaluation for cases where it truly adds value. The test suite now integrates both methods, running against every code change to validate response content, accuracy, and format.

The eval suite keeps getting more robust, which means that when new AI models drop, we can quickly catch any weirdness before it hits production. The generative AI space moves crazy fast and the code changes a lot, but the evals are a safety net - catching hallucinations, maintaining quality, and making sure we don’t ship anything that’ll annoy our users.

So, while hallucinations are now much rarer, what about that one with the --force flag? Yes, it’s “just another bug,” but it taught us something fascinating about these AI errors.

You might also like:

LLMs think like humans (sort of)

The --force hallucination wasn’t totally wrong - it was revealing what users intuitively expect from the CLI, and the LLM accidentally showed us what was missing. Force deletion is a common pattern across developer tools, and the LLM, trained on vast amounts of documentation and code, simply reflects these established conventions.

This has fundamentally changed how we view hallucinations. While the team works constantly to minimize them – and our eval work means they happen way less frequently – some of them are clearly product signals. The LLM, in this light, becomes an unexpected source of user research, drawing on its training across thousands of developer tools and experiences.

This insight is one of the key lessons of building Copilot:

Minimize LLM Usage: Let traditional code handle deterministic tasks, reserve LLMs for natural language work
Decompose into Skills: Break complex tasks into modular units that combine LLM and traditional code appropriately
Test Rigorously: Use multiple validation approaches, including LLMs testing LLMs
Learn from Hallucinations: Sometimes incorrect outputs reveal user expectations
Learn from Users Continuously: User interactions improve our AI systems - from training better skills to catching hallucinations and revealing product opportunities.

These lessons helped shape our latest release: the Pulumi Copilot REST API, now available in preview. You can integrate these same capabilities and skills into your own tools and workflows. Whether you’re building CLI extensions, chat integrations, or automated deployment checks, the API provides a contextual understanding of Copilot. Try it out.

We can’t wait to see what you build!

Pulumi Copilot is Now Integrated with Pulumi Docs: A New Way to Learn and Explore

Meagan Cojocar — Thu, 24 Oct 2024 23:59:00 -0700

Note: This post discusses Pulumi Copilot, which Pulumi Neo has replaced. Learn about Neo →

Pulumi Copilot has been making our customers’ day-to-day tasks easier since its release, and today we’re excited to expand its capabilities—Pulumi Copilot is now available across Pulumi Documentation and pulumi.com, and comes equipped with a powerful new Documentation Skill!

These additions make learning about Pulumi more interactive and intuitive by allowing users to explore cloud infrastructure concepts directly within the documentation through a conversational interface. Whether you’re a seasoned Pulumi user or just starting out, Pulumi Copilot is here to help you when you need it most, where you need it most.

New Enhancements in Pulumi Copilot

Expanded Website Integration

Pulumi Copilot is now embedded throughout the Pulumi website, providing contextual assistance no matter where you are. Whether you’re exploring documentation, reading a blog post, browsing the Pulumi registry, or reviewing case studies, Pulumi Copilot offers helpful suggestions, explanations, and links to guide you to relevant resources.

Pulumi Documentation Skill

The new Documentation Skill is designed to make Pulumi’s docs more accessible from wherever. You can ask questions like “How do I use the dependsOn resource option?” or “How do I use SAML in my organization?" or “How can I store and retrieve secrets using Pulumi ESC?” and Pulumi Copilot will answer them by searching through our documentation on your behalf. This new skill is available everywhere Pulumi Copilot is: in Pulumi Cloud, in Pulumi documentation, across the Pulumi website, and the Registry.

Try Pulumi Copilot Today

Pulumi Copilot’s new Documentation Skill and website integration are available now! These features are free to use today. You can unlock more messages by being logged in to Pulumi Cloud.

To get started, visit the Pulumi Cloud console and enable Pulumi Copilot in your settings under Access Management.

Pulumi Copilot is already making waves, and with these latest enhancements, it’s now easier than ever to manage and learn about your cloud infrastructure. We can’t wait for you to experience it firsthand!

Enhancing Pulumi Copilot: Introducing System Prompts for Your Organization

Meagan Cojocar — Thu, 10 Oct 2024 00:00:00 -0700

Note: This post discusses Pulumi Copilot, which Pulumi Neo has replaced. Learn about Neo →

We are excited to announce a new feature for Pulumi Copilot: System Prompts. This enhancement empowers organizations to customize Pulumi Copilot’s responses for your organization, making your interactions with our AI assistant even more personalized to save you even more time.

Pulumi Copilot is a conversational chat interface integrated throughout Pulumi Cloud, enabling users to quickly accomplish a variety of cloud infrastructure management tasks by leveraging the power of large language models plus the rich capabilities of Pulumi Cloud. We released Pulumi Copilot in June this year, and have seen remarkable uptake across our customer base. We are excited to be announcing enhancements on the Pulumi Copilot experience- keep an eye out for more to come in the near future.

System prompts allow organization administrators to set default preferences and guidelines for Pulumi Copilot. By configuring these prompts, you can tailor Pulumi Copilot’s behavior to better suit your team’s needs and policies. Here are some ways you might use organization system prompts:

Set a Default Programming Language: Ensure that all code snippets and examples provided by Pulumi Copilot are in your team’s preferred language.
Specify a Default Cloud Provider: Streamline resource searches and code generation by focusing on your primary cloud environment. Specify providers your organization prefers to use, such as using Azure Native instead of Azure Classic.
Enforce Compliance Guidelines: Instruct Copilot to generate code and provide guidance that adheres to specific compliance standards, such as SOC 2.
Standardize Infrastructure Components: Provide example code templates for resources to ensure consistency across your organization’s projects, such as “here is how we create VPCs”

How to Set Organization System Prompts

Setting up system prompts is straightforward:

Log in to Pulumi Cloud and go to your organization’s settings in the left hand navigation
Under the settings menu, select “Access Management.”
In the Copilot section, you’ll find the option to set your system prompts.

Though the system prompt supports 10k characters of context, we recommend keeping it concise to ensure optimal performance. See more information on setting system prompts in our Pulumi Copilot documentation.

Wrapping it up

This is just the beginning. We’re dedicated to making Pulumi Copilot an indispensable part of your cloud infrastructure management toolkit. Stay tuned for more updates and enhancements based on your feedback.

If you have suggestions or encounter any issues, please let us know through our Pulumi Cloud requests GitHub repository.

Pulumi Blog: Artur Laksberg

Announcing Pulumi Remote MCP Server

The Evolution of Pulumi MCP

Why Remote MCP?

Zero local setup, universal access

Centralized authentication & secrets

What Can You Do With It?

Discover and query infrastructure

Generate infrastructure code

Autonomous infrastructure with Pulumi Neo

Real-World Workflow

Getting Started

Learn More

Pulumi Updates, Explained: AI-Powered Features in Pulumi CLI

Update Explanation

Diagnostics Summarization

Getting Started

Share Your Feedback

A Recipe for a Better AI-based Code Generator

Using RAG for code generation

The anatomy of Pulumi Copilot RAG

Assessing search quality using recall and precision

Practical concerns

Semantic search with vector embeddings

Full text vs semantic search

Pruning the results

Prompt generation

Self-debugging

From kitchen to table: testing what works

Building confidence through testing

Monitoring quality in production

Wrapping up

Announcing the Pulumi Copilot REST API Preview

Enhancing Copilot Capabilities

Implementation Guide

Initial Configuration

Understanding Cloud Context

Making API Requests

Supporting Multi-turn Conversations

Moving Forward

AI Engineering Lessons from Building Pulumi Copilot

Engineering for Reality: Prompt Engineering vs Software Engineering

Copilot in Action: Real-World Dogfooding

Skillful Slicing: Modular Mastery

Debug Dispatch

The Illusion of Correctness

Testing the Untestable: Validating LLM Outputs

LLMs think like humans (sort of)

Pulumi Copilot is Now Integrated with Pulumi Docs: A New Way to Learn and Explore

New Enhancements in Pulumi Copilot

Expanded Website Integration

Pulumi Documentation Skill

Try Pulumi Copilot Today

Enhancing Pulumi Copilot: Introducing System Prompts for Your Organization

How to Set Organization System Prompts

Wrapping it up