Pulumi Blog: Engineering

How We Eliminated Long-Lived CI Secrets Across 70+ Repos

Boris Schlosser — Tue, 31 Mar 2026 00:00:00 +0000

Supply chain attacks on CI/CD pipelines are accelerating. A growing pattern involves attackers compromising popular GitHub Actions through tag poisoning — rewriting trusted version tags to point to malicious code that harvests environment variables, cloud credentials, and API tokens from runner environments. The stolen credentials are then exfiltrated to attacker-controlled infrastructure, often before anyone notices.

For every engineering organization, the question is no longer if your CI pipeline will encounter a compromised dependency, but what is exposed when it does.

At Pulumi, we asked ourselves that question and decided the answer should be “nothing useful.” Here’s how we got there.

The problem with static CI secrets

Most organizations store long-lived cloud credentials, API tokens, and service account keys as GitHub repository or organization secrets. But this approach has several well-known problems:

Broad availability. Every workflow run on a repository can access every secret stored in that repo. A compromised action in any workflow can read them all.
No expiration. Secrets persist until someone manually rotates them. If exfiltrated, they give attackers persistent access for weeks or months.
No granular audit trail. GitHub tells you a secret was used, but not which workflow, which step, or what it was used for.
Secret sprawl. Across dozens or hundreds of repos, the same credentials are often duplicated, making rotation a coordinated, error-prone effort.

In a supply chain attack scenario, this is exactly what attackers count on: a single compromised action that can dump a trove of long-lived credentials.

Our approach: zero static secrets

We replaced every static GitHub Secret across our CI pipelines with short-lived, dynamically fetched credentials using Pulumi ESC and OpenID Connect (OIDC). The credential flow works in layers, each scoped and ephemeral:

GitHub generates a short-lived OIDC token scoped to the specific workflow run, repository, and branch. This token is cryptographically signed by GitHub’s OIDC provider.
The token is exchanged with Pulumi Cloud for a short-lived Pulumi access token. Pulumi Cloud validates the OIDC claims (organization, repository, branch) against a configured trust policy before issuing the token.
The Pulumi access token opens an ESC environment to retrieve the credentials the workflow needs — cloud provider keys, API tokens, or other secrets.
Cloud credentials themselves are dynamic. ESC environments use OIDC login providers to fetch short-lived credentials directly from AWS, Azure, or GCP. No static keys or cloud credentials are stored anywhere.

The pulumi/esc-action GitHub Action handles this entire flow in a single workflow step.

sequenceDiagram
participant Runner as GitHub Actions Runner
participant GH as GitHub OIDC Provider
participant PC as Pulumi Cloud
participant ESC as Pulumi ESC
participant Cloud as Cloud Provider (AWS/Azure/GCP)
Runner->>GH: Request OIDC token (scoped to workflow run)
GH-->>Runner: Short-lived JWT
Runner->>PC: Exchange JWT for Pulumi access token
PC-->>Runner: Short-lived access token
Runner->>ESC: Open environment with access token
ESC->>Cloud: OIDC login (assume role / federated identity)
Cloud-->>ESC: Short-lived cloud credentials
ESC-->>Runner: Cloud credentials + secrets
Note over Runner,Cloud: Nothing is stored. Everything expires.

What the change looks like

Before this migration, our workflows referenced static secrets stored in GitHub:

env:
 AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
 AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

After the migration, an ESC environment handles credential fetching via OIDC. Here is what the environment definition looks like:

values:
 aws:
 login:
 fn::open::aws-login:
 oidc:
 duration: 1h
 roleArn: arn:aws:iam::123456789012:role/pulumi-esc-role
 sessionName: esc-${context.pulumi.user.login}
 # Optional: scope down the session beyond what the role allows
 policyArns:
 - arn:aws:iam::123456789012:policy/ci-build-minimal
 environmentVariables:
 AWS_ACCESS_KEY_ID: ${aws.login.accessKeyId}
 AWS_SECRET_ACCESS_KEY: ${aws.login.secretAccessKey}

The roleArn and optional policyArns make least-privilege straightforward: each login provider assumes a specific role, and policyArns can scope the session down further. You can use multiple login providers in one environment or separate environments per workflow to match permissions to each job’s needs.

The workflow itself becomes minimal — a single step that authenticates via OIDC and injects the credentials:

permissions:
 contents: read
 id-token: write  # Required for OIDC

steps:
 - name: Fetch secrets from ESC
 uses: pulumi/esc-action@v3
 with:
 environment: '<your-organization>/<your-esc-env>'

The static secrets.* references are gone entirely. Every credential is fetched at runtime through ESC.

Scale: 70+ repos, zero static secrets

We didn’t do this for one or two flagship repos; we rolled it out across every Pulumi provider repository: AWS, Azure, GCP, Kubernetes, and over 60 more. The migration was managed centrally through our ci-mgmt tooling, which generates consistent workflow configurations across all provider repos.

The pattern is the same everywhere:

Each repo has a corresponding ESC environment under a github-secrets/ project.
All workflow-level ${{ secrets.* }} references have been removed.
The pulumi/esc-action step with OIDC auth is the single entry point for all credentials.

When every repo follows the same pattern like this, security posture is much more easily verifiable and auditable.

Auditability and centralized control

Beyond eliminating static secrets, this migration gave us centralized visibility and control that GitHub Secrets cannot provide:

Audit logging. ESC records which credentials were accessed, when, and by which workflow. This is a meaningful improvement over GitHub’s binary “secret was used” signal.
Centralized access policies. Access rules are defined once in ESC rather than scattered across individual repository settings pages.
Single-point rotation. Because ESC environments can import other environments, shared credentials live in a common base that all 70+ repo environments are composed of. Update it once, and every repo picks up the change on its next run.
Dynamic credentials by default. For cloud providers like AWS, Azure, and GCP, ESC fetches credentials via OIDC at open time. There is nothing to rotate because nothing is stored.

What happens if a GitHub Action is compromised

With this architecture in place, here is what an attacker gets if a compromised GitHub Action runs in our CI:

No GitHub Secrets to dump. The repository settings page has no stored secrets for a malicious action to exfiltrate.
OIDC tokens are scoped and short-lived. The GitHub-issued JWT is valid only for the specific workflow run and expires within minutes.
Cloud credentials are ephemeral. Any AWS, Azure, or GCP credentials fetched through ESC are short-lived and scoped to the role assumed during that run.
No persistent access. There are no long-lived tokens to reuse hours or days later.

Compare this to the traditional model, where a single compromised action could exfiltrate AWS access keys that remain valid until someone manually rotates them — which could be weeks or months.

The goal is not to prevent every possible attack. It is to make the blast radius as small as possible when something goes wrong.

Get started

If you want to adopt the same pattern in your own CI pipelines:

Tutorial: Using ESC with GitHub Actions — Step-by-step setup guide.
Announcing the Pulumi ESC GitHub Action — Full feature overview and capabilities.
Configuring OIDC for ESC — Set up OIDC trust between ESC and your cloud providers.
Pulumi ESC documentation — Full documentation for environments, secrets, and configuration.

Your CI secrets do not have to be a liability. With OIDC and Pulumi ESC, they do not have to exist at all.

How We Built a Distributed Work Scheduling System for Pulumi Cloud

Levi Blackstone — Thu, 26 Feb 2026 00:00:00 +0000

Pulumi Cloud orchestrates a growing number of workflow types: Deployments, Insights discovery scans, and policy evaluations. Some of that work runs on Pulumi’s infrastructure, and some of it runs on yours via customer-managed workflow runners. We needed a scheduling system that could handle all of these workflow types reliably across both environments. In this post, we’ll take a look at the system we built.

Where we started

For our first workflow integration, Deployments, scheduling wasn’t too complicated. A deployment was queued, a worker picked it up, and it ran. The queue was purpose-built for deployments, and it worked well for that single use case. Over time, we added more sophisticated logic to handle retries, ordering, rate limiting, observability, and more.

With the launch of Insights, the number of workflow types grew. Now Pulumi Cloud manages discovery scans to catalog cloud resources and runs audit policy evaluations to continuously verify compliance. While these workflows share similarities, each type needed its own scheduling, retry logic, and failure handling.

Later we added the option for customers to run workflows on their own infrastructure using customer-managed workflow runners. As the complexity of these requirements grew, we knew that our initial approach for Deployments wasn’t going to scale. We needed a single system that could schedule any type of work, route it to the right place, and handle the messy reality of distributed execution: crashes, network failures, rate limits, and retries.

We call this the background activity system.

Why not use an off-the-shelf queue?

Why build this instead of using Amazon SQS, RabbitMQ, or one of the many existing queue libraries? We considered these options but chose to build our own for a few reasons.

Pulumi Cloud supports self-hosted installations, including air-gapped environments. We intentionally minimize external dependencies so that self-hosted customers don’t have to stand up additional infrastructure. A system built on an external queue works fine for our hosted service, but it means self-hosted customers would need to provide a compatible backend. By building on top of the database we already require, we avoid adding another system to maintain.

More importantly, queueing is only part of the problem. What we actually need is scheduling with durability. This means ensuring that remote workers don’t lose activities on restart, priority so that urgent work gets compute resources first, constraints like “only so many scans per org at a time,” structured logging for observability, and checkpointing so that long-running operations can resume after a failure.

These features can be layered onto a generic queue library but can require more code than implementing them directly. For example, priority queues are often implemented with multiple ranked queues, but this breaks single-activity-at-a-time constraints. A second queue wouldn’t see a job already running in the first one. There’s no way for producers in a distributed system to coordinate across the queues without support in the queuing system itself.

Capacity management is another area where generic queues fall short. Distributed systems need to respond dynamically to slowdowns, network interruptions, and rate limits from downstream services. These are common low-level details that every workflow type needs, and building them into the scheduling layer means individual handlers don’t have to solve them independently.

We also need structured logging that works everywhere, including on customer-managed runners behind firewalls where centralized logging services aren’t accessible.

Building this ourselves gave us a system that works with existing infrastructure and handles these requirements natively.

Design constraints

With that context, here are some of the constraints that shaped the design:

Pull-only agents. Customer-managed workflow runners live behind NATs, corporate proxies, and air-gapped networks. They can’t accept inbound connections, so all communication has to be agent-initiated.
Mixed execution environments. The same system needs to work for Pulumi-hosted workers (with direct access to internal systems) and customer-managed runners (communicating entirely over REST). We didn’t want to maintain two separate code paths.
Different workflow types. Deployments, Insights scans, and audit policy evaluations have different payloads and execution semantics, but they all need the same scheduling guarantees: exactly-once execution, automatic retries, failure recovery, and observability.
Automatic fault tolerance. Agents crash, networks drop, and machines get recycled by autoscalers. The system needs to detect these failures and recover without needing a person to step in.
Extensibility. We knew we’d keep adding workflow types. Adding a new one should mean writing a handler and registering it, not building new infrastructure.

The background activity

At the center of the system is the background activity, a persistent, typed work unit. Each activity includes:

A type discriminator that identifies what kind of work it represents (e.g., “insights-discovery” or “policy-evaluation”)
A payload specific to that type, containing whatever data the handler needs
A routing context that determines which runner pool should execute it
Scheduling metadata like priority, activation time, and retry configuration
A status tracking where the activity is in its lifecycle

The type discriminator makes this system polymorphic. The scheduling engine doesn’t need to know what’s inside the payload. It moves activities through their lifecycle and delegates the actual work to a type-specific handler.

The state machine

Every activity follows the same lifecycle regardless of type:

---
config:
flowchart:
curve: linear
---
graph LR
Start(( )) -->|Created| Ready
Ready -->|Leased| Pending
Pending -->|Started| Executing
Executing -->|Success| Completed
Completed --> End(( ))
Executing -->|Error| Failed
Failed --> End
Executing -->|Canceled| Canceled
Canceled --> End
Executing -->|Dependencies| Waiting
Waiting -->|Unblocked| Ready
Pending -->|Lease expired| Restarting
Executing -->|Lease expired| Restarting
Restarting -->|Re-lease| Ready
style Start fill:#000,stroke:#000,color:#000
style End fill:#000,stroke:#000,color:#000

The states fall into two groups:

Running states (work is in flight or can be resumed):

Ready: queued and eligible to be claimed by a worker
Pending: claimed by a worker, execution about to start
Executing: actively running on a worker
Waiting: parked, blocked on one or more dependency activities
Restarting: recovered after a worker failure, ready to be re-claimed

Terminal states (work is done):

Completed, Failed, Canceled

New workflow types get these features automatically: scheduling, retries, dependency management, and observability.

Leases: distributed execution without coordination

A central challenge of any distributed work queue is preventing double-execution. If two agents try to execute the same activity simultaneously, you get duplicate work and data corruption. A central coordinator can solve this, but it becomes a single point of failure.

We use lease-based optimistic concurrency instead. This is a well-known pattern, adapted here for long-running, stateful workflows.

How it works

When an agent is ready for new work, it asks the service to lease an activity. The service atomically selects the highest-priority ready activity, assigns a lease token with an expiration time, and transitions the activity to Pending. No other agent can claim the same activity.

sequenceDiagram
participant Agent
participant Service
Agent->>Service: Poll for work
Note right of Service: Select highest-priority<br/>Ready activity
Note right of Service: Atomically set lease<br/>token + expiration
Service->>Agent: Lease (token, expiration)
Agent->>Service: Begin execution
Note right of Service: Transition to Executing
Note over Agent: Work in progress...
Agent->>Service: Renew lease
Service->>Agent: New expiration
Note over Agent: Work continues...
Agent->>Service: Complete (token, result)
Note right of Service: Transition to Completed<br/>Archive activity
Service->>Agent: Acknowledged

While executing, the agent periodically renews its lease to signal that it’s still working. If the agent crashes, loses network connectivity, or is terminated, it stops renewing. Once the lease expires, the service transitions the activity to Restarting, making it available for another agent to claim.

sequenceDiagram
participant A as Agent A
participant S as Service
participant B as Agent B
A->>S: Lease activity
S->>A: Token + expiration
Note over A: Executing...
A--xS: Agent A crashes
Note right of S: Lease expires
Note right of S: Transition → Restarting
B->>S: Poll for work
Note right of S: Lease to Agent B<br/>Transition → Pending
S->>B: Token + expiration
Note over B: Agent B continues execution

The service doesn’t need to explicitly coordinate between workers because leases are acquired using atomic database operations. The lease expiration is the failure detector; if a lease expires, then the work needs to be rescheduled.

Routing work to the right runner pool

Pulumi Cloud supports multiple workflow runner pools. An organization might have one pool for production in us-east-1, another for staging in eu-west-1, and use Pulumi-hosted runners for development. Work needs to reach the right pool.

Each activity carries a routing context that identifies which runner pool should execute it. When a runner polls for work, it filters by its own pool identifier so that it only sees activities meant for it.

We use prefix matching for this filtering. A runner matches activities whose context starts with its pool’s identifier. This means the service can use hierarchical contexts (e.g., pool-abc/insights/scan-123) and runners will still match on the pool prefix. Cleanup is also straightforward; when a runner pool is deleted, all activities with that context prefix are bulk-canceled.

This routing mechanism works the same way regardless of workflow type, and adding a new workflow type doesn’t require changes to the routing layer.

Dependencies and multi-step workflows

Some workflows are naturally multi-step. An Insights discovery scan might discover resources that then need policy evaluation. Rather than building a separate orchestration engine, we built dependency management into the activity system.

An activity can declare a dependency set: a list of other activities that must complete before it can run. A dependent activity enters the Waiting state when created. As its dependencies complete, the system checks whether all prerequisites are satisfied. When the last one finishes, the waiting activity transitions to Ready and enters the scheduling queue.

graph TD
A["Insights Discovery<br/><b>Executing</b>"] -->|depends on| B["Policy Evaluation<br/><b>Waiting</b>"]
A -->|completes| C["Insights Discovery<br/><b>Completed</b>"]
C -->|triggers| D["Policy Evaluation<br/><b>Ready</b><br/>(auto-scheduled)"]

This gives us a lightweight DAG of work without requiring a separate workflow engine. Dependent activities get the same guarantees as any other activity: lease-based execution, automatic recovery, and observability.

Two execution modes, one interface

This is where the design really pays off for customer-managed runners. The system supports two execution modes:

Direct mode runs in-process alongside the Pulumi Cloud service. Workers have low-latency access to internal systems and can process activities with minimal overhead. This is what Pulumi-hosted runners use.
Remote mode communicates over REST APIs. The runner polls for activities, leases them, executes work locally, and reports results back over HTTP. This is what customer-managed runners use. No database access, no internal network access, no inbound connectivity required.

Both modes share the same handler interface so that a workflow handler doesn’t need to know where it’s running. Whether it’s running on Pulumi’s hosted infrastructure or on a customer’s Kubernetes cluster, the handler simply processes the payload and reports a result.

Putting it all together

Let’s walk through a concrete example. A user wants to run an Insights discovery scan on an AWS account using a customer-managed workflow runner.

sequenceDiagram
participant User
participant PC as Pulumi Cloud
participant Runner as Workflow Runner<br/>(Customer Infrastructure)
User->>PC: 1. Configure Insights scan<br/>for runner pool
Note right of PC: 2. Create background activity<br/>type: insights-discovery<br/>context: runner-pool-xyz<br/>status: Ready
Runner->>PC: 3. Poll for work (filtered by pool)
PC->>Runner: 4. Lease activity (token + expiration)
Runner->>PC: 5. Initialize workflow
PC->>Runner: Return cloud credentials + job token
Note over Runner: 6. Execute scan locally<br/>(talks directly to cloud APIs —<br/>credentials are used only on<br/>the runner)
Runner->>PC: 7. Renew lease
Note right of PC: Extend expiration
Runner->>PC: 8. Report completion
Note right of PC: 9. Mark completed<br/>Archive activity
Note right of PC: 10. Unblock dependent activities<br/>(e.g., policy evaluation)

A user configures an AWS account for Insights scanning in Pulumi Cloud and assigns it to a workflow runner pool.
Pulumi Cloud creates a background activity with the type set to insights discovery, the routing context set to the runner pool, and the payload containing the account configuration.
A customer-managed workflow runner polling that pool detects new work.
The runner leases the activity, acquiring an exclusive lock via the lease token.
The runner initializes the workflow, receiving any required cloud provider credentials (e.g., resolved from Pulumi ESC (Environments, Secrets, and Configuration)) and a job token from Pulumi Cloud.
The runner executes the scan locally on the customer’s infrastructure, talking directly to the cloud provider APIs.
During execution, the runner periodically renews its lease to signal liveness.
The scan completes, and the runner reports the result back to Pulumi Cloud.
The service marks the activity as completed and archives it.
If a dependent policy evaluation activity was waiting on this scan, it automatically transitions to Ready and enters the scheduling queue, where another runner in the pool can pick it up.

This flow works the same way whether the runner is hosted by Pulumi or by the customer. The only difference is whether the execution mode is direct or remote.

Retries and scheduling

Failures are expected in distributed systems. The background activity system handles them at several levels:

Lease expiration covers hard failures like agent crashes, network partitions, and machine terminations. If a lease expires, the activity moves to Restarting, and is available for another agent to pick up.
Handler-controlled retries cover soft failures like transient API errors and rate limits. A handler can request a reschedule with a delay, putting the activity back in Ready with a future activation time.
Automatic retries provide a configurable retry budget per activity. Each activity can specify how many times it should be retried and the delay between attempts, preventing runaway retry loops.
Priority scheduling ensures urgent work gets processed first. Higher-priority activities are leased before lower-priority ones, even if the lower-priority activity has been waiting longer.
Lease renewal during slowdowns keeps the activity alive without blocking other work, even if a downstream service is slow. The agent continues renewing its lease while it waits, and the scheduler remains free to assign other activities to other agents.

Observability

Every activity generates a structured log of its execution, including timestamps, severity levels, and code context. Logs are stored with the activity record and are accessible via an API and admin tooling.

This is especially useful for customer-managed runners, where the service can’t directly observe the execution environment. The structured log gives operators visibility into the execution context, even when the runner is behind a firewall. Handlers can also use these logs as a progress journal, encoding checkpoints that allow a restarted activity to pick up where it left off rather than starting from scratch.

Retention policies are configurable per organization and per workflow type. Completed activities can be retained for auditing or purged to manage storage, and failed activities are typically retained longer for debugging.

What we learned

A generic system pays off quickly. Our initial instinct was to build targeted solutions for each workflow type. Investing in a generic activity system required more upfront design work, but now adding a new workflow type requires a fraction of the effort it would take otherwise. New workflows ship with full scheduling, retry, and observability support from day one.

Leases handle many failure modes. We evaluated several approaches for distributed work coordination, including message queues with explicit acknowledgment and coordinator-based assignment. The lease model works well because all failure modes are handled through timeouts. If an agent is running as expected, it renews. If it isn’t, the lease expires.

Keeping the execution paths symmetric requires discipline. Making the hosted and self-hosted paths share the same handler interface was a deliberate choice. It would be easy to add shortcuts for the hosted path that bypass the remote API, but resisting that temptation means that features work for both cases automatically.

The hard part isn’t running the work. Running a scan or a deployment is straightforward once you have the right credentials. The real complexity is in everything around the execution: scheduling, routing, leasing, retrying, resolving dependencies, and cleaning up. These operational concerns aren’t visible to users, but they are essential to providing a reliable experience.

Wrapping it up

Today this system powers deployments, Insights discovery scans, and policy evaluations across both Pulumi Cloud and customer-managed infrastructure. The architecture is general enough that every new workflow type we add inherits the full scheduling, routing, retry, and observability stack without additional plumbing.

If you’re interested in running workflows on your own infrastructure, check out customer-managed workflow runners. To see how Insights can help you understand and manage your cloud infrastructure, get started with Pulumi Insights.

How We Built Platybot: An AI-Powered Analytics Assistant

Pablo Seibelt — Wed, 11 Feb 2026 00:00:00 +0000

Before Platybot, our #analytics Slack channel was a support queue. Every day, people from every team would ask questions: “Which customers use feature X?”, “What’s our ARR by plan type?”, “Do we have a report for template usage?” Our two-person data team was a bottleneck.

Our #analytics channel, before Platybot (dramatized).

We didn’t want to just throw an LLM at our Snowflake warehouse either. Without guardrails, large language models generate SQL that may work but silently gets the answer wrong. Different join logic, wrong filters, missing snapshot handling, incorrect summarization. We needed something that could answer reliably for most queries, otherwise we’d switch to fixing LLM SQL queries.

So we built Platybot (platypus + bot, named after our mascot), an AI-powered analytics assistant that any Pulumi employee can use to query our Data Warehouse in natural language. It’s available as a Web App, a Slack bot, and a Model Context Protocol (MCP) server. The infrastructure is deployed with Pulumi IaC (Infrastructure as Code). But the most important thing we learned building it is that the AI was the easy part. The semantic layer is what makes it work.

The problem with throwing AI at your Data Warehouse

The naive solution is obvious: connect an LLM to your database and let it write SQL. But this fails in practice, and the failure mode is insidious. Consider a few examples from our warehouse:

ARR is a snapshot metric. If you query ARR (Annual Recurring Revenue) without filtering by end-of-period dates (last day of month or quarter), you get duplicate rows and wildly inflated numbers. An LLM doesn’t automatically know this.
Account queries need exclusions. Most queries should exclude Pulumi’s own internal accounts and deleted ones. Without these filters, you’re counting test data alongside real customers.
User queries need employee filters. Querying active users without excluding Pulumi employees inflates adoption metrics.

The danger shows up when results feel decision-ready before anyone has validated how the numbers were derived. A confidently wrong ARR figure presented to leadership is worse than no answer at all.

Many organizations run into the same constraint once data usage spreads. Dashboards answer yesterday’s questions, not today’s. Ad-hoc LLM queries answer today’s questions, but incorrectly. The gap between “I have a question” and “I have a trustworthy answer” is where data teams can get stuck.

Why we built a semantic layer first

Before writing a single line of AI code, we built a semantic layer using Cube (open source). This was the hardest, least glamorous, and most important part of the entire project.

A semantic layer is a shared, versioned definition of what your business metrics mean.

“Monthly active users” starts with COUNT(DISTINCT user_id), but the aggregation is only the outer layer. It has to be applied to the right table (fct_pulumi_operations), with the right filters (exclude Pulumi employees, exclude deleted organizations), scoped to a calendar month, and counting only users who performed real operations — not just previews. The semantic layer encodes all of this once, and the AI can use it to build queries without needing to guess which tables are related to each other, whether it’s one-to-one or many-to-many, etc.

We organized our data into seven domains: Revenue, Cloud, Core, Clickstream, Community, Support, and People. Each domain contains cubes (think of them as well-defined, composable views) with explicit measures, dimensions, and joins. Here’s a real example from our Cloud domain (trimmed for readability):

cubes:
 - name: fct_pulumi_operations
 sql_table: CLOUD.FCT_PULUMI_OPERATIONS
 description: >
 Pulumi CLI operations (update, preview, destroy, refresh).
 Each row is one operation with resource changes, duration,
 and CLI environment.

 joins:
 - name: dim_organization
 sql: '{CUBE}.ORGANIZATION_HK = {dim_organization}.ORGANIZATION_HK'
 relationship: many_to_one
 - name: dim_stacks
 sql: '{CUBE}.STACK_PROGRAM_HK = {dim_stacks}.STACK_PROGRAM_HK'
 relationship: many_to_one
 - name: dim_user
 sql: '{CUBE}.USER_HK = {dim_user}.USER_HK'
 relationship: many_to_one

 measures:
 - name: count
 type: count
 - name: resource_count
 sql: '{CUBE}.RESOURCE_COUNT'
 type: sum
 description: Number of resources active when the operation finished
 - name: operations_succeeded
 sql: "CASE WHEN {CUBE}.OPERATION_STATE = 'succeeded' THEN 1 ELSE 0 END"
 type: sum
 description: Count of operations that succeeded
 # ... plus other measures

 dimensions:
 - name: operation_type
 sql: '{CUBE}.OPERATION_TYPE'
 type: string
 description: Type of operation (Refresh, Update, Destroy, etc)
 - name: operation_state
 sql: '{CUBE}.OPERATION_STATE'
 type: string
 description: Last known state of this operation
 # ... 20+ more dimensions

The key insight: the semantic layer makes the AI’s job tractable. Instead of generating arbitrary SQL from scratch, where the search space is “any possible SQL query against hundreds of tables,” the AI picks from a defined set of measures, dimensions, and joins. The search space shrinks from almost infinite to a well-bounded set of valid combinations.

A semantic layer is to an AI data assistant what a type system is to a programming language. It doesn’t eliminate errors, but it makes entire categories of mistakes structurally impossible. The AI can’t calculate ARR wrong because it doesn’t calculate ARR at all. It references a pre-defined measure that already encodes the correct logic.

Building the semantic layer was mainly data engineering work. We already had agreed-upon metric definitions across the company. The challenge was encoding those definitions into Cube: specifying the correct joins between tables, wiring up the right filters, and making sure every measure matched the logic our dashboards already used. Tedious, but essential.

Adding the AI layer

With the semantic layer in place, the AI becomes a translation problem: convert natural language into a Cube query.

Platybot supports multiple models: Claude Opus 4.6, Claude Sonnet 4.5, and Gemini 3 Pro. Users can choose which model to use. We found that different models have different strengths. Claude excels at structured data queries, while Gemini performs very well on text-heavy tasks like analyzing call transcriptions.

The system prompt gives the model awareness of available cubes, their measures, dimensions, and joins. When a user asks “What’s the ARR breakdown by plan type?”, the model doesn’t write SQL. Instead, it constructs a Cube query, selecting the total_arr measure from the ARR table and grouping by the sku from the subscriptions dimension. Cube handles the SQL generation, the joins, and the filters. For edge cases the semantic layer doesn’t cover, the model can fall back to direct (read-only) SQL against Snowflake, but it may already have a basic query that is already close to what it needs.

Query generation follows a workflow that maps to the same tools a human analyst would use:

flowchart TB
subgraph Entry["Entry points"]
Web["Web UI"]
Slack["Slack (@platybot)"]
MCP["MCP Server"]
end
subgraph Core["Platybot core"]
BE["Backend (Express + LLM)"]
end
subgraph Data["Data layer"]
Cube["Cube semantic layer"]
SF["Snowflake"]
end
Web --> BE
Slack --> BE
MCP --> BE
BE -->|"Structured queries"| Cube
BE -->|"Fallback SQL (read-only)"| SF
Cube --> SF

The workflow is: discover domains, explore cubes, understand the schema, construct and execute the query. This mirrors how a data analyst would work. You don’t jump straight to SQL; you first understand what data is available and what the metrics mean.

The system’s role is narrower than people expect. It’s a translator between human intent and a well-defined data model, not a general-purpose data scientist. This is a feature, not a limitation. By constraining Platybot to operate within the semantic layer, we get reliability. The creativity goes into understanding the question, not into inventing SQL.

Meeting users where they are

The backend solved query generation. It did not solve usage and if people don’t use it, it doesn’t matter. We launched Platybot across three interfaces, each designed for a different workflow.

The web UI

The web app is the primary interface: a React 19 + TypeScript + Vite + Tailwind conversational UI where employees can explore data through multi-turn conversations. It supports table visualization, data export, and conversation history so you can pick up where you left off.

Every analysis produces a shareable report with a permanent link, protected behind company authentication. The report shows the agent’s reasoning so you can verify its approach, and lists every query and table used along with a sample of the data. Each query includes a direct link to run it in Metabase, our reporting tool, so any useful query can be saved, scheduled, or extended without starting from scratch.

The web UI sees the highest adoption of all three access points. It’s where people go for deeper data exploration: multi-step analyses, follow-up questions, and comparing metrics across different dimensions.

One fun detail we’ve added during our last hackathon: while queries iterate through analysis steps (discovering domains, exploring cubes, executing queries), users are entertained by a platypus-themed runner game. Think Chrome’s dinosaur game, but with our mascot on a skateboard. Sometimes Platybot can take a long time iterating, so you can try to beat your high score in the meantime!

Slack

Platybot is also available as a Slack bot. Users @mention it in any channel, and it replies in a thread with the answer, keeping the channel clean while making results visible to the whole team. It’s best suited for quick lookups — “what’s the ARR for account X?” — where the answer fits in a message rather than a deep analysis. Most usage stayed in the web UI, but the Slack bot fills a different niche: answers that benefit from being shared in context.

MCP: Making Platybot usable by Agents

The Model Context Protocol (MCP) server is the newest addition, launched February 4, 2026. It exposes six tools that any MCP-compatible client can use:

list_domains - discover available data domains
list_cubes - explore cubes within a domain
get_cube_details - get measures, dimensions, and joins for a cube
execute_cube_query - run a structured query
execute_sql_query - raw SQL fallback (read-only)
get_generated_sql - preview SQL without executing

Setup is a single command:

claude mcp add --transport http platybot <platybot-mcp-url>

This is a paradigm shift. Platybot goes from a destination (open the app, ask a question) to a capability that any AI tool can use. An engineer writing a postmortem can pull live metrics without leaving their terminal. An analyst building a report in Claude Code can query data inline. The Data Warehouse becomes ambient, always available, never in the way.

Security uses OAuth 2.0 Device Authorization Flow with PKCE for CLI-friendly authentication, with @pulumi.com domain restriction, read-only enforcement, rate limiting, and full audit logging.

A meta moment: we used the Platybot MCP server to query our Slack messages table to find the real usage data cited above.

Deploying with Pulumi

Platybot’s infrastructure runs on AWS and is managed entirely with Pulumi IaC. The stack includes ECS Fargate for the application, an Application Load Balancer for traffic routing, EFS for persistent storage, ECR for container images, and Route 53 for DNS. The setup is intentionally simple and we use a small instance size since it’s just for internal usage.

Here’s a representative snippet of the Fargate service definition:

const service = new awsx.ecs.FargateService("platybot", {
 cluster: cluster.arn,
 assignPublicIp: false,
 taskDefinitionArgs: {
 container: {
 name: "platybot",
 image: image.imageUri,
 essential: true,
 portMappings: [{
 containerPort: 3000,
 targetGroup: targetGroup,
 }],
 environment: [
 { name: "CUBE_API_URL", value: cubeApiUrl },
 { name: "NODE_ENV", value: "production" },
 ],
 secrets: [
 { name: "ANTHROPIC_API_KEY", valueFrom: anthropicKeySecret.arn },
 { name: "SNOWFLAKE_PASSWORD", valueFrom: snowflakePasswordSecret.arn },
 ],
 logConfiguration: {
 logDriver: "awslogs",
 options: {
 "awslogs-group": logGroup.name,
 "awslogs-region": aws.config.region!,
 "awslogs-stream-prefix": "platybot",
 },
 },
 },
 },
});

Dogfooding Pulumi for our internal tools has real benefits beyond the obvious. Staging environments are trivial: spin up a full copy of the stack with different parameters. Secrets management is built in, so API keys and database credentials never touch a config file. And when something needs to change, the diff-and-preview workflow (pulumi preview) catches mistakes before they hit production.

Results

Since launch in September 2025: over 1,700 questions from 83 employees across every team. Usage grew steadily — from around 8 questions per day in the first month to 18 per day by January 2026, with 51 unique users that month alone. This was real production work, not experimentation. Customer analysis for sales calls, resource breakdowns for account managers, blog performance metrics for marketing, policy adoption research for product, ARR deep-dives for leadership.

The impact on the data team was immediate. Questions that used to land in the #analytics channel and wait for a human now get answered in seconds. The data team shifted from answering routine queries to building better models and improving data quality. We went from being a help desk to being a platform team.

Rumors of our replacement have been greatly exaggerated. Platybot just freed us up for the work that doesn't fit in a Slack reply.

Accuracy is harder to quantify, but the semantic layer gives us confidence. Because Platybot uses pre-defined measures instead of generating arbitrary SQL, entire categories of errors (wrong joins, missing filters, incorrect aggregations) are structurally less likely. When the model does get something wrong, it’s usually in interpreting the question, not in the data — and users can verify that through the report’s reasoning trace.

What we learned

The semantic layer matters more than the model. We spent more time defining metrics in Cube than we did on any AI work. That investment pays for itself: swap the model, and the answers stay correct. Swap the semantic layer, and nothing works.

Meet users where they already are. Different UIs handle different cognitive loads. Slack catches the quick questions, the web UI handles deep exploration, and MCP makes data ambient for AI-native workflows. Each interface serves a different mode of thinking.

Transparency builds trust. Showing the AI’s reasoning, the queries it ran, and linking to Metabase for verification turned skeptics into regular users. People don’t trust a black box, but they’ll trust a tool that shows its work.

Speeding up Pulumi Operations by up to 20x

Thomas Gummerer — Mon, 12 Jan 2026 18:57:55 +0200

Today we’re introducing an improvement that can speed up operations by up to 20x. At every operation, and at every step within an operation, pulumi saves a snapshot of your cloud infrastructure. This gives pulumi a current view of state even if something fails mid-operation, but it comes with a performance penalty for large stacks. Here’s how we fixed it.

Benchmarks

Before getting into the more technical details, here are a number of benchmarks demonstrating what this new experience looks like. To run the benchmarks we picked a couple of Pulumi projects: one that can be set up massively parallel, which is the worst case scenario for the old snapshot system, and another that looks a little more like a real world example. Note that we conducted all of these benchmarks in Europe, connecting to Pulumi Cloud, which runs in AWS’s us-west-2 region, so exact numbers may vary based on your location and internet connection. This should however give a good indication of the performance improvements.

We’re benchmarking two somewhat large stacks, both of which are or were used at Pulumi. The first program sets up a website using AWS bucket objects. We’re using the aws-ts-static-website example here with a small subset of the fraction from our docs site. This means we’re setting up more than 3000 bucket objects, with 3222 resources in total.

The benchmarks were measured using the time built-in command and using the best time in a best-of-three benchmark. The network traffic was measured using tcpdump, limiting the measured traffic to only the IP addresses for Pulumi Cloud. Finally, tshark was used to process the packet captures and count the bytes sent.

All the benchmarks are run with journaling off (the default experience) and with journaling on (the new experience). To begin with, let’s look at the results when creating our stack from scratch:

	Time	Bytes sent
Without journaling	58m26s	16.5MB
With journaling	02m50s	2.3MB

Now let’s have a look at what this looks like if we only change half the resources, but the remaining ones remain unchanged:

	Time	Bytes sent
Without journaling	34m49s	13.8MB
With journaling	01m45s	2.3MB

The second example is setting up an instance of the Pulumi app and API. Here we’ll have an example that’s a bit more dominated by the cost of setting up the actual infrastructure in the cloud, but we still have a very noticeable improvement in the time it takes to set up the stack.

	Time	Bytes sent
Without journaling	17m52s	18.5MB
With journaling	9m12s	5.9MB

To use this feature, you need a pulumi version newer than v3.211.0, and set the PULUMI_ENABLE_JOURNALING environment variable to true.

If you are interested in the more technical details read on!

Introduction to snapshotting

pulumi keeps track of all resources in a stack in a snapshot. This snapshot is stored in the stack’s configured backend, which is either the Pulumi Cloud or a DIY backend. Future operations on the stack then use this snapshot to figure out which resources need to be created, updated or deleted.

pulumi creates a new snapshot at the beginning and at the end of each resource operation to minimize the possibility of untracked changes even if a deployment is aborted unexpectedly (for example due to network issues, power outages, or bugs).

At the beginning of the operation, pulumi adds a new “pending operation” to the snapshot. Pending operations declare the intent to mutate a resource. If a pending operation is left in the snapshot (in other words the operation started, but pulumi couldn’t record the end of it), the next operation will try to resolve this. If we have an ID for the resource already, for example on partial updates/deletes, pulumi will try to read the resource state from the cloud and resolve it internally. If there is no ID yet, pulumi will ask the user to check the actual state of the resource. Depending on the user’s response, pulumi will either remove the operation from the snapshot or import the resource. This is because it is possible that the resource has been set up correctly or that the resource creation failed. If pulumi aborted midway through the operation, it’s impossible to know which state the resource is in.

Once an operation finishes, the pending operation is removed and the resource’s final state is recorded in the snapshot.

There’s also some additional metadata that is stored in the snapshot that is only updated infrequently.

Here’s how the snapshot looks in code. This snapshot is serialized and sent to the backend. Resources holds the list of known resource states and is updated after each operation finishes, and PendingOperations is the list of pending operations described above.

type Snapshot struct {
 Manifest Manifest // a deployment manifest of versions, checksums, and so on.
 SecretsManager secrets.Manager // the secrets manager to use when serializing this snapshot.
 Resources []*resource.State // all resources and their associated states.
 PendingOperations []resource.Operation // all currently pending resource operations.
 Metadata SnapshotMetadata // metadata associated with the snapshot.
}

Before we dive in deeper, we also need to understand a little bit about how the pulumi engine works internally. Whenever a pulumi operation is run (e.g. pulumi up, pulumi destroy, pulumi refresh etc.), the engine internally generates and executes a series of steps, to create, update, delete etc. resources. To maintain correct relationships between resources, the steps need to be executed in a partial order such that no step is executed until all of the steps it depends on have executed successfully. Steps may otherwise execute concurrently.

As each step is responsible for updating a single resource, we can generate a snapshot of the state before each step starts, and after it completes. Before each step starts, we create a pending operation, and add it to the PendingOperations list. After that step completes, we remove the pending operation from that list, and update the Resources list, either adding a resource, removing it, or updating it, depending on the kind of operation we just executed.

After this introduction, we can dive into what’s slow, how we fixed it, and some benchmarks.

Why is it slow?

To make sure the state is always as up-to-date as possible, even if there are any network hiccups/power outages etc., a step won’t start until the snapshot that includes the pending operation is confirmed to be stored in the backend. Similarly an operation won’t be considered finished until the snapshot with an updated resources list is confirmed to be stored in the backend.

To send the current state to the backend, we simply serialize it as a JSON file, and send it to the backend. However, as mentioned above, steps can be executed in parallel. If we uploaded the snapshot at the beginning and end of every step with no serialization, there would be a risk that we overwrite a new snapshot with an older one, leading to incorrect data.

Our workaround for that is to serialize the snapshot uploads, uploading one snapshot at a time. This gives us the data integrity properties we want, however it can slow step execution down, especially on internet connections with lower bandwidth, and/or high latency.

This impacts performance especially for large stacks, as we upload the whole snapshot every time, which can take some time if the snapshot is getting big. For the Pulumi Cloud backend we improved on this a little at the end of 2022. We implemented a diff based protocol, which is especially helpful for large snapshots, as we only need to send the diff between the old and the new snapshot, and Pulumi Cloud can then reconstruct the full snapshot based on that. This reduces the amount of data that needs to be transferred, thus improving performance.

However, the snapshotting is still a major bottleneck for large pulumi operations. Having to serially upload the snapshot twice for each step does still have a big impact on performance, especially if many resources are modified in parallel. Furthermore, the time spent performing textual diffs between snapshots scales in proportion to the size of the data being processed, which adds additional execution time to each operation.

Fast, but lacking data integrity?

As long as pulumi can complete its operation, there’s no need for the intermediate checkpoints. We could allow pulumi operations to skip uploading the intermittent checkpoints to the backend. This, of course, avoids the single serialization point we have sending the snapshots to the backend, and thus makes the operation much more performant.

However, it also has the serious disadvantage of compromising some of the data integrity guarantees pulumi gives you. If anything goes wrong during the update, pulumi has no notion of what happened until then, potentially leaving orphaned resources in the provider, or leaving resources in the state that no longer exist.

Neither of these solutions is very satisfying, as the tradeoff is either performance or data integrity. We would like to have our cake and eat it too, and that’s exactly what we’re doing with journaling.

Enter journaling

To achieve this, we went back to the drawing board, and asked ourselves, “What would a solution look like that’s both performant and preserves data integrity throughout the update?”.

Making that happen is possible because of three facts:

We always start with the same snapshot on the backend and the CLI.
Every step the engine executes affects only one resource.
We have a service that can reconstruct a snapshot from what is given to it.

(The third point here already hints at it, but this feature is only available and made possible by Pulumi Cloud, not on the DIY backend).

What if instead of sending the whole snapshot, or a diff of the snapshot, we could send the individual changes to the base snapshot to the service, which could then apply it, and reconstruct a full snapshot from it? This is exactly what we are doing here, in the form of what we call journal entries. Each journal entry has the following form:

const (
 JournalEntryKindBegin JournalEntryKind = 0
 JournalEntryKindSuccess JournalEntryKind = 1
 JournalEntryKindFailure JournalEntryKind = 2
 JournalEntryKindRefreshSuccess JournalEntryKind = 3
 JournalEntryKindOutputs JournalEntryKind = 4
 JournalEntryKindWrite JournalEntryKind = 5
 JournalEntryKindSecretsManager JournalEntryKind = 6
 JournalEntryKindRebuiltBaseState JournalEntryKind = 7
)

type JournalEntry struct {
 // Version of the journal entry format.
 Version int `json:"version"`
 // Kind of journal entry.
 Kind JournalEntryKind `json:"kind"`
 // Sequence ID of the operation.
 SequenceID int64 `json:"sequenceID"`
 // ID of the operation this journal entry is associated with.
 OperationID int64 `json:"operationID"`
 // ID for the delete Operation that this journal entry is associated with.
 RemoveOld *int64 `json:"removeOld"`
 // ID for the delete Operation that this journal entry is associated with.
 RemoveNew *int64 `json:"removeNew"`
 // PendingReplacementOld is the index of the resource that's to be marked as pending replacement
 PendingReplacementOld *int64 `json:"pendingReplacementOld,omitempty"`
 // PendingReplacementNew is the operation ID of the new resource to be marked as pending replacement
 PendingReplacementNew *int64 `json:"pendingReplacementNew,omitempty"`
 // DeleteOld is the index of the resource that's to be marked as deleted.
 DeleteOld *int64 `json:"deleteOld,omitempty"`
 // DeleteNew is the operation ID of the new resource to be marked as deleted.
 DeleteNew *int64 `json:"deleteNew,omitempty"`
 // The resource state associated with this journal entry.
 State *ResourceV3 `json:"state,omitempty"`
 // The operation associated with this journal entry, if any.
 Operation *OperationV2 `json:"operation,omitempty"`
 // If true, this journal entry is part of a refresh operation.
 RebuildDependencies bool `json:"isRefresh,omitempty"`
 // The secrets manager associated with this journal entry, if any.
 SecretsProvider *SecretsProvidersV1 `json:"secretsProvider,omitempty"`

 // NewSnapshot is the new snapshot that this journal entry is associated with.
 NewSnapshot *DeploymentV3 `json:"newSnapshot,omitempty"`
}

These journal entries encode all the information needed to reconstruct the snapshot from them. Each journal entry can be sent in parallel from the engine, and the snapshot will still be fully valid. All journal entries have a Sequence ID attached to them, and they need to be replayed in that order on the service side to make sure we get a valid snapshot. It is however okay to replay without journal entries that have not yet been received by the service, and whose sequence ID is thus missing. This is safe because the engine only sends entries in parallel for resources whose parents/dependencies have been fully created and confirmed by the service.

This way we make sure that the resources list is always in the correct partial order that is required by the engine to function correctly, and for the snapshot to be considered valid.

The algorithm looks as follows:

# Apply snapshot writes. This replaces the full snapshot we have on the service.
# We do this if default providers change, because we don't emit steps for that, as
# we do for the rest of the operations.
snapshot = find_write_journal_entry_or_use_base(base, journal)
# Track changes
deletes, snapshot_deletes, mark_deleted, mark_pending = set(), set(), set(), set()
operation_id_to_resource_index = {}
# Process journal entries. This is the main algorithm, that adds new resources
# to the snapshot, removes existing ones, deals with refreshes, and operations
# that update outputs.
incomplete_ops = {}
has_refresh = false
index = 0
for entry in journal:
match entry.type:
case BEGIN:
incomplete_ops[entry.op_id] = entry
case SUCCESS:
del incomplete_ops[entry.op_id]
if entry.state and entry.op_id:
resources.append(entry.state)
operation_id_to_resource_index.add(entry.op_id, index)
index++
if entry.remove_old:
snapshot_deletes.add(entry.remove_old)
if entry.remove_new:
deletes[remove_new] = true
if entry.pending_replacement:
mark_pending(entry.pending_replacement)
if entry.delete:
mark_deleted(entry.delete)
has_refresh |= entry.is_refresh
case REFRESH_SUCCESS:
del incomplete_ops[entry.op_id]
has_refresh = true
if entry.remove_old:
if entry.state:
snapshot_replacements[entry.remove_old] = entry.state
else:
snapshot_deletes.add(entry.remove_old)
if entry.remove_new:
if entry.state:
deletes[entry.remove_new] = true
else:
resources.replace(operation_id_to_resource_index(entry.remove_new), entry.state)
case FAILURE:
del incomplete_ops[entry.op_id]
case OUTPUTS:
if entry.state and entry.remove_old:
snapshot_replacements[entry.remove_old] = entry.state
if entry.state and entry.remove_new:
resources.replace(operation_id_to_resource_index(entry.remove_new), entry.state)
deletes = deletes.map(|i| => operation_id_to_resource_index[i])
# Now that we have marked all the operations, and created a new list of resources, we can
# go through them, and merge the list of new resources and old resources from the snapshot
# that remain together.
for i, res in resources:
if i in deletes:
remove_from_resources(resources, i)
# Merge snapshot resources. These resources have not been touched by the update, and will
# thus be appended to the end of the resource list. We also need to mark existing resources as
# `Delete` and `PendingReplacement` here.
for i, res in enumerate(snapshot.resources):
if i not in snapshot_deletes:
if i in snapshot_replacements:
resources.append(snapshot_replacements[i])
else:
if i in mark_deleted:
res.delete = true
if i in mark_pending:
res.pending_replacement = true
resources.append(res)
# Collect pending operations. These are stored separately from the resources list
# in the snapshot.
pending_ops = [op.operation for op in incomplete_ops.values() if op.operation]
pending_ops.extend([op for op in snapshot.pending_ops if op.type == CREATE])
# Rebuild dependencies if necessary. Refreshes can delete parents or dependencies
# of resources, without affecting the resource itself directly. We need to now remove
# these relationships to make sure the snapshot remains valid.
if has_refresh:
rebuild_dependencies(resources)

The full documentation of the algorithm can be found in our developer docs.

Rollout

pulumi state is a very central part of pulumi, so we wanted to be extra careful with the rollout to make sure we don’t break anything. We did this in a few stages:

We implemented the replay interface inside the pulumi CLI, and ran it in parallel with the current snapshotting implementation in our tests. The snapshots were then compared automatically, and tests made to fail when the result didn’t match.
Since tests can’t cover all possible edge cases, the next step was to run the journaler in parallel with the current snapshotting implementation internally. This was still without sending the results to the service. However we would compare the snapshot, and send an error event to the service if the snapshot didn’t match. In our data warehouse we could then inspect any mismatches, and fix them. Since this does involve the service in a minor way, we would only do this if the user is using the Cloud backend.
Next up was adding a feature flag for the service, so journaling could be turned on selectively for some orgs. At the same time we implemented an opt-in environment variable in the CLI (PULUMI_ENABLE_JOURNALING), so the feature could be selectively turned on by users, if both the feature flag is enabled and the user sets the environment variable. This way we could slowly start enabling this in our repos, e.g. first in the integration tests for pulumi/pulumi, then in the tests for pulumi/examples and pulumi/templates, etc.
Allow users to start opting in. If you want to opt-in with your org, please reach out to us, either on the Community Slack, or through our Support channels, and we’ll opt your org into the feature flag. Then you can begin seeing the performance improvements by setting the PULUMI_ENABLE_JOURNALING env variable to true.
Turn on the feature flag for everyone, but still require the PULUMI_ENABLE_JOURNALING env variable to be set to true. (We are here right now).
Flip the feature on by default, but still allow users to opt out using a PULUMI_DISABLE_JOURNALING env variable.

What’s next

While these performance improvements hopefully make your day to day use of pulumi quicker and more enjoyable, we’re not quite done here. We’re looking at some other performance improvements, that will hopefully speed up your workflows even more.

A Recipe for a Better AI-based Code Generator

Artur Laksberg — Tue, 07 Jan 2025 20:00:00 -0500

Note: This post discusses Pulumi Copilot, which Pulumi Neo has replaced. Learn about Neo →

When asked about his research process, Anthony Bourdain would describe how he’d blend his formal culinary training with deep dives into local food culture - from market stalls to family recipes. Modern AI code generation follows a similar path: it can’t just rely on what it knows - it must tap into continuously evolving, domain-specific knowledge bases. Just as Bourdain would combine his classical French training with techniques learned from local kitchens, AI code generators blend their built-in knowledge with retrieved code snippets and type definitions to generate code that accurately represents the user’s intent.

This fusion of base knowledge with contextual understanding is especially valuable for Infrastructure as Code (IaC), where rapidly evolving cloud providers and libraries make manual development challenging, traditional debugging cycles impractical, and errors catastrophically expensive.

The role of IaC won’t diminish in the age of AI - if anything, it will become even more central as systems grow increasingly complex and automated. Trustworthy code generators will be a key ingredient in the recipe for modern infrastructure management.

In this post, we share how we developed code generation for Pulumi and what we learned, based on both our production IaC generator powering Pulumi AI and Pulumi Copilot, as well as features and approaches we’re still exploring.

Using RAG for code generation

Pulumi supports over 120 providers, including major cloud providers such as AWS, Azure, Google Cloud, and Kubernetes, as well as many other services and platforms. New providers are continuously added, and existing providers change as their capabilities grow.

Our goal is to generate the most accurate code for every provider - code that is not only correct but also reflects their latest capabilities.

LLMs are great at generating code however they are limited to what they have learned before their knowledge cutoff date. This means that the latest changes in the providers will not be reflected in the generated code. This also could lead to hallucinations when the model struggles to answer a question that requires up to date knowledge.

To help us solve this, we rely on the technique known as the Retrieval Augmented Generation (RAG). RAG helps code generation by integrating information retrieved from external data sources. In Pulumi, we call this data source the Registry - it’s the database we maintain that contains type schema and usage information for every provider.

At a high level, using RAG involves the following steps:

Analyze user’s question.
Look up the pertinent information from the Registry.
Format that information in a system prompt that LLM can understand.
Make the LLM call asking it to generate the code using the additional information in the prompt.

There is lot of fascinating details here, so let’s dig in!

The anatomy of Pulumi Copilot RAG

Before going into the details, let’s consider a simple yet essential insight: an effective dataset for RAG must meet two key requirements:

It must contain the necessary information.
The information must be easily searchable.

Let’s take a sample user query:

“Generate code for S3 Bucket”

To fulfil the request, the system needs to intuit the following:

The cloud provider. This can be determined based on the fact that S3 is a common storage solution in AWS, and possibly the fact that the user’s organization has stated AWS as their preferred cloud provider.
The programming language. This information can again come directly from the organizational preferences, or from the user’s prior conversations.
The information about the type (or types) that must be created - its name and schema, the package it is in, and the capabilities it supports. Some of this information comes from the built-in LLM knowledge, supported and augmented by the Registry.

Putting it all together, we can now expand the original user query into the following prompt that is going to guide the code generation:

“Generate TypeScript code for S3 Bucket, the AWS resource defined in package @pulumi/aws, type aws.s3.Bucket with its schema defined as follows: …

While we had to rely on some guesswork to come up with this prompt, fortunately this process can be iterative - if we don’t get all of it right the first time, we can try again with additional information that will help us refine the results. This is an important point we return to later in the post.

Assessing search quality using recall and precision

To assess how effective the RAG is, we need to first understand the two fundamental concepts used in the information retrieval systems: the recall and the precision. Imagine that you’re looking for apple pie recipes in one of Jamie Oliver’s cookbooks. The book has a recipe for a classic American apple pie, a Dutch apple pie and a modern take on a French apple tart. Due to the book’s narrative approach with the recipes woven into the stories and context, you’ve managed to retrieve only the first two recipes but missed the French apple tart. Having retrieved 2 ouf 3 relevant documents, you have achieved a 67% recall.

Because you were looking for the word “pie”, you also retrieved a recipe for a Shepherd’s pie, which, while delicious, does not qualify as an apple pie. Another document that came up was a fish pie - a classic British dish that does not contain apples or even a pastry crust. Since only 2 of your 4 retrieved documents can be legitimately classified as apple pies, you have achieved a 50% precision.

Now let’s formalize this a bit. Recall measures the ratio of the relevant documents retrieved to the total number of relevant documents in RAG dataset:

$$Recall = \frac{N(Retrieved \cap Relevant)}{N(Relevant)}$$

Where

$N(Retrieved \cap Relevant)$ is the number of documents that are both retrieved and relevant.
$N(Relevant)$ is the total number of relevant documents in the database.

Good recall means that many documents relevant to the query were retrieved.

Precision is the ratio of the relevant documents retrieved to the total number of retrieved documents:

$$Precision = \frac{N(Retrieved \cap Relevant)}{N(Retrieved)}$$

Where $N(Retrieved)$ is the total number of documents that were retrieved.

High precision means that many of the retrieved documents were relevant.

Naturally, we want to maximize both the recall and the precision. It’s been said that high recall is essential to ensure relevant content is available to the code generator while precision is the parameter you want to optimize for to avoid hallucinations.

Practical concerns

Precision and recall are essential in understanding the information retrieval quality, but they are quite hard to measure in practice. Unlike a cookbook, Pulumi registry contains thousands of ever changing documents, and evaluating how many of them are relevant for every user-submitted query is impractical. This makes recall evaluation for live traffic next to impossible. Things are a little easier with precision, where we’re dealing with a small number of documents, but even that metric requires a non-trivial evaluation of relevance, which needs an LLM call or a human judge.

Fortunately, other metrics that often can effectively estimate retrieval quality have been developed. We have found a metric that can predict, with some degree of accuracy, whether the generated code will successfully compile. For this metric, we compare the tokens present in the LLM-produced prompt with the number of tokens present in the generated code. (By token here we understand a compiler token - an identifier such as the name of a class, method or a field and not a traditional LLM token concept), Intuitively, if a token present in the prompt also appears in the generated program, it can be assumed that the token contributed to the generated program. Tokens in the generated program that were not part of the prompt are not necessarily wrong but they are less trusted (they can come from the LLM built-in knowledge or were guessed)

$$prompt \ coverage = \frac{N(\text{Tokens in prompt} \cap \text{Tokens in code})}{N(\text{Tokens in code})}$$

Prompt coverage is a metric we can observe in production, and it’s one of several metrics we use when updating providers to ensure we haven’t regressed the quality of the RAG.

Semantic search with vector embeddings

Semantic search is based on the conceptual similarity of the term you’re looking for with the elements in the data store. For example, searching for “dumplings” can return terms like pierogi and gyoza - words with different spelling but both representing different types of filled dough preparations.

A common way to determine the similarity between the two strings is to first turn these strings into vector embeddings - arrays of floating point values representing the semantic meaning of each string - and then calculate the cosine similarity between the two vectors, which is the cosine of the angle between the vectors. Various methods of producing vector embeddings are fascinating but we will not cover them here in depth.

For Pulumi code generation we are using the OpenAI’s Ada-002 embedding model which at this moment represents a good balance between performance and cost.

Producing vector embeddings from the user query is the standard approach in this situation. However, for Pulumi code generator we added a little twist - to increase the odds of getting more relevant information from the Registry (i.e. to increase the recall) we first make an LLM call to generate a small set of relevant search terms that will produce an array of vector embeddings.

Getting vector embeddings from user query

We then use the array of vector embeddings to retrieve the set of relevant documents from the Registry.

Full text vs semantic search

While semantic search is essential for any modern information retrieval system, we should not forget simple and effective methods for text search already exist. The “S3 Bucket” part of the user search happens to be easily searchable using traditional text search operations (such as SQL LIKE operator).

To be effective, our RAG must be able to handle queries that require semantic understanding (such as “simple storage service in AWS”) as well as the traditional text search to support situations where the user knows exactly what they are looking for. To that end, the industry has adopted an approach known as the “hybrid search”, in which the results of full-text search and semantic search are combined to provide the final result.

For each of these search terms we generate a query that combines the full text search with the semantic search. The resulting documents are then evaluated based on the following criteria:

Dense score: Vector similarity using cosine distance between query embedding and stored embeddings
Sparse score: Text search relevance according to the full-using algorithm (PostgreSQL full-text search in our case)

Both the dense and the sparse scores contribute to the final score that is used to sort the documents. Even though the final score is the average of the two in our current implementation, it would not be correct to say that they have the same weight, or the influence on the end result. This is so because the normalization of the scores must take into account their distribution in the real-world queries and calculating that is quite complex.

Boosting the influence of one score relative to the other is an area we’re actively exploring, and achieving the optimal result depends highly on what parameters we decide to optimize for - high quality code, time to result and so on.

Finally, we apply the rank-based scoring technique in which the results are penalized based on their position in the ranking. This creates separation between results that might have similar initial scores and gives preference to results that appear earlier in the ranking.

Pruning the results

Our Pulumi code generator employs a two-phase document selection strategy. The first phase casts a wide net, gathering all potentially relevant documentation to maximize recall. The second phase applies precision-focused filtering to distill this collection down to the most pertinent documents.

This filtering step serves two purposes. First, it prevents LLM hallucinations that arise from similarly-named types across different providers. Second, it optimizes performance by keeping prompts concise - a critical consideration given that larger prompts increase both latency and computational costs, even when within context window constraints.

Through empirical testing with the Pulumi Registry search, we’ve established these baseline parameters: a maximum of 10 documents per query term selected by relevance score, and a 20K token ceiling for prompts. While these parameters have yielded good results in practice, they are likely not optimal for all scenarios. We continue to iterate on these values through ongoing experimentation.

Prompt generation

We’re now ready to create the system prompt for the code generation LLM call! We’ve already discussed some of the elements needed to build the effective prompt - the original user query, the search terms and the vector embeddings that produce the set of documents that will guide the code generation process.

There is another element that goes into the prompt - a concise set of instructions produced by an LLM call based on the original user’s query. The approach when the output of one prompt is used as input for another is known as “prompt chaining”, and we used it to provide step-by-step instructions to the code generator. For our query, this set of instructions can look as follows:

Create an S3 bucket using Pulumi in TypeScript using the following steps:

Import the necessary Pulumi AWS package.

Define a new S3 bucket resource with basic configuration.

Export the bucket name as an output.

Composition of the system prompt for code generation

Finally, we use the resulting prompt to call the LLM and ask it to generate the code. We’re done!

Self-debugging

Unfortunately, this isn’t always the final step of the process. Despite our best efforts, the code produced by the LLM will not always be correct.

At this point, it’s worth pondering what “correct code” means. The generated program might have the following problems:

It might be syntactically or semantically incorrect, i.e. it might not compile or fail to typecheck.
It might fail at runtime - for example by referring to a non-existing resource, a region and so on.
It might run “successfully” but not do what the user intended - either because the user did not express themselves clearly or because the request was misunderstood.
Finally, the code might run and do exactly what the user wants, but lead to an undesired outcome, for example loss of an asset, or a security vulnerability.

Solutions to many of these problems go well beyond the domain of code generation.

However, the first of these problems can be addressed by the approach known as “self-debugging”.

We observed that many generated TypeScript programs that fail to typecheck contain only a few errors, and asking the LLM to fix these errors often produces a valid program. While this approach is still experimental, we see promising results where getting to a correct program requires only 1-2 iterations of self-debugging. The biggest challenge with this approach is doing it in real-time. The user is staring at the screen waiting for an answer, so latency is a major concern.

Monitoring these typechecking errors in production can also provide valuable insight into the quality of the data used by the RAG and even suggest specific solutions. For example, failure to typecheck a member-access expression is a likely indicator of a missing type schema (a recall problem) or a “wrong” schema brought in by an irrelevant document (a precision problem).

Self-debugging can also be extended to include the pulumi preview command, which is a “dry run” operation before the actual deployment and can detect many real or potential problems such as destructive actions, incorrect configurations that cannot be detected at compile time, dependency conflicts, and policy violations.

From kitchen to table: testing what works

The landscape of LLM-based code generation is moving fast, and we need to keep learning and adapting as we go. But with all this rapid technological change, it’s crucial to ground our decisions in real numbers. We need to make sure each new advancement actually makes things better, both in our test environments and out in the real world.

The probabilistic nature of LLM-based code generation means we can’t rely solely on pre-production testing. Instead, we adopt multiple layers of quality control working together. Here’s what we’ve learned works best:

Building confidence through testing

Our testing strategy combines several approaches:

Systematic experimentation: We run “what-if” scenarios against various test datasets to understand how the system behaves under different conditions.
Local evaluation pipeline: While local testing has its limitations, it helps catch obvious issues early in development. We evaluate the performance of the code generator and run the code through the typechecker.
Deterministic generation: We set the LLM temperature to 0 to ensure consistent outputs. For code generation, consistency and repeatability matters more than creative variations.

Monitoring quality in production

We track several key metrics to ensure the system performs well:

RAG effectiveness: We measure prompt coverage to evaluate how well our retrieval system performs.
Code quality indicators: We track the percentage of generated code that successfully typechecks for all supported languages along with how this metric responds to system changes.
User feedback: Every “thumbs down” report gets analyzed to identify patterns and potential improvements. This direct user feedback has been invaluable in refining our system.

Wrapping up

Building an effective AI-powered code generator requires carefully balancing multiple concerns: the raw capabilities of LLMs with retrieved contextual knowledge, semantic search with traditional text matching, and thorough testing with real-world scenarios.

Our experience has taught us that success lies in treating code generation like a delicate soufflé - it requires precise measurements, careful monitoring, and constant refinement of technique. The key ingredients are:

A robust RAG system with well-tuned recall and precision.
End-to-end testing and monitoring across development and production.
Self-debugging capabilities to handle common issues like type checking errors.

As we continue to evolve Pulumi’s code generation capabilities, we’re excited about expanding our self-debugging features and further refining our RAG implementation.

We invite you to try these capabilities in your own projects and share your experiences. Your feedback helps us continue improving and advancing the state of AI-assisted infrastructure development.

AI Engineering Lessons from Building Pulumi Copilot

Artur Laksberg — Thu, 12 Dec 2024 13:56:13 -0500

Note: This post discusses Pulumi Copilot, which Pulumi Neo has replaced. Learn about Neo →

Building AI-powered developer tools comes with unique challenges, and now that we’ve launched our REST API, we want to share some lessons we’ve learned building Pulumi Copilot, an AI assistant for cloud infrastructure.

One of the big challenges was determining what ‘working’ really meant. So when a message landed in our feedback channel after months of rigorous testing - ‘Your tool doesn’t know anything!’ - it caused some mild panic. We’d just made some changes, so we braced for the worst. But our evals were still looking strong, so what was going on?

The user was attempting to force-delete a stack that still contained resources. But when we dug deeper, we found something fascinating: Copilot had confidently suggested a ‘–force’ flag, which would have been a logical solution… except this flag doesn’t exist in Pulumi. Our AI was hallucinating exactly what the user wanted. But this wasn’t just a bug - it was the first of many insights that would reshape how we approach AI-powered tools.

To understand how we got here - and why this “error” actually taught us something valuable about our product - let’s start with the core challenge we faced: balancing traditional software engineering with this new world of prompt engineering.

Engineering for Reality: Prompt Engineering vs Software Engineering

When building LLM-powered applications, it’s tempting to throw every task at the model. Modern LLMs can generate code, format text, and create clickable links. But this approach carries hidden costs.

Working on Copilot taught us a key lesson: let the LLM do what it does best and use good old imperative code for everything else.

Take a seemingly simple feature: listing a user’s Pulumi stacks with clickable links based on data from a backend API. Our first implementation used a complex prompt instructing the LLM to construct URLs in the format app.pulumi.com/org/project/stack. The prompt explained the format, provided examples, and asked the LLM to generate these links from JSON data it had.

It worked - almost all of the time. But we were seeing occasional malformed URLs and more importantly, this was burning input tokens (and money) on a complicated prompt that made AI construct strings that could be deterministically generated.

The solution was straightforward: generate the full links in the backend service and include them directly in the context. The LLM then needs no instructions on how to create them. Simple stuff, but it gave faster responses and perfect URLs at a lower cost.

When you find yourself writing elaborate prompts to handle structured data transformations, stop and ask: Could traditional code do this better? Could this be decomposed so that the LLM does less?

To validate this approach, we tested Copilot ourselves to see what worked.

Copilot in Action: Real-World Dogfooding

The internal testing phase taught us invaluable lessons about how people would actually use the tool. We watched our team try Copilot in their daily work, and three common use cases emerged:

Debugging Deployments: LLMs clearly excel at summarization. One of the first questions our internal users asked was, ‘Why did my latest infrastructure deployment fail?’ Using Copilot to extract a clear natural language explanation requests like these has been a clear win.

Understanding Complex Infrastructure: Copilot helped our engineers gain insights into Pulumi’s own infrastructure. Asking, ‘How many resources are in production?’ ‘What expensive compute is running’ or ‘What version are the EKS clusters in EU?" shows the value of allowing users to express infrastructure questions in natural language.

Generating Code: One of the first queries logged was, ‘I want a static website on AWS behind a CloudFront CDN.’ Another came from a Solutions Engineer, tasked with demonstrating Pulumi’s CrossGuard policy engine to a prospect, asking Copilot to generate policy code.

These early experiences showed the value of Copilot. But they also revealed the need for a systematic approach to handling diverse user queries. This led to the development of what we call skills.

Skillful Slicing: Modular Mastery

As Copilot grew, we broke it into smaller pieces we call skills. Each skill does one specific job. The Insights skill handles queries about resource usage and configuration (“How many S3 buckets do I have?”), the Cloud Skill interacts with the Pulumi Service API to manage infrastructure (“Show me my stacks.”), the Code Skill generates Pulumi code snippets (“Write a Typescript program…”), and the Docs Skill retrieves information from Pulumi documentation (“How do I use update plans?”).

When you ask Copilot something, it figures out what you need and picks the right skill for the job – like a manager deciding which expert to send your question to. This function-calling approach, orchestrated by a component we call the “outer loop,” allows Copilot to access and process information beyond its internal knowledge base.

Fortunately, Pulumi Cloud already exposes a rich API - in fact, this is what powers the Pulumi Console and the Pulumi CLI - so all we had to do is to build a skill that maps the user query to the appropriate Pulumi Cloud REST API. A question like “Show me my stacks” translates into the List Stacks API call. A question like ‘Show me my untagged EC2 instances’ is a bit more complex but it breaks down into clear components - resource type (EC2), filter condition (untagged) - that route to the Insights skill. This mapping helped us handle the many ways users phrase the same technical request.

Refining this routing system revealed another opportunity: streamlining the Debug button workflow.

Debug Dispatch

Originally, when a user clicked ‘Debug with Copilot’, the system would send a text query like “Analyze this update and explain any errors.” Copilot would then:

Determines the user wants to analyze an update
Identifies which API to call
Calls the API
Summarize the result

Having the LLM figure out what the user wants is the right approach in general, but in this case, we already know the user’s intent - they clicked a debug button. So, we can directly call the analysis API to get the results and use the LLM solely for what it does best: summarizing technical output into clear, actionable explanations.

This is another small win for the “Software Engineering over Prompt Engineering” approach. Traditional code handles the predictable parts, while AI focuses on the human-facing explanations.

But while minimizing the LLM workload helped with efficiency, we soon faced an even trickier challenge: the deceptive polish of AI-generated outputs.

The Illusion of Correctness

Large language models excel at generating well-structured, grammatically correct output. They make neat tables, tell good stories, and generally sound confident. That’s what makes them dangerous because this polished presentation can mask underlying flaws in the information itself, creating a false sense of confidence for users.

One of our early testers, Pablo, a data engineer at Pulumi, encountered this firsthand. He posed a query to Pulumi Copilot, asking for a summary of resources within a specific project. The response he received was impeccably formatted, neatly categorizing resources by type and providing counts for each. It looked right, and for us humans sometimes looking right carries a lot of weight.

However, a closer inspection revealed the numbers were way off. Copilot had asked for the wrong data and then summarized it beautifully - but incorrectly. This highlighted our next challenge: how do you systematically test a system that can be confidently wrong while sounding completely right?

Testing the Untestable: Validating LLM Outputs

When testing traditional code, we expect consistent, predictable outputs. With LLMs, even correct answers can vary. Here’s how we tackle this challenge.

Our first approach was simple: keyword checks. For example, when testing the update analysis feature, we checked if the LLM’s response included the word “security” and described the error. This worked for straightforward cases but broke down quickly. Take a question like, “How many Lambdas am I running?” The LLM might give the right numbers but skip the word “running,” failing the test even though the answer was correct.

These early failures revealed the limitations of keyword-based validation and underscored the need for a more nuanced approach. Inspired by platforms like LangSmith and Promptfoo, we began leveraging LLMs themselves as evaluators. For deterministic tasks, simple keyword checks suffice, but for more complex scenarios—like assessing whether a response answers a specific question—we rely on an “LLM Judge.” This approach balances efficiency and flexibility, reserving LLM evaluation for cases where it truly adds value. The test suite now integrates both methods, running against every code change to validate response content, accuracy, and format.

The eval suite keeps getting more robust, which means that when new AI models drop, we can quickly catch any weirdness before it hits production. The generative AI space moves crazy fast and the code changes a lot, but the evals are a safety net - catching hallucinations, maintaining quality, and making sure we don’t ship anything that’ll annoy our users.

So, while hallucinations are now much rarer, what about that one with the --force flag? Yes, it’s “just another bug,” but it taught us something fascinating about these AI errors.

You might also like:

LLMs think like humans (sort of)

The --force hallucination wasn’t totally wrong - it was revealing what users intuitively expect from the CLI, and the LLM accidentally showed us what was missing. Force deletion is a common pattern across developer tools, and the LLM, trained on vast amounts of documentation and code, simply reflects these established conventions.

This has fundamentally changed how we view hallucinations. While the team works constantly to minimize them – and our eval work means they happen way less frequently – some of them are clearly product signals. The LLM, in this light, becomes an unexpected source of user research, drawing on its training across thousands of developer tools and experiences.

This insight is one of the key lessons of building Copilot:

Minimize LLM Usage: Let traditional code handle deterministic tasks, reserve LLMs for natural language work
Decompose into Skills: Break complex tasks into modular units that combine LLM and traditional code appropriately
Test Rigorously: Use multiple validation approaches, including LLMs testing LLMs
Learn from Hallucinations: Sometimes incorrect outputs reveal user expectations
Learn from Users Continuously: User interactions improve our AI systems - from training better skills to catching hallucinations and revealing product opportunities.

These lessons helped shape our latest release: the Pulumi Copilot REST API, now available in preview. You can integrate these same capabilities and skills into your own tools and workflows. Whether you’re building CLI extensions, chat integrations, or automated deployment checks, the API provides a contextual understanding of Copilot. Try it out.

We can’t wait to see what you build!

Next-level IaC: Bridging the Declarative Gap

Troy Howard — Fri, 07 Jun 2024 00:00:00 +0000

Pulumi stands out in the world of infrastructure-as-code (IaC) for its flexibility and ease of use. This is due to its unique approach of using general purpose programming languages to describe system configuration. It accomplishes this through a small but brilliant implementation detail that bridges the worlds of declarative and standard programming using asynchronous semantics.

In Pulumi, you describe your infrastructure in code – real code, not a DSL or YAML – using your preferred general purpose programming language. You don’t have to become a specialist in a niche proprietary declarative language, like Terraform’s HCL. Rather, you can reuse your existing programming skills, writing in standard imperative, object-oriented, and even functional language styles, while still gaining all of the benefits of the declarative style that other tools emphasize.

Pulumi provides a unique mix of a declarative model embedded and implemented inside of a standard programming language, allowing all the flexibility of custom imperative code, while still enabling Pulumi’s deployment engine to infer opportunities for parallel asynchronous execution and to converge a partially-realized system.

The secret sauce that lets Pulumi achieve this is the apply function, and the way it handles inputs and outputs via asynchronous programming constructs, similar to futures or promises. We’ll go deeper on that in this article, but first let’s talk a little bit about the flow of operations in Pulumi.

Pulumi’s architecture and flow of operations

Pulumi is a system that is made up of a number of components. At a high level, those are:

Pulumi Programs: The code you write to describe the infrastructure you need.
Deployment Engine: A process that executes requests to create/modify infrastructure.
Resource Providers: Plugins that interface with cloud services like AWS, Azure, and others.

These components interact together to manage your infrastructure across multiple cloud providers using a consistent programming and execution model.

Figure: Pulumi’s Program, Deployment Engine, and Resource Provider architecture

Let’s get into an example of how information flows back and forth between these components. Suppose you want to write a Pulumi program to create an AWS S3 Bucket to host a static website. When that bucket is created, it will be given a unique identifier that you need to use in other parts of your infrastructure. Every time you create the bucket the identifier will change, and it’s generated by the AWS API, so we can’t know it ahead of time. We need to handle this as a variable, not a fixed configuration value.

To do that in a Pulumi program, you can create a bucket and then refer to its id property later, like this:

import * as pulumi from '@pulumi/pulumi';
import * as aws from '@pulumi/aws';

// Create an S3 bucket
const bucket = new aws.s3.Bucket("bucket");

// Export the name of the bucket
export const bucketName = bucket.id;

import pulumi
import pulumi_aws as aws

# Create an S3 bucket
bucket = aws.s3.Bucket('bucket')

# Export the name of the bucket
pulumi.export('bucket_name', bucket.id)

When this program runs, the call to create the bucket object will issue a request to the deployment engine. The deployment engine will load the resource provider for AWS, and ask it to create the S3 bucket. The AWS resource provider then issues the request to the AWS API, which will create the bucket and then return the id of the newly created resource. The provider will return that to the deployment engine, which will then set the realized value on your Pulumi program’s bucket.id property and then move on to creating anything that depends on that property.

This all seems pretty straight-forward, except that Pulumi programs don’t wait and imperatively do each operation, one at a time. Instead the deployment engine will run them in parallel, only waiting in cases where the output of one operation is needed to run the next operation. This can create complex structures with some operations needing to wait for inputs from other operations, and some that don’t, all finishing at different times, depending on how long the cloud service takes to process that request.

Inputs and Outputs as asynchronous futures

Infrastructure automation tasks can be long-running and can result in intermittent errors and may need to be retried before moving on to the next step. A robust IaC solution will need to run concurrently, be able to restart, retry, and resume itself, and understand that a set of tasks could happen in any order, not necessarily in the order you defined them in code.

Back in our Pulumi program code, that means that some of the time the bucket object from our earlier example has its id property set, and sometimes it doesn’t. We wouldn’t want to run any operation that needed that variable before it was set, but we also wouldn’t want to wait to start operations that don’t need that value.

To address this, Pulumi models the input and output values as futures. This allows you to describe a complex set of needs using a familiar value-like syntax. Instead of using a string result directly, which would only exist some of the time, you instead work with an Output<string>, that Pulumi understands as a reference to a value that currently doesn’t exist, but will exist when the operation completes. Similarly, Input<string> describes a dependency on an input string that will at some point be provided as the result of another operation.

This notion of future completion is essential to understanding why apply blocks are necessary within Pulumi programs. The apply function is called by the deployment engine only when the operation has completed and the fully realized value is available. By using Input<T> and Output<T> to build the connections between different parts of your infrastructure, the code is leveraging asynchronous value semantics not just to wrap and provide access to a future value, but also to describe a relationship between two components. This allows Pulumi programs to be very natural to author and as readable as traditional code, while also enabling Pulumi’s deployment engine to execute the entirety of your graph of operations in parallel, only waiting when necessary. This maximizes the efficiency of interacting with potentially slow and unreliable external cloud services.

Realizing values using the `apply` function

An important distinction though, is that Input<T>/Output<T> values are just declarative placeholders for the values that will eventually be present. At a certain point you will need to access the values directly, for example in the context of a transformation. The output of one operation might produce a string, which you need to transform into another data structure before passing to the next operation. Pulumi really shines in this context!

Using the apply function we can pass an Input<T> or Output<T> to a transformation function which will accept the realized value as its input (and only run when that value is available). Within that function you can handle the string as a normal string, then return it back as an Output<string>. This allows you to use the full power of your programming language to perform a custom operation inline with the rest of your infrastructure automation.

Here’s a simple example of using apply to transform an output to something new using your programming language’s libraries. In this case, Base64 encoding the id of our bucket object.

Example: Using `apply` to Base64 encode an Output

import * as pulumi from '@pulumi/pulumi';
import * as aws from '@pulumi/aws';

// Create an S3 bucket
const bucket = new aws.s3.Bucket("bucket");

// Export the name of the bucket
export const bucketName = bucket.id;

// Export the name of the bucket, in Base64 encoding
export const bucketNameBase64 = bucket.id.apply(id=> {
 // Transform the bucket name into a Base64 encoded string
 Buffer.from(id).toString('base64')
})

import base64
import pulumi
import pulumi_aws as aws

# Create an S3 bucket
bucket = aws.s3.Bucket('bucket')

# Export the name of the bucket
pulumi.export('bucket_name', bucket.id)

# A Base64 encoder function to use in the apply function's lambda
def base64_encode(data):
 data_bytes = data.encode("ascii")
 base64_bytes = base64.b64encode(data_bytes)
 base64_string = base64_bytes.decode("ascii")

# Export the name of the bucket, in Base64 encoding
pulumi.export('bucket_name', bucket.id.apply(lambda data: base64_encode))

Our example shows how to do Base64 encoding, but this technique can be used for any kind of transformation function. These can be as elaborate as your use case requires, and leverage the full power of your programming language and its entire ecosystem. Doing custom transformations like this in other popular tools is often not possible. This leaves you stuck using external coordination (e.g. gluing things together with Bash scripts) which breaks their declarative-only model (because you cannot describe everything you need to do in it) and interrupts these tools’ ability to fully understand your infrastructure’s dependency graph.

While Pulumi’s asynchronous semantics may seem somewhat complex for basic use cases, they unlock abilities in more complex use cases, which is when you really need your IaC tooling to show up for you. Because real life is never as simple as demo code.

Next steps

If you haven’t already, install Pulumi today, and follow our self-directed Getting Started guides to learn more about making the most of Pulumi’s next-level infrastructure management features at your organization.

To learn more, you can watch the following video which provides a high level overview of how Pulumi works:

*{padding:0;margin:0;overflow:hidden}html,body{height:100%}img{position:absolute;width:100%;top:0;bottom:0;margin:auto}">

Pulumi Cloud

The Pulumi Cloud is a fully managed service that helps you adopt Pulumi’s open source SDK with ease. It provides built-in state and secrets management, integrates with source control and CI/CD, and offers a web console and API that make it easier to visualize and manage infrastructure. It is free for individual use, with features available for teams.

Create an Account

Details of the Pulumi Outage on October 6, 2023

Casey Huang — Wed, 18 Oct 2023 11:52:58 -0700

At Pulumi, we understand that Pulumi Cloud plays an important role in how our customers address their infrastructure management challenges. As a result, we strive for the highest levels of availability and performance in Pulumi Cloud. Unfortunately, on Friday, October 6, 2023, Pulumi Cloud suffered a 24 minute outage during which we failed to process 74.7% of received requests. In this post, we’d like to share our findings on the root cause of this outage, and the steps we are taking to ensure this sort of outage doesn’t happen again.

On October 6th at approximately 17:15 UTC we shipped a database migration modifying foreign keys on a table to our production environment, clearing it for release after testing in several non-production environments and a few rounds of peer review. However, the pre-production testing was not an adequate substitute to test the behavior of the migration when running on our production dataset under full traffic load.

Adding foreign key constraints to a table can be done “in place” with an asterisk. Testing and review missed that we weren’t abiding by that asterisk. The resulting table copy operation caused by the bad migration held a lock for a significant amount of time, and caused a query pileup that starved our database of all available connections.

This is the first time in six years that Pulumi has seen an outage of this scale. We are careful not to make changes to high traffic tables on the core API path responsible for handling updates and storing state. When we’ve needed to update these tables in the past, we were heedful in standing up new tables; duplicating writes; and cutting over to new tables without downtime.

During the review process, we had categorized the migration as low risk, as the affected table is low traffic relative to our other workloads (<0.2% of the traffic of our busiest table). We learned through this incident that we’ve arrived at a scale where a block on even a relatively low-traffic table is enough to cause writes into this table to block the entire connection pool and starve the rest of the database and API workloads.

Within three minutes of the change hitting production, our alarms alerted us to our service degradation and within five minutes, multiple team members were huddled together on an incident call. Our responders parallelized in gaining access to our production database and working to terminate the offending blocked writes that were starving other API workloads. Others assessed impact and worked on customer comms. Unfortunately, high database load also affected the responsiveness of our operational tooling. While we were preparing to execute mitigation actions, the migration completed successfully and both the database and API availability immediately and fully recovered.

But failing three quarters of our traffic for a twenty minute outage is unacceptable. A migration that caused this level of locking should not have been allowed to run. Following an internal team postmortem, we have identified improvements we will be applying to our Software Development Lifecycle to automatically and accurately assess the impact of proposed migrations, as well as to block dangerous migrations from running in production. These include:

New guidance, enforced by SQL linting, that migrations must always explicitly specify safe ALGORITHM and LOCK constraints.
Improving the fidelity of our pre-production environments to better simulate production workloads.
New tooling to quickly create ephemeral database performance testing environments at the click of a button. (pulumi makes this easy!)

We are also making changes to split parts of our service into different failure domains. This ensures that even if a table on a non-critical path gets blocked, we reserve connections for core workloads. We want the Pulumi Cloud to remain available to continue to save stack state from customer updates even if other parts of the service are failing.

At Pulumi, we value honesty and transparency. It’s easy to repeat the past; harder to learn from it and move past it. If Pulumi is to be the best developer tools platform on the market, that begins with how we build and operate our software. We deeply apologize to our users for the disruption in service, and we commit ourselves further in driving towards excellence - in both the features we provide and the operational standards to which we hold ourselves.

Thank you for your understanding, and for sticking with us on the road to better.

Converting Bicep code to Pulumi

Zaid Ajaj — Fri, 29 Sep 2023 00:00:00 +0000

Bicep is a DSL developed by Microsoft to simplify the authoring of ARM templates and deploy resources to Azure. Today I will be sharing with you a new Pulumi converter plugin that I have been working on that converts Bicep code to any of the supported Pulumi languages.

The Bicep language is a domain-specific language for Infrastructure-As-Code built by Microsoft that uses declarative syntax to deploy Azure resources. It was made to address the shortcomings of authoring ARM templates that are written in JSON. Compared to ARM templates, Bicep is much more concise and readable. However, here at Pulumi we believe that using general-purpose programming languages that are familiar to developers is the best way to author Infrastructure-As-Code. Besides being able to use a general-purpose language, you and your team can benefit from using Pulumi Cloud as you scale to larger projects and teams with features such as Pulumi Deployments, Organization Access Tokens, Review Stacks and more.

That is why I have been working on a new Pulumi converter plugin that converts Bicep code to any of the supported Pulumi languages using the Pulumi CLI via pulumi convert.

Installing and using the plugin

First, install the converter plugin using the Pulumi CLI:

pulumi plugin install converter bicep

This command will install the plugin from the latest GitHub release of the repository.

Once installed, you can use it to convert Bicep code to Pulumi. For example, let’s say we have the following Bicep code in a file called example.bicep which deploys a storage account:

resource storage 'Microsoft.Storage/storageAccounts@2022-09-01' = {
 name: 'storageaccount'
 location: resourceGroup().location
 kind: 'StorageV2'
 sku: {
 name: 'Standard_LRS'
 }
}

Navigate to the directory where the file is located and run the following command. For example to generate TypeScript code from the Bicep code above, run:

pulumi convert --from bicep --language typescript --out output -- --entry example.bicep

This command will use the Bicep converter plugin we just installed to convert the code in example.bicep to TypeScript and place the generated code in the output directory.

You can change typescript to any of the supported languages such as csharp, python, go, java or yaml and you are good to go.

If you don’t specify the --entry flag, the plugin will look for the first file with the .bicep extension in the current directory.

Let us look at the generated code from the example above:

import * as pulumi from "@pulumi/pulumi";
import * as azure_native from "@pulumi/azure-native";

const config = new pulumi.Config();
// The name of the resource group to operate on
const resourceGroupName = config.require("resourceGroupName");
const currentResourceGroup = azure_native.resources.getResourceGroupOutput({
 resourceGroupName: resourceGroupName,
});
const storage = new azure_native.storage.StorageAccount("storageaccount", {
 accountName: "storageaccount",
 kind: "StorageV2",
 location: currentResourceGroup.apply(currentResourceGroup => currentResourceGroup.location),
 resourceGroupName: currentResourceGroup.apply(currentResourceGroup => currentResourceGroup.name),
 sku: {
 name: "Standard_LRS",
 },
});

import pulumi
import pulumi_azure_native as azure_native

config = pulumi.Config()
# The name of the resource group to operate on
resource_group_name = config.require("resourceGroupName")
current_resource_group = azure_native.resources.get_resource_group_output(resource_group_name=resource_group_name)
storage = azure_native.storage.StorageAccount("storageaccount",
 account_name="storageaccount",
 kind="StorageV2",
 location=current_resource_group.location,
 resource_group_name=current_resource_group.name,
 sku=azure_native.storage.SkuArgs(
 name="Standard_LRS",
 ))

package main

import (
 "github.com/pulumi/pulumi-azure-native-sdk/v2/resources"
 "github.com/pulumi/pulumi-azure-native-sdk/v2/storage"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi/config"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 cfg := config.New(ctx, "")
 // The name of the resource group to operate on
 resourceGroupName := cfg.Require("resourceGroupName")
 currentResourceGroup, err := resources.LookupResourceGroup(ctx, &resources.LookupResourceGroupArgs{
 ResourceGroupName: resourceGroupName,
 }, nil)
 if err != nil {
 return err
 }
 _, err = storage.NewStorageAccount(ctx, "storageaccount", &storage.StorageAccountArgs{
 AccountName: pulumi.String("storageaccount"),
 Kind: pulumi.String("StorageV2"),
 Location: *pulumi.String(currentResourceGroup.Location),
 ResourceGroupName: *pulumi.String(currentResourceGroup.Name),
 Sku: &storage.SkuArgs{
 Name: pulumi.String("Standard_LRS"),
 },
 })
 if err != nil {
 return err
 }
 return nil
 })
}

using System.Collections.Generic;
using System.Linq;
using Pulumi;
using AzureNative = Pulumi.AzureNative;

return await Deployment.RunAsync(() =>
{
 var config = new Config();
 // The name of the resource group to operate on
 var resourceGroupName = config.Require("resourceGroupName");
 var currentResourceGroup = AzureNative.Resources.GetResourceGroup.Invoke(new()
 {
 ResourceGroupName = resourceGroupName,
 });

 var storage = new AzureNative.Storage.StorageAccount("storageaccount", new()
 {
 AccountName = "storageaccount",
 Kind = "StorageV2",
 Location = currentResourceGroup.Apply(getResourceGroupResult => getResourceGroupResult.Location),
 ResourceGroupName = currentResourceGroup.Apply(getResourceGroupResult => getResourceGroupResult.Name),
 Sku = new AzureNative.Storage.Inputs.SkuArgs
 {
 Name = "Standard_LRS",
 },
 });

});

package generated_program;

import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.azurenative.resources.ResourcesFunctions;
import com.pulumi.azurenative.resources.inputs.GetResourceGroupArgs;
import com.pulumi.azurenative.storage.StorageAccount;
import com.pulumi.azurenative.storage.StorageAccountArgs;
import com.pulumi.azurenative.storage.inputs.SkuArgs;
import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;

public class App {
 public static void main(String[] args) {
 Pulumi.run(App::stack);
 }

 public static void stack(Context ctx) {
 final var config = ctx.config();
 final var resourceGroupName = config.get("resourceGroupName");
 final var currentResourceGroup = ResourcesFunctions.getResourceGroup(GetResourceGroupArgs.builder()
 .resourceGroupName(resourceGroupName)
 .build());

 var storage = new StorageAccount("storage", StorageAccountArgs.builder()
 .accountName("storageaccount")
 .kind("StorageV2")
 .location(currentResourceGroup.applyValue(getResourceGroupResult -> getResourceGroupResult.location()))
 .resourceGroupName(currentResourceGroup.applyValue(getResourceGroupResult -> getResourceGroupResult.name()))
 .sku(SkuArgs.builder()
 .name("Standard_LRS")
 .build())
 .build());

 }
}

configuration:
 resourceGroupName:
 type: string
resources:
 storageaccount:
 type: azure-native:storage:StorageAccount
 properties:
 accountName: storageaccount
 kind: StorageV2
 location: ${currentResourceGroup.location}
 resourceGroupName: ${currentResourceGroup.name}
 sku:
 name: Standard_LRS
variables:
 currentResourceGroup:
 fn::invoke:
 Function: azure-native:resources:getResourceGroup
 Arguments:
 resourceGroupName: ${resourceGroupName}

Notice how the in the original Bicep code, we reference the location of the implicit resource group via resourceGroup().location. When the converter generates code for the target language, it replaces this implicit resource group with an explicit one by parameterizing the program by a resource group name and using that resource group when deploying the resources.

Early days for the converter

Currently the converter supports most of the features of Bicep such as parameters, variables, modules, resources, outputs and a number of built-in functions. That said, the converter plugin is still in its early days and it’s not yet tested against many real-world Bicep programs. I am actively working on improving it and making it support most of the Bicep language features. The source code is available on this GitHub repository. If you find any issues or have any feedback, please open an issue on the repository.

How does it work?

The converter plugin is a native binary built using .NET in F# and new Pulumi Converter SDK. Using the core Pulumi nuget package, we shipped an experimental API that allows you to easily build converter plugins that can extend the functionality of pulumi convert.

Building a language converter plugin involves taking the source language, in this case Bicep, and converting it to our internal Pulumi Configuration Language, or PCL for short. The converter implements a function of the following shape:

let convertProgram (request: ConvertProgramRequest): ConvertProgramResponse =
 let sourceDirectory = request.SourceDirectory
 let targetDirectory = request.TargetDirectory
 let args = request.Args
 // conversion logic here

In our case, we read the Bicep code from the source directory, convert it to PCL and write out the PCL code in the target directory. Then, Pulumi uses the built-in program generation facilities to take care of the rest of the work of generating the target language code from the PCL code so that we don’t have to worry about language-specific details. The converter only needs to know how to generate PCL.

To build the actual transformation from Bicep to PCL, I made use the of the Azure.Bicep.Core package available for .NET which allowed me to parse the Bicep code and generate a typed Abstract Syntax Tree (AST) from it. Building an AST from the source language allows us to easily traverse the tree, analyze it and symbolically rewrite pieces of it. Working at the AST level also makes it easy to test source code transformation using structure rather than text. Once we have obtained the Bicep AST, we transform it into a Pulumi AST that represents a PCL program. Finally, we print out the Pulumi AST to a string and write it to the target directory.

If you are curious about the specification of the AST that represents Bicep programs, head over to this file from the source code. You will find types such as BicepProgram and BicepSyntax that model almost every aspect of Bicep code. As for the AST of PCL programs, you can find the type definitions here.

The F# language is great for almost everything, but is especially amazing for writing language converters because of its powerful pattern-matching capabilities and the ability to write code in a functional style. Many of us working at Pulumi are big fans of F#!

Here is a snippet of the transformation pipeline written in F# that takes the input Bicep program and converts it to the final PCL text

bicepProgram
|> BicepProgram.dropResourceUnknowns
|> BicepProgram.reduceScopeParameter
|> BicepProgram.parameterizeByTenantAndSubscriptionId
|> BicepProgram.addResourceGroupNameParameterToModules
|> BicepProgram.parameterizeByResourceGroup
|> Transform.bicepProgramToPulumi
|> Transform.modifyComponentPaths rewriteComponentPath
|> Printer.printProgram

You can learn more on the implementation of these functions from the source code on GitHub. I’ve written it in a way that is easy to follow, understand and contribute to. Most of these functions are fully unit-tested and I am working on adding more tests to cover more of the Bicep language features.

Bonus converter: ARM to Pulumi

Those who are familiar with the Bicep CLI, know that it has a built-in decompiler that converts ARM templates to Bicep. I thought it would be fun to build another converter that takes ARM templates to Pulumi but without writing any conversion logic for it. Instead, I would decompile the ARM template to Bicep, then use the logic from the Bicep converter to generate the final PCL code. Fortunately, the decompiler code is embedded in the nuget package Azure.Bicep.Core so we don’t have to rely on the existence of the Bicep CLI in order to make this converter work.

From all of this, I created another converter pulumi-converter-arm that works exactly like the Bicep converter but takes ARM templates as input. Head over to the repository to learn more about how to install it and how to use it.

Depending on how well this ARM converter does its job, we might consider deprecating arm2pulumi in favor of it. However, we first need to test it against more templates and iron out the rough edges. It would be the ideal situation because there are virtually zero maintenance costs, any improvements to the Bicep converter would automatically get picked up here. Also, improvements to the decompiler from the Azure.Bicep.Core nuget package would benefit the converter as well.

Building your own Pulumi language converter

Converter plugins are a great way to extend the Pulumi CLI with pulumi convert to support new languages. When we shipped support for Converting full Terraform Programs To Pulumi, we extended the pulumi convert command to allow installing and using converter plugins that are shipped independently from the Pulumi CLI.

In essence, those plugins are executable binaries that can be written in any language, which serves up a gRPC server that implements the Converter contract:

// Converter is a service for converting between other ecosystems and Pulumi.
// This is currently unstable and experimental.
service Converter {
 // ConvertState converts state from the target ecosystem into a form that can be imported into Pulumi.
 rpc ConvertState(ConvertStateRequest) returns (ConvertStateResponse) {}

 // ConvertProgram converts a program from the target ecosystem into a form that can be used with Pulumi.
 rpc ConvertProgram(ConvertProgramRequest) returns (ConvertProgramResponse) {}
}

This Converter interface is considered experimental and is subject to change in the future.

Now, learning how to setup gRPC servers and implement the Converter contract is not an easy task unless you are familiar with it. For .NET, we shipped the experimental Pulumi converter SDK to the main Pulumi nuget package that makes it extremely easy to build converter plugins.

A bare-bones converter plugin that does nothing looks like this in F#:

open Pulumi.Experimental.Converter
open Pulumi.Codegen

let convertProgram (request: ConvertProgramRequest): ConvertProgramResponse =
 // TODO: Implement the conversion logic here
 ConvertProgramResponse.Empty

convertProgram
|> Converter.CreateSimple
|> Converter.Serve
|> Async.AwaitTask
|> Async.RunSynchronously

This will spin up a gRPC server that implements the Converter contract and will serve it on a random port assigned during startup. The Pulumi converter SDK automatically implements gRPC reflection which allows tools such as Postman to discover the available gRPC services and methods and send example requests to them. See Postman with gRPC.

Publishing converter plugins

The easiest way to make your converter plugin available to others is to publish it on GitHub releases. The Pulumi CLI expects a naming convention pulumi-converter-<name> and will automatically download the plugin from the latest GitHub release of the repository when given the --server flag during pulumi plugin install converter <name> --server <url>.

Converter plugins written in .NET are simple console applications that are compiled to native binaries for each target platform. To see how this is done, refer to this createAndPublishArtifacts function written for the Bicep converter that does the heavy lifting of building the native binaries for each platform, putting them in an archive then publishing them to GitHub releases.

Note: when publishing native binaries for macOS, the Github Action executing the publishing process must be using a macOS runner so that the generated binary from dotnet publish gets signed with a developer certificate. This is required for the binary to be executable on macOS.

Final thoughts

I hope you enjoyed this article and learned something new about the Pulumi converter plugins and Pulumi internals. I am very excited about the Bicep converter plugin and I am looking forward to hearing your feedback on it from the community. It was a lot of fun writing this in F# and made writing complex pieces of code very trivial.

If you have any questions or feedback, don’t hesitate to reach out!

Benchmarking Python Performance

Justin Van Patten — Thu, 28 Sep 2023 00:00:00 +0000

This is the second post in a series about performance optimizations we’ve made to the Pulumi CLI and SDKs. In this post, we’ll go deep on a performance improvement we made for Pulumi Python programs. You can read more about Amazing Performance in the first post in the series.

Late last year, we took a hard look at the performance of Python programs when we realized they weren’t performing up to our expectations. We uncovered a major bug limiting Python performance, and we ran a number of rigorous experiments to evaluate just how performant Pulumi Python programs are after the bug had been repaired. The results indicate Pulumi Python programs are significantly faster than they were, and now Pulumi Python has reached performance parity with Pulumi Node.js!

The Bug

When you execute a Pulumi program, Pulumi internally builds a dependency graph between the resources in your program. In every Pulumi program, some resources have all their input arguments available at the time of their construction. In contrast, other resources may depend on Outputs from other resources.

For example, consider a sample program where we create two AWS S3 buckets, where one bucket is used to store logs for the other bucket:

import pulumi
import pulumi_aws as aws

log_bucket = aws.s3.Bucket("logBucket", acl="log-delivery-write")

bucket = aws.s3.Bucket("bucket",
 acl="private",
 loggings=[aws.s3.BucketLoggingArgs(
 target_bucket=log_bucket.id,
 target_prefix="log/",
 )])

Because bucket takes an Output from log_bucket as an input, we can’t create the bucket until after the log_bucket is created. We have to create the log_bucket first to compute its ID, which we can pass to bucket. This idea extends inductively for arbitrary programs – before any resource can be run, we must resolve the Outputs of all of its arguments. To do this, Pulumi builds a dependency graph between all resources in your program. Then, it walks the graph topologically to schedule provisioning operations.

Provisioning operations that are not dependent on each other can be executed in parallel, and Pulumi defaults to unbounded parallelism, but users can ratchet this down if they so desire. Consider this embarrassingly parallel Python program:

import pulumi
import pulumi_aws as aws

# SQS
for i in range(100):
 name = f'pulumi-{str(i).rjust(3, "0")}'
 aws.sqs.Queue(name)

# SNS
for i in range(100):
 name = f'pulumi-{str(i).rjust(3, "0")}'
 aws.sns.Topic(name)

In this program, we can create 200 resources in parallel because none of them take inputs from other resources. This program should be entirely network-bound because Pulumi can issue all 200 API calls in parallel and wait for AWS to provision the instances. We discovered, however, that it did not! Strangely, API calls were issued in an initial batch of 20; as one completed, another would start.

The Fix

The culprit was the Python default future executor, ThreadPoolExecutor. We observed that benchmark was run on a four-core computer, and in Python 3.5 to Python 3.7, the number of max workers is five times the number of cores, or 20 (in Python 3.8, this number was changed to min(32, os.cpu_count() + 4)). We realized we shouldn’t be using the default ThreadExecutor, and instead we should provide a ThreadExecutor with an adjusted number of max_workers based on the configured parallelism value. That way, when users run pulumi up --parallel, which issues an upper bound on parallel resource construction, the ThreadExecutor will respect that bound. We merged a fix that would plumb the value of --parallel through to a custom ThreadExecutor and measured the impact this change had on the performance of our benchmark.

Experimental Setup

We designed and implemented two independent experiments to evaluate this change. The first experiment measures how well the patched Python runtime stacks up against the control group, Pulumi Python without the patch. The second experiment compares Pulumi Python to Pulumi TypeScript using the same benchmark ported to TypeScript. We used the awesome benchmarking tool hyperfine to record wall clock time as our indicator of performance.

The experiments ran overnight on a 2021 MacBook Pro with 32GB RAM, the M1 chip, and 10 cores. Experimental code is available on GitHub, and release tags pin the version of the code used for each experiment. We also made an effort to run the experiments on a quiet machine connected to power. For all experiment groups, --parallel was unset, translating to unbounded parallelism.

Before between samples, we ran pulumi destroy –yes to ensure a fresh environment. Hyperfine measures shell startup time and subtracts the value before final measurements are recorded to more precisely represent the true cost of execution. All groups collected 20 samples each. We also discard stderr and stdout to reduce noise associated with logging to a tty, but we do record the status code of each command so can show they executed successfully.

Python: Pre- and Post-patch

This experiment compares the performance of Pulumi Python before and after the patch was applied. The control group used Pulumi v3.43.1, while the experimental group used Pulumi v3.44.3. The primary difference between these two groups is that a fix was introduced for a Python runtime concurrency bug as part of v3.44.0. Both groups use the same benchmark program, which created 100 AWS SNS and 100 AWS SQS resources in parallel, as described earlier. Only the version of the Pulumi CLI is different between groups.

Control vs. Fix

Group	Mean	Standard Deviation
Control	222.232 s	0.908 s
Experimental	70.189 s	1.497 s

Summary: The Experimental Group ran 3.17 ± 0.07 times faster than the Control Group, accounting for a +300% speedup in performance. Running Welch T-Test indicated statical significance (p = 2.93e-59, α=0.05).

Python vs. TypeScript

After seeing very promising results from the first experiment, we wanted to determine just how promising these results were. We decided to compare Pulumi Python to Pulumi TypeScript to see if this fix had narrowed the gap in performance between the two runtimes. We ported the Python program to TypeScript:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

// SQS
[...Array(100).map((_, i) => {
 const name = `pulumi-${i}`;
 new aws.sqs.Queue(name);
})];

// SQS
[...Array(100).map((_, i) => {
 const name = `pulumi-${i}`;
 new aws.sqs.Queue(name);
})];

For this experiment, we fixed the version of the CLI to v3.44.3, which included the patch to the Python runtime. Here are the result.

TypeScript vs. Python

Group	Mean	Standard Deviation
Python	70.975 s	0.909 s
TypeScript	73.741 s	1.574 s

Summary: The Python Group performed the best and ran 1.04 ± 0.03 times faster than the TypeScript Group. This accounts for a 4% difference in performance. A second T-Test indicated statical significance (p = 1.4e-07, α=0.05). Not only did Python close the gap with TypeScript, but it is also now marginally faster than its Node.js competitor.

Conclusion

It’s rare to have a small PR result in such a massive performance increase, but when it happens, we want to shout it from the rooftops. This change, which shipped last year in v3.44.3, does not require Python users to opt-in; their programs are now faster. This patch has closed the gap with the Node.js runtime. Users can now expect highly parallel Pulumi programs to run in a similar amount of time between either language.

Artifacts

You can check out the artifacts of the experiments on GitHub, including the source code.

Here are some useful links:

The GitHub repository
Artifacts from the first experiment, “Control vs. Fix” or “Pre- and Post-patch”.
More statistics about the first experiment.
Artifacts from the second experiment
More statistics about the second experiment.
Pulumi Internals

Read Every Single Error

Evan Boyle — Tue, 21 Mar 2023 00:00:00 +0000

At Pulumi we read every single error message that our API produces. This is the primary mechanism that led to a 17x YoY reduction in our error rate. You’re probably wondering how reading error messages make them go away.

Doesn’t common wisdom tell us that we need a fancy observability toolchain, or to follow the Google SRE model? I can confidently say that you don’t. I’ll go a step further and state that throughout my career, every system I’ve worked on that relied on aggregate views of errors was a complete dumpster fire. In every team where we instead chose to read all the errors, reliability naturally improved over time.

I offer a concrete process that will drive your error rates down over time with math to back it up.

Read Every Error Message That Your System Produces

You should read every error message that your system produces. Simple but effective. Our team pumps every 5XX into a Slack channel and reviewing each of these is a top priority for the current on-call engineer. There’s a little more to it, but that’s the gist! Commit to this process and your error rates are guaranteed to drop. And I can prove it!

Reliability from First Principles

Why does reading error messages imply improving error rates? It isn’t magic. You still have to dedicate time to fix the bugs you’re shipping and make proactive investments. But you can model this process with a simple inequality:

(API Call Volume) * (Error Rate) * (Time to Triage an Error) < On-Call Attention

This comes with a few important constraints:

On-call attention is a fixed commodity
API call volume scales with your business
This process requires reading every error message

The end result is simple. Your error rate must improve over time to keep the scales balanced. If it doesn’t, the process becomes untenable.

Let’s assume you have one engineer on call, and they spend at most an hour a day triaging errors (they have other responsibilities too!) on each of the ~250 business days per year. That’s a cap of 250 hours per year that can be spent triaging errors. This is effectively fixed. Sure you can hire and split systems out into separate on-call rotations to increase capacity, but our goal is to scale exponentially with respect to humans, not linearly!

Let’s say that triaging an error is a 5-minute process that might involve any of the following:

Checking the issue tracker for a pre-existing ticket to add additional context, a +1, etc.
Reaching out to a teammate who just shipped some buggy code so they can get a fix out
Filing a well-documented bug
Starting a Slack thread with the team to raise a known issue that seems to be cropping up more regularly
Just opening a PR if it’s a simple fix

Remember we only have 250 hours or 15,000 minutes per year to triage with a single on-call engineer. At five minutes a pop we can triage ~3000 errors per year until things start dropping on the floor.

Now imagine you work on a new product and you just launched your MVP. It is early days, but there is some traction and you’re seeing API traffic at a rate of 1,000,000 requests per year. Your annual budget for triaging as we previously determined is 3000 errors, which yields a maximum permissible error rate of 0.3%, or a 99.7% success rate.

Over the next six months, the team iterates, listens to customers, and delivers a ton of value. As a result, traffic levels grow to a rate of 10,000,000 requests per year. But still, our team hasn’t grown much and we still have just one engineer on call at any given point in time meaning our error triage capacity remains fixed at 3000 errors per year. In order to keep up with triaging the error stream at the increased levels of traffic, the team must improve the error rate from 0.3% to 0.03%. And hopefully, this team continues to be successful, increasing API traffic superlinearly in years to come.

If you want to be able to read every error message, then the error rate has to come down as API traffic increases.

Why Should You Care?

Pulumi aspires to be the most reliable infrastructure that our customers interact with, and the benefits towards that end are reason enough for us. But this process is by no means free, and there is always an opportunity cost.

However, we noticed a powerful second-order effect emerge over time. The team began obsessing over the user experience. Following this process builds a visceral understanding of how your system behaves. You know when a new feature has a bug before the first support tickets get opened. You know which customer workloads will require scaling investments in the next 90 days. You know what features see heavy usage and which ones customers ignore. You’re forced to confront every wart in your application. Slowly, your team builds a better understanding of your customers and this trickles down into every aspect of product development. You begin to hold yourselves to higher standards.

A friend and engineer at a large-cap software company read a draft of this post and told me:

“I can’t imagine this process being set on any of the systems I work on at [redacted]. They are so prone to 500s and everybody shrugs their shoulders as if it is just commonplace.”

Apathy in place of customer obsession is not an option if you are a startup that wants to disrupt anything.

The SRE’s Folly

Error budgets and the SRE model are haute couture. Some preach that we should never look at errors at this level of granularity and instead use expensive tools that aggregate, categorize, and collect statistics on errors flowing through your system. But all of this automation can actually make things worse when you reach for it prematurely. Aggregating errors is a great way to gloss over important details early on. Collecting fancy metrics does not matter if your users are not happy. Cutting your teeth with the tools and processes that make sense for your level of scale is the only way to build a high-performance culture. Skipping straight to step 100 does not always help.

Admittedly, this process does not work for Google-level scale. But it works a lot longer than you might imagine. Pulumi manages a meaningful percentage of the resources deployed across all clouds. I asked large-cap software engineer about traffic levels and believe it or not, it is in the same order of magnitude as what we see at Pulumi.

We’re still reading every error.

Achieving Amazing Performance in the Pulumi CLI

Robbie McKinstry — Thu, 26 Jan 2023 00:00:00 +0000

This is the first post in a series about performance optimizations we’ve made to the Pulumi CLI. Over the last six months at Pulumi, the Platform Team has been working on a project we call “Amazing Performance.” Amazing Performance is a new initiative to improve the throughput and latency of the Pulumi CLI not only for power users but for everyone. By the end of June 2022, we assembled a list of issues containing both high-value improvements requiring a sizable investment and low-hanging fruit for quick wins. The full list, including the items we have yet to tackle, is contained in a tracking issue on GitHub. This blog series will cover the highlights.

This post has two sections. First, we’ll describe the tools we built previously to track performance. Secondly, we’ll recap a few quick wins. Future posts will detail our major wins as part of the Amazing Performance initiative.

Measuring and Tracking Performance

The Amazing Performance initiative didn’t come from a vacuum. We’ve been working to understand and improve the performance of the Pulumi CLI for years. Much of the earlier effort has focused on gaining performance insights – building an understanding of where our performance pains were the sharpest. The two most impactful tools at our disposal for understanding performance pains are our analytics dashboard and our OpenTracing support.

Analytics Dashboard

We’ve built an analytics dashboard and monitoring system to let us know when performance dips. This system has three components: a suite of benchmarks, a data warehouse for analytics, and an alerting system.

Benchmark Suite: We’ve written a suite of benchmarks measuring different performance characteristics of the CLI. Each benchmark answers a simple question: “How long does the CLI take to run when the plan has no changes?” or “How long does the CLI take to run an empty plan?” In addition to these simple questions, we also have a few benchmarks to stress test creating many resources. We’ve implemented these benchmarks in Go, JavaScript, TypeScript, C#, and Python. Each night, we exercise the benchmarks, capturing samples. Once execution is complete, our nightly job uploads the results to a serverless function which stores them in our data warehouse.

Warehouse Analytics: We use Metabase to query and analyze data in our data warehouse. Pulumi engineers can log into Metabase and view our performance dashboard, which charts the nightly data as a time series, where each data point is the average of that night’s samples.

Plotting the chart as a line graph allows us to identify any performance dips visually. Sometimes the data from our nightly runs are noisy, so it can take a few nights before we can attribute a change in the runtime to a change in the code. In the absence of noise, the line graph allows us to pinpoint the day the regression was introduced so we can leaf through the pull requests merged that day. More often, there’s enough noise in the data that we go through a few days of PRs to find the regression.

Shortly after shipping the performance dashboard internally, we used the initial numbers we observed to establish a service-level objective (SLO).

Each language has an SLO for each of the benchmarks implemented in that language. Our goal for the SLO is to ensure that we set a “do not breach” expectation across the Platform Core team. We want everyone on the team to know when performance slips beyond an acceptable level.

Alerting: What good is an SLO if it’s not observable? Whether the breach comes from a major regression or creeps up over time, any time a benchmark violates an SLO, a Slackbot alerts the Platform Core team that performance has slowed beyond an acceptable level. One objective we had for the Amazing Performance initiative was to ensure there were no alerts by the end of Q3 2022, which we achieved.

Lastly, we established the accuracy of these benchmarks as part of Amazing Performance. We compared the benchmark results we collected in the nightly job with numbers observed on our local laptops and found they were similar. Consequently, we believe our benchmarks are a good predictor of an actual developer’s experience.

Tracing Support

If you ever encounter a particularly sluggish Pulumi program and want to know what is taking so long, you can use Pulumi’s tracing support to look into the details. When run with the --tracing flag, Pulumi will capture an OpenTracing trace of the program’s execution, giving you an idea of where time is spent. We regularly trace programs large and small to see how we can shave off time.

Pulumi can send traces to a Zipkin server, or aggregate them to a local file and let you view them offline. If you provide an HTTP URI to the --tracing flag, then Pulumi assumes that URI points to a web server that accepts Zipkin-format traces, and it will send all tracing data there. If you provide a file URI instead, Pulumi spins up a local Zipkin server to which child processes send their traces. Once execution is complete, Pulumi aggregates the traces into a single local file and spins down the Zipkin server. You can run PULUMI_DEBUG_COMMANDS=1 pulumi view-trace ./up.trace to view the trace in an embedded web UI.

Not all spans are exceptionally well-named, and there are blind spots in our traces. However, traces are typically a reliable way to subdivide the program execution, so hotspots are easily identifiable. In the future, we plan to migrate to OpenTelemetry and improve our trace coverage. (Feel free to get in touch on the Community Slack if you want to help with this effort!)

Quick Wins for Performance Gains

Some of the work we completed for Amazing Performance targeted simple, isolated changes to make the system a little snappier. The rest of this blog details two quick examples.

Skipping a Slow Import

We cut 300ms in boot time a class of programs by dynamically importing TypeScript.

When Pulumi evaluates NodeJS programs, it compiles TypeScript into JavaScript using TS-Node. Consequently, Pulumi bundles the TypeScript compiler into its NodeJS runtime so TS-Node can perform the compilation. However, we noticed JavaScript programs were importing the TypeScript compiler even when unused.

Users typically hit this bug in one of two ways. Some users choose to write Pulumi programs in pure JavaScript. Pulumi was needlessly importing the TypeScript compiler, even when there was nothing to compile.

The second way a user might encounter this bug is in a more advanced deployment pipeline. One common advanced pattern in CI environments or with Automation API is to typecheck Pulumi programs separately from execution. CI will enforce typechecking during pull requests to guarantee that all code hitting the repository’s main branch is valid. Once it’s typechecked and merged, the Pulumi program can precompile the TypeScript into JavaScript before execution. Finally, when it comes time to execute the Pulumi program, the deployment pipeline will run the compiled JavaScript, which is known to be well-typed. The motivation for precompiling comes from using your own preprocessor. Some customers want to ditch TS-Node for compilation and prefer to use SWC, ESBuild, or other options for increased performance, or for consistency with the rest of their codebase.

In both scenarios, importing TypeScript when it’s not used results in a 300ms slowdown. In a small PR, we detect cases that don’t require TypeScript and dynamically import it only when needed.

Timely Lease Renewal

Another quick win involved the renewal of a new lease, which resulted in another 120ms shaved off of boot time. When the Pulumi CLI kicks off a preview, projects that use the Service backend are authenticated with the Service so the CLI can fetch the current state. During authentication, the CLI exchanges user credentials for a short-lived access token. This access token is valid for a fixed amount of time, called a lease, after which they expire. The CLI can renew a lease by making another request to the Service to extend the duration for which the lease is valid.

We noticed that the first time the CLI renewed the lease, it blocked further execution until the renewal was complete, introducing an unnecessary slowdown of 120ms. The fix was to remove this renewal from the critical path by running it in the backend. By starting the HTTP call on a background thread and only blocking if the token is needed but not renewed, we’ve eliminated the 120ms slowdown.

Since this bug affects anyone using the Pulumi Service backend, most users will benefit from this improvement. It’s small, but it’s noticeable to most users. Program start-up will feel slightly more snappy.

Stay Tuned!

This post covered tools we built before Amazing Performance and a few smaller changes we made as part of the initiative. Future posts will describe some of the larger advances we made to drive Pulumi performance ever forward! Keep an eye out for our next post in the series!

Enabling Rapid Pulumi Prototyping with Rust

Aaron Friel — Wed, 21 Sep 2022 00:00:00 +0000

Pulumi enables engineers to employ the best practices of their field to infrastructure as code. The pulumi watch command is an example of this, enabling rapid prototyping and a “hot reload” style developer experience for prototyping Pulumi programs. In this post you’ll see what watch mode enables, the challenges encountered in maintaining the feature, and how we were able to use Rust to bring that feature to more of our users.

When developing a web application, hot reload frameworks enable quickly iterating and confirming the divs are centered. When implementing new features, a test runner with a watch mode makes test-driven development a breeze. In languages with an interactive shell, it’s natural to use that to experiment with new APIs. In each of these cases, engineers find value in being able to focus on writing code, not typing console commands.

That’s why Pulumi Watch Mode exists and is one of my favorite features, and it’s why I was disappointed to learn this feature was not available to users on Apple Silicon Macs. As a Windows and Linux user, I discovered this while root causing a Pulumi provider’s failing build to the library used to implement watch mode. Thus began a deep dive into cgo, file-watching libraries in Go, operating system event APIs, and finding a solution for our build issues and missing feature.

To close this feature gap for one of our most important platforms, Pulumi v3.39.0 includes a binary built with Rust to implement file-watching. pulumi watch works as it always has. And as the first component to use the Rust language, it validated Pulumi’s approach to engineering of using the best languages and tools available.

But first, what is and why use watch mode?

Watch mode monitors a project folder and automatically runs an equivalent to pulumi up with the appropriate flags to skip prompts and displays simplified output. To see how quickly this can work, I wrote a Pulumi TypeScript program to deploy an S3 bucket and bucket object “test.txt” before writing a timestamp to the same file. That allows me to measure the inner loop with pulumi watch or using an external program to run pulumi up. On each deployment, the program changes one of the files in the directory kicking off the next deployment:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as fs from 'fs/promises'

const bucket = new aws.s3.Bucket("my-bucket");

const bucketObject = new aws.s3.BucketObject("my-bucket-object", {
 bucket: bucket.id,
 key: "test.txt",
 source: new pulumi.asset.FileAsset("test.txt"),
});

bucketObject.content.apply(() => {
 fs.appendFile('test.txt', `${new Date().valueOf()}\n`);
});

On a laptop with this simple program, pulumi watch consistently completed up to 50% faster, at 2.1 seconds per iteration than when using an external tool. Watch mode was more consistent, too: the standard deviation over 50 samples was reduced to 46ms from 216ms. It does this while being easier to remember and therefore use correctly, and with more terse output tuned for the use case of quickly iterating on changes to a stack.

Watch mode has changed how I write infrastructure as code, and it brings that benefit to everyone without needing to install and learn third-party tools for file-watching.

The Challenges of File-Watching in Go

Returning to the cross-compilation issue, the team found builds failed after updating the Pulumi library dependency. Root causing this, I found that through a chain of Go package dependencies, the file-watching library became a compilation unit in provider binaries. That library had several reported issues with cross-compilation, with users reporting that it required enabling cgo and building on macOS to target macOS.

That raised two flags. First, cgo is not Go. Using cgo entails wide-ranging side effects and results in a more complex build, packaging, and support lifecycle. Second, continuous integration tools that support macOS runners are limited and typically much higher cost. It was unacceptable to allow unintentional changes in a provider or the Pulumi repository to impose those requirements on providers.

Why did the notify library have these constraints? This library depended on C library support for a macOS API called FSEvents, and its Go source files required a complex set of build tags. While this didn’t seem impossible to work around, the library remains unmaintained and would require us to maintain a fork to address these deficiencies. With no recent contributions to the library to address this or a path to removing the cgo dependency, I had to find another library.

The strongest candidate was the fsnotify library. This library supported the platforms required, did not require cgo, and supported cross-compilation. It did lack recursive directory watching, though the repository listed it as a planned feature. However, in August the maintainer archived the repository and stepped down from maintenance. Though contributors stepped up to take over ownership of the repo, it left the state of the project in doubt. Without features like recursive file watching, the team would have to maintain a fork. The search continued.

The last candidate looked at was the watcher library, which supported recursive file watching across all platforms without cgo. Was this a panacea? Unfortunately, no. It did this by falling back to the simplest method of file-watching: polling. This method works well enough that the library has not changed in four years. The downside to polling is substantially higher CPU usage and battery drain. It would also require workarounds for projects that contain large folders of dependencies, such as a node.js node_modules or python venv directory. In spiking on this, I was unsurprised to see that watching a folder with vendored dependencies was orders of magnitude slower.

The Go language ecosystem lacks libraries that meet our needs. It was possible to watch only one directory, or only support one operating system, or to be insensitive to performance concerns. I needed an option that didn’t make any of these sacrifices and, ideally, was also actively maintained.

From library to binary and the carcinization of Pulumi

One doesn’t have to look hard to find CLIs that tackle this problem, many of which were at the top of search results while looking for libraries. Using one of these as a plugin and reading stdout for events instead would avoid any link-time dependency for providers. With this in mind, I considered building a one-off binary using the previous utilities but ruled them out. The team would become de facto maintainers of a fork of a complex library, in addition to needing to solve the deficiencies identified above. That wasn’t an innovation token I could spend.

I evaluated the complexity of building on the watchman service, an excellent file-watching CLI open-sourced by Facebook. Unfortunately, it would be costly to maintain a wrapper with watchman as a dependency. While C++ isn’t inaccessible, it does impose a complex build system and a higher barrier to entry for engineers to contribute to it. And as both a client and server application, it wouldn’t be as simple as packaging a single binary and shelling out to it.

Finally, I spiked on a library written in Rust, using the library underpinning the watchexec CLI. All-in, this worked out to just over 100 lines of code to maintain on top of the actively maintained watchexec libraries, with a similarly small patch to the Pulumi CLI. And importantly, building the binary is a simple cargo build command that the team can run on any platform. The watchutil-rs implementation lives in GitHub and is packaged as pulumi-watch, a single-purpose binary that logs on stdout when files change.

Pulumi’s mission is to enable everyone to apply the best programming languages and tools to infrastructure as code. Implementing watch mode in Rust applied that principle to watch mode while maintaining performance, moving the feature to a well-maintained footing, and expanding support to every platform Pulumi supports.

Give pulumi watch a try with our getting started guide!

Node.js Native Binary Compilation Using vercel/pkg

Daniel Bradley — Tue, 13 Sep 2022 00:00:00 +0000

In Pulumi’s engineering department, we often build and distribute tools as native binaries to avoid the need for additional dependencies on user machines. Most of these tools are written in Go, which has good support for building self-contained binaries that target modern operating systems.

While other Pulumi-supported languages like Node.js, Python, and .NET require additional runtime dependencies, it’s possible to bundle dependencies with the program. In this article, we’ll show you how to do that for a Node.js program.

This is a problem that can be solved using vercel/pkg command line tool. Here’s the summary from their readme.

This command line interface enables you to package your Node.js project into an executable that can be run even on devices without Node.js installed.

Let’s take a look at how to use pkg and some issues we encountered on the way.

Setting up `pkg`

1 - Install

pkg is distributed as an npm package which can be installed into your “devDependencies” using:

npm install -D pkg

yarn add -D pkg

or run without installing with npx:

npx pkg [args]

2 - Set `bin` in `package.json`

pkg will use the bin field from your package.json to find the entry point so you just have to specify a path to the directory containing your package.json.

{
 "name": "my-program"
 "bin": "bin/index.js",
 ...
}

If no target is specified, then a set of defaults will be chosen for you. If the output path is not specified, pkg will infer the name from the package.json “name” field and write to the current working directory.

3 - Execute

The main inputs that pkg needs is:

The entry point to your program for packaging
The target machine to build for
The output path to write the finished binary

Here’s an example of building the project in the current directory using node v18 for macOS ARM architectures:

# pkg [options] <input>
# -t, --targets comma-separated list of targets
# -o, --output output file name or template for several files
pkg -t node18-macos-arm64 -o bin/my-program .

It’s possible to specify multiple targets in a comma-separated list to build them all at the same time, but it does come with the limitation where the output file names follow a fixed pattern. Instead, we chose to just run the pkg command multiple times with different arguments from our makefile in parallel as this fits with our existing workflows well.

Issues encountered

“Inspector Not Available”

As soon as we started executing the provider we started seeing some interesting warnings printed to the console stating “Inspector is not available” (here’s our tracking issue and some pkg discussion). This is because we use the Node.js Inspector API as part of our automatic closure serialization, however this is not available by default when packaging with pkg.

pkg provides an option to fix this by building your own base image with custom Node.js flags set to enable debugging. However, on investigation, these issues were caused by the Pulumi Typescript SDK creating and caching an inspector instance at the point of being imported even though we never actually call this code in our plugin. Therefore, we opted to make this eager singleton creation to be lazy – only created on first use, as we’re not using in our providers at this time.

Unrunnable MacOS ARM binaries

When trying to use the binaries produced by our CI, we found that the binaries weren’t runnable on MacOS ARM architectures - and were forcibly killed by the operating system.

This led me down quite a bit of a rabbit hole investigating signing of binaries, but it was actually resolved by simply installing the ‘ldid’ tool on our linux CI environments. The ldid source is available via git.saurik.com/ldid and binaries are available from various sources. Our solution was to use the “Install Ldid” GitHub Action to install the ldid binary and add it to the PATH in our CI workflow. My learning here is that sometimes reading the warnings in logs more carefully can save you lots of time!

Static Binaries

One adjustment to our configuration came from a community contribution by @afreakk where the provider was being used in a nixos environment. Nixos adds the requirement for all binaries to be static rather than dynamic - so there’s no requirement for the operating system to dynamically map link functions from system libraries at runtime. Statically compiled programs sometimes result in a larger size, but avoid any possible issues with different versions of the libraries it depends on.

To resolve this issue, it’s as simple as changing the ‘linux’ targets to ‘linuxstatic’. E.g. node18-linux-amd64 becomes node18-linuxstatic-amd64.

Multi-platform builds with a makefile

We use makefiles to build our providers, so here’s a brief outline of how we build for multiple platforms using GNU Make.

# Set the correct pkg TARGET for each binary we build
# when building for linux-amd64, set the pkg target to node18-linuxstatic-x64
bin/linux-amd64/my-program: TARGET := node18-linuxstatic-x64
# output binary file ^ ^ variable ^ pkg target
bin/linux-arm64/my-program: TARGET := node18-linuxstatic-arm64
bin/darwin-amd64/my-program: TARGET := node18-macos-x64
bin/darwin-arm64/my-program: TARGET := node18-macos-arm64
bin/windows-amd64/my-program.exe: TARGET := node18-win-x64

# Wildcard rule to build any of binary outputs
# "To build any bin file, ensure node_modules are up to date..."
bin/%: node_modules
 # "... then run pkg for actual output name & target"
 yarn run pkg . --target ${TARGET} --output $@
 # "TARGET" is the variable defined above, depending on the output
 # "$@" is the current makefile target - e.g. 'bin/linux-amd64/my-program'

# Running `make bins` will build all the listed outputs
bins: bin/linux-amd64/my-program
bins: bin/linux-arm64/my-program
bins: bin/darwin-amd64/my-program
bins: bin/darwin-arm64/my-program
bins: bin/windows-amd64/my-program.exe

# Let Make know that `bins` doesn't really exist (is phony) - it's just a helpful shortcut
.PHONY: bins

In summary the above code does the following:

Define each output file we need
- Set the correct pkg TARGET for each output
Define a rule for building any binary output:
- Ensure module_modules are up-to-date (this is another make target not shown here).
- Run pkg with the $TARGET set in (1) and the name of the current output ($@).
Define bins “phony target” which builds all listed bins.

You might also like:

Recap

I hope this gives you a good overview of the vercel/pkg tool and how you can use it to create standalone programs using Node.js.

Historical Note

We used to do a similar process using the nexe project, but there’s been no releases since 2017 and therefore no support for newer versions of Node.js and we therefore consider this package as unmaintained at this point in time.

Exploring how to solve circular dependencies

Daniel Bradley — Wed, 12 Jan 2022 00:00:00 +0000

This blog post originated from a hackathon and exploratory exercise, and should be considered in that context. None of the solutions exist in the Pulumi SDKs. For more information on this, please view our blog post on Circular Dependencies.

As part of our hackathon near the end of last year, we decided to explore solutions to a common problem when people are using Pulumi for their systems. A question that’s been asked in a few different forms is how to resolve circular dependencies between resources in a Pulumi program.

A simple example of this idea is a modern web application with a static front-end and an API, where the front-end needs to know the URL of the API to be able to call it and the API needs to know the source domain of the front-end to allow it access via CORS. As these two resources rely on one another to be created, they are circular dependencies.

For a deployable example, let’s build a serverless game of ping-pong—two functions that invoke one another until one misses!

const ping = new CallbackFunction("ping", {
 callback: pingPongHandler,
 environment: {
 variables: {
 OPPONENT_FN_NAME: pong.name, // <-- Can't access `pong` yet!
 },
 },
});

const pong = new CallbackFunction("pong", {
 callback: pingPongHandler,
 environment: {
 variables: {
 OPPONENT_FN_NAME: ping.name,
 },
 },
});

Programming languages don’t let us create one object that takes a value from another object that doesn’t yet exist. Therefore, we can’t express a circular dependency in this way.

The simplest workaround for this error is to get a reference to our own stack, which contains the outputs from the previous deployment. Now, we set the reference to pong within ping from the output of the last deployment.

const currentStack = new pulumi.StackReference(
 `acme-org/${pulumi.getProject()}/${pulumi.getStack()}`
);

const ping = new CallbackFunction("ping", {
 callback: pingPongHandler,
 environment: {
 variables: {
 OPPONENT_FN_NAME: currentStack
 .getOutput("pongName")
 .apply((pongName) => pongName ?? ""),
 },
 },
});

const pong = new CallbackFunction("pong", {
 callback: pingPongHandler,
 environment: {
 variables: {
 OPPONENT_FN_NAME: ping.name,
 },
 },
});

export const pingName = ping.name;
export const pongName = pong.name;

The downside of this approach is that we have to deploy twice to reach the desired state because we need the results of the previous deployment, though this approach might be fine for simple one-off circumstances.

How then could we make the process easier and clearer for users who need to resolve this kind of circular dependency regularly? We explored four options as part of the hackathon, so let’s go through each one, getting progressively closer to a possible solution.

Option 1: State

Instead of referencing outputs directly from our stack’s previous deploy and manually deploying twice, we could introduce a special type called “State”.

The State object does 3 tasks:

Read the value from the previous deployment, or provide a default if it’s the first deployment.
Update the value in the stack state using a “set” method.
Automatically triggers another deployment if its value has changed.

const pongName = new pulumi.State("pong-name", {
 initialValue: "",
});

const ping = new CallbackFunction("ping", {
 callback: pingPongHandler,
 environment: {
 variables: {
 OPPONENT_FN_NAME: pongName.value,
 },
 },
});

const pong = new CallbackFunction("pong", {
 callback: pingPongHandler,
 environment: {
 variables: {
 OPPONENT_FN_NAME: ping.name,
 },
 },
});

// Engine re-runs "up" if this changes the value
pongName.set(pong.name);

The first deployment will match the preview, but when triggering the re-deployment, we’re making a change that wasn’t in the first preview. This action could also result in a never-ending loop where every deployment makes another change to the state and triggers another deployment. So let’s explore a different option that might not have this issue.

Option 2: Refs

This option looks almost identical to the “state” concept above, except we’re calling them references (refs):

const pongName = new pulumi.Ref("pong-name", {
 initialValue: "",
});

const ping = new CallbackFunction("ping", {
 callback: pingPongHandler,
 environment: {
 variables: {
 OPPONENT_FN_NAME: pongName.value, // Special type the Pulumi engine will track
 },
 },
});

const pong = new CallbackFunction("pong", {
 callback: pingPongHandler,
 environment: {
 variables: {
 OPPONENT_FN_NAME: ping.name,
 },
 },
});

// Determine affected resources and re-update
pongName.set(pong.name);

The difference between refs and state is that, instead of triggering a whole re-deploy when the value changes, refs would be integrated into the dependency graph similarly to resources. When the value of the ref changes, we could internally work out which resources depend on that ref and change the target state for only those resources. This avoids the possibility of a never-ending update loop.

Both the “state” and “refs” approaches have the downside of not making the final outputs of the updated resource available to the rest of the program. The resource object would still have the values from after it first deployed. This means that other parts of the program might still deploy differently on subsequent deployments. Let’s explore another option that might address that problem.

Option 3: Create patch resources

We could explicitly model the third step by using a new resource type suffixed with “patch”:

const ping = new CallbackFunction("ping", {
 callback: pingPongHandler,
});

const pong = new CallbackFunction("pong", {
 callback: pingPongHandler,
 environment: {
 variables: {
 OPPONENT_FN_NAME: ping.name,
 },
 },
});

const pingPatch = new CallbackFunctionPatch("ping-patch", {
 id: ping.id,
 environment: {
 variables: {
 OPPONENT_FN_NAME: pong.name,
 },
 },
});

The provider presents pseudo-resources for update or patch operations. The patch resource is using the same underlying resource as the original resource. This option is a very flexible approach that could be implemented by providers right now. These patch resources could also be used for resources that exist outside the Pulumi stack. The drawback is that the patch operation has to be implemented in each provider individually, and it significantly increases the overall size of each provider and its associated SDK. We’re still evaluating whether this option would be a good one considering that size increase, which increases download times as an example impact, and the engineering resources to implement the change. We have one more option to consider, as well.

Option 4: Allow a resource to update its goal state

When using the SDK, we could write something like the following code to create then update a little later on:

const ping = new CallbackFunction("ping", {
 callback: pingPongHandler,
});

const pong = new CallbackFunction("pong", {
 callback: pingPongHandler,
 environment: {
 variables: {
 OPPONENT_FN_NAME: ping.name,
 },
 },
});

const updatedPing = ping.update({
 environment: {
 variables: {
 OPPONENT_FN_NAME: pong.name,
 },
 },
});

When declaring a resource in a Pulumi program, you’re essentially saying, “This is the state I want my resource to be in.” The update method returns a whole new copy of the resource with new properties containing the outputs of the update.

What’s next?

The final option is quite close to a fully working solution, but there still remains a few tasks to complete before we could release it as a feature:

Finalise what the internal messages between the Pulumi’s internal components should look like.
Keep a record of circular dependencies in the stack state without causing knock-on issues elsewhere in Pulumi’s internal workings.
Decide what should happen if an update triggers a replacement—if this is potentially valid behaviour or not. We should be able to generate the set of properties which are able to be updated without causing replacement and provide these properties to the user as the arguments for the update method to avoid these issues before execution of the program.

Some other ideas that would be interesting to explore in the future are as follows:

How could we utilise patching of resources where a provider can support it?
Could we delete a resource that was created earlier in the program to allow the creation of temporary resources?

This hackathon project has opened up a number of new pathways for us to explore. If you want to participate, we always welcome contributions in our open-source repos at https://github.com/pulumi. We hope you have enjoyed this dive into how our team explored this concept as part of the hackathon. We’ll be exploring more of the hackathon projects soon on the blog!

Supercharging our GitHub Action with the Automation API

Paul Stack — Mon, 01 Mar 2021 00:00:00 +0000

Guest Article: Simen A. W. Olsen is a Software Architect and Manager at Bjerk, a software development agency based in Oslo, Norway. He joins Paul Stack to talk about the new GitHub Action powered by the Pulumi Automation API.

In this article, we’ll take a look at how our GitHub Action has been re-architected to be built on top of our Automation API.

As a GitHub Actions launch partner in 2018, we released our original GitHub Action, built as a Docker container action. This Action packaged all the prerequisites with the Pulumi CLI to let users run Pulumi commands. As the Pulumi ecosystem has grown, so too did the container run in GitHub Actions.

Since then, GitHub released an alternative way to create a GitHub Action by writing JavaScript. At Pulumi, and in our community, we love expressing everything we possibly can in code, so the new JavaScript support got us quite excited about the “better together” possibilities! Enter Pulumi’s new Automation API, a new feature we launched in public preview in October 2020. This new capability lets you build infrastructure as code—in pure code—without even needing to run a CLI. This sounded like a perfect match for a JavaScript Action. Creating a JavaScript Action allows us to programmatically invoke the right Pulumi commands in response to actions, without needing to awkwardly script a CLI. Instead, we were able to leverage the Automation API to fundamentally rethink our GitHub Action.

Under the Hood

Let’s take a look at the new JavaScript-based GitHub Action and see how it uses Automation API to execute Pulumi:

const main = async () => {
 const config = await makeConfig();

 if (environmentVariables.PULUMI_ACCESS_TOKEN !== '') {
 await pulumiCli.run('login');
 } else if (config.cloudUrl) {
 await pulumiCli.run('login', config.cloudUrl);
 }

 const workDir = resolve(
 environmentVariables.GITHUB_WORKSPACE,
 config.workDir,
 );

 const stack = await LocalWorkspace.selectStack({
 stackName: config.stackName,
 workDir: workDir,
 });

 const onOutput = (msg: string) => {
 .....
 };

 const actions: Record<Commands, () => Promise<string>> = {
 up: () => stack.up({ onOutput }).then((r) => r.stdout),
 refresh: () => stack.refresh({ onOutput }).then((r) => r.stdout),
 destroy: () => stack.destroy({ onOutput }).then((r) => r.stdout),
 preview: async () => {
 const { stdout, stderr } = await stack.preview();
 onOutput(stdout);
 onOutput(stderr);
 return stdout;
 },
 };

 const output = await actions[config.command]();

 const outputs = await stack.outputs();

 for (const [outKey, outExport] of Object.entries(outputs)) {
 core.setOutput(outKey, outExport.value);
 if (outExport.secret) {
 core.setSecret(outExport.value);
 }
 }
};

The Action code is clean and concise because the power lies in the Automation API itself. The Action uses the Automation API to drive Pulumi as part of your GitHub Actions workflows. This means that your workflows can take advantage of changes to Pulumi and the Automation API as soon as they’re released! And, of course, you can always write your own custom logic using the Automation API like our Action does.

Customizing Your Environment

The new Action puts you in control of the environment that it runs on. By comparison, the previous Docker-based Action bundled specific tool versions, which left your CI pipeline at the mercy of our opinionated container. Now, if you want to pin to specific versions of NodeJS / Python / Go / .NET or even Pulumi, you have that power!

You can even include a matrix strategy to test your Pulumi code against different language runtime versions and different operating systems:

name: 'Sample Pulumi NodeJS Application'
on:
 pull_request:

jobs:
 test-workflow:
 runs-on: ${{ matrix.os }}
 name: Testing Pulumi Preview on ${{ matrix.os }} with NodeJS ${{ matrix.node_version }}
 strategy:
 matrix:
 os: [ubuntu-latest, macos-latest, windows-latest]
 node_version: [10.x, 12.x, 14.x]
 fail-fast: false
 steps:
 - uses: actions/checkout@v2
 - uses: actions/setup-node@v1
 with:
 node-version: ${{ matrix.node_version }}
 - name: Install pulumi
 uses: pulumi/action-install-pulumi-cli@v1.0.1
 - run: npm install
 - uses: pulumi/actions@v2
 env:
 PULUMI_CONFIG_PASSPHRASE: not-a-secret
 with:
 command: preview
 cloud-url: file://~
 stack-name: dev

Cloud Interaction

The previous Docker-based Action would look for the GOOGLE_CREDENTIALS environment variable; if your workflow set that variable, the Action would run the following commands on your behalf:

gcloud auth activate-service-account --key-file=$GOOGLE_APPLICATION_CREDENTIALS
gcloud --quiet auth configure-docker $GOOGLE_DOCKER_HOSTNAME_LIST

With the new Action, you’re fully in control of authentication to cloud providers. To authenticate with AWS / Azure / Google Cloud, you can now use their official Actions:

 - name: Configure AWS Credentials
 uses: aws-actions/configure-aws-credentials@v1
 with:
 aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
 aws-region: ${{ secrets.AWS_REGION }}
 aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

 - uses: azure/login@v1
 with:
 creds: ${{ secrets.AZURE_CREDENTIALS }}

 - uses: google-github-actions/setup-gcloud@v0
 with:
 service_account_key: ${{ secrets.GCP_KEY }}
 project_id: ${{ env.PROJECT_ID }}
 export_default_credentials: true

We recommend that you store all sensitive environment variables, like cloud credentials, in the Actions secret storage and consume them using the secrets attribute on your workflow’s action.

Faster

The new GitHub Action is much smaller and no longer needs to download a (very!) large Docker container. In the Docker-based Action, we frequently saw workflows take up to 3 minutes to download the container. The new Action is much faster; let’s take a look:

Old Docker-based Action (v1)

New JavaScript-based Action, powered by Automation API (v2)

The new Pulumi GitHub Action is ready for you to use. Here’s how to get started:

Already use the v1 Pulumi GitHub Action? Migrate to the new one with our migration guide.
New to Pulumi and GitHub Actions? Use our getting started guide to set up your first Pulumi workflow.

We’re eager to hear your feedback on this new Action. If you haven’t already signed up for our Community Slack, it’s quick and easy! You can join in on conversations you like and get help from other community members, as well as the Pulumi Team. That’s it for now!

Lambdas as Lambdas: The magic of simple serverless Functions

Cyrus Najmabadi — Wed, 10 Oct 2018 00:00:00 +0000

Pulumi’s approach to infrastructure as code uses familiar languages instead of YAML or DSLs. One major advantage of this approach is that AWS Lambdas, Azure Functions, Google Cloud Functions, et al. can just be real lambdas in your favorite language, offering a flexible and simple path to serverless. Such functions behave as normal functions, allowing you to treat serverless code as part of your application instead of separate “infrastructure” that needs to be configured, managed, and versioned manually. In this post, we’ll examine this capability in JavaScript, which is already very function- and callback-oriented, making serverless feel like a natural extension of the language we already know and love.

While Functions as a Service (FaaS) systems have become more popular, getting up and running can still feel overly complex compared to normal application development. FaaS offerings today divide the development experience between “infrastructure” – doing all the work to configure the Lambda runtime itself (i.e. how much memory to use, what environment variables should be present, etc.) – and writing and maintaining the code that will execute in the function itself when triggered. Most developers just want to focus on the latter, write some code, and have it work.

For AWS Lambda, for example, the code itself needs to written and packaged up with all its dependencies into a single zip-file, stored as a blob that is then placed in AWS S3. This blob is then referenced by Lambda so that it can be loaded on-demand when necessary. Updating just a single character of this code, like fixing a bug, then requires fully repackaging and re-uploading everything. Compounding all of that, this code is necessary independent of any other pieces of your application. If the code for the Lambda needs to read or write to an S3 bucket, publish to an AWS SNS topic, etc., it will need to be manually configured to have that bucket’s information passed to it, often using out of band in environment variables. These same issues also apply when creating FunctionApps using Node.js on Azure.

At Pulumi, we thought there could be a better way to do this. Why have to jump out of your application code just to create another bit of configuration (in YAML or JSON) just to run your application? Put simply, we wanted developers to write event-driven code, in their favorite language and in their favorite IDE, and simply hit “deploy” when done.

Magic Functions

From a desire to do better here, the “closure serialized function” was born. However, “closure serialized function” is a geeky mouthful, so I like just calling them “magic functions.” Our team is comprised of cloud, language, and runtime geeks alike, so we figured out a way to hook into the language runtimes to just make lambdas as lambdas work.

Before diving more deeply into how magic functions work, let’s take a look at how they look in action:

// Example simplified to not get too bogged down :)
const sharp = require("sharp");
const videoBucket = new aws.s3.Bucket("videos");
const videoTopic = new cloud.Topic("videoTopic");
videoBucket.onObjectCreated("onNewVideo", async (event) => {
for (const record of event.Records) {
const dimensions = computeDimensions(record.s3.object.key);
videoTopic.publish({ s3ObjectKey: record.s3.object.key, time: record.eventTime });
}
});
function computeDimensions(objectKey: string): { height: number, width: number } {
/* simplified */
const metadata = await sharp(...);
return { height: metadata.height, width: metadata.width };
}
videoTopic.subscribe("compressVideo", async (event) => {
const { s3ObjectKey, time } = event;
// actually compress video.
});
videoTopic.subscribe("createThumbnail", async (event) => {
const { s3ObjectKey, time } = event;
// actually create thumbnail.
});

Let’s still walk through it to see what’s happening. It can be a little subtle because we are mixing deployment-time code – code that declares infrastructure as code resources – and *runtime *code – application code that runs as a function.

First, we’re just defining two simple resources: 1) an s3.Bucket, where we expect new video files to be uploaded to, and 2) a cloud.Topic that will inform interested parties when new videos are uploaded. (The full surface area of AWS is available in the aws package, and the cloud package offers multi-cloud abstractions that work at a higher level of abstraction.) Right after defining the resources, we start creating our first FaaS resources. videoBucket has an onObjectCreated event subscription function that will fire when new objects appear, and we invoke it with a real JavaScript (or TypeScript) arrow-function defining the code that will execute in an AWS Lambda at runtime.

Not So Fast…

This code is pretty magical in a few ways. To see how it’s magic, let’s consider a fairly naive way to create a Lambda for this onObjectCreated event handler function – something we need to do by hand with FaaS solutions today. Imagine we took the raw source text of this arrow-function by toStringing it and using that as the text of the Lambda code:

// index.js
module.exports = async (event) => {
foreach (var record in event.Records) {
const dimensions = computeDimensions(record.s3.object.key);
videoTopic.publish({ s3ObjectKey: record.s3.object.key, time: record.eventTime });
}
};

Now, the above translation could have worked for an extremely basic function, but totally breaks down here for several reasons. First, computeDimensions isn’t included, which will make this immediately fail if ever called. Second, if computeDimensions was included, it wouldn’t work unless the sharp module was properly imported for it. Third, even if the module was imported in the code, the actual NPM package would have to be available along with the code in the cloud. Fourth, if that module was properly required, and the package was properly included, the code is both referencing videoTopic (a resource defined when the application originally ran), and it is invoking a helper method on it. Remember, this code is going to be executing at cloud runtime, not at the time when Pulumi is actually executing the application on your developer machine, and actually doing things like running the const videoTopic = new cloud.Topic("videoTopic") code.

To summarize, we have quickly realized, we can’t just analyze everything locally to figure out what has to be included, instead the entire transitive closure must be considered – a task usually left up to FaaS developers to perform manually! This coupled with the manual configuration using YAML can lead to plenty a headache.

Compilers to the Rescue

Despite all those issues, Pulumi is able to make that original application code work and it is able to magically effectively translate the above into a working Lambda. The real secret sauce here is that, by leveraging familiar languages, we can stand on the shoulders of all the great compiler work underpinning them.

So let’s dig in – how does this all really work?

(Note that, if you want some of the more nitty gritty details, we have a doc going in depth. For now, we’ll try to stick to providing a high-level overview of what’s going on, so that we can grok the basic approach.)

Pulumi executes your application as a normal Node.js application. As such, when it hits the const videoBucket = new aws.s3.Bucket("videos") lines, it actually goes and creates those resources. When it then hits the videoBucket.onObjectCreated("onNewVideo", async (event) => { line this executes a normal line of JavaScript. Namely, onObjectCreated is a normal JavaScript function that will be passed the name parameter as the first arg, and the JavaScript function as the second arg. onObjectCreated eventually bottoms out with a call to a call to pulumi.runtime.serializeFunction. This function is the root of where our deep analysis of your JavaScript function analysis and translation will happen.

It’s worth pausing here. We can use all the standard approaches to function composition to build bigger functions out of smaller functions, even before we hit that serialization routine. The closures are built up in the V8 runtime as usual, and are eagerly awaiting the transformation into magic functions as soon as they are needed. Pretty cool!

Once we get to that point, using the awesome analysis APIs provided by TypeScript, we start introspecting the function instance, seeing what code it contains, and what external objects it references. The legal objects that can be included in the captured function’s environment include (but aren’t limited) to things like simple JavaScript objects, other functions (like computeDimensions), modules (like sharp), other Pulumi resources (like videoTopic), and even complex beasts like Promises. Basically, anything that Pulumi knows how to serialize and rehydrate inside the Lambda.

When Pulumi determines that these objects are referenced, it automatically includes enough information about them into the final translated Lambda code file so that they are available and can be effectively used at runtime. Thanks to JavaScript’s deep, dynamic introspection capabilities, as well as its rich APIs for creating nearly anything at runtime, it’s very possible to effectively hydrate out all the information present when Pulumi runs, and put it into a form that can be cheaply and accurately rehydrated into runtime objects again when the Lambda runs.

This analysis also understands the NPM package structure of an application and its dependencies, ensuring that necessary imported packages are properly included with the final Lambda. This analysis is also very smart about not including things that aren’t necessary at all. For example, it might drop information from the serialized form if it can prove that would never be observable, in addition to relying on built-in runtime functions being there without needing any sort of serialization. There are definite subtleties and interesting cases that can arise here, and for those very curious, reading the documentation linked above may be very enlightening.

Pulumi also understands diffs and dependencies, so updating is a cinch. Any changes to the code in the JavaScript function (including anything it depends on) is properly tracked and understood by Pulumi, allowing an easy development model where FaaS code can be easily modified and republished without jumping through extra hoops.

The Result? Simple serverless Functions

With this approach, serverless programming becomes dramatically simpler, whether it’s an AWS Lambda, Azure Function, or GCP Cloud Function. (Stay tuned, we’re hoping to bring this to Kubernetes FaaS offerings soon.)

In short, you just write event-driven JavaScript code that’s familiar and easy, you run pulumi up, and everything is taken care of from there. No YAML configuration files, no point and clicking in a cloud console UI, no fuss.

On top of all of that, because this is just code itself, and because these are normal APIs, other tooling (like your editor) knows exactly what is going on and can guide you through this. For example:

While there’s a lot of really cool tech going on under the covers here, we really hope that this ends up just feeling super-awesome to use for you. Now that we have this functionality, we wouldn’t want to ever write an AWS Lambda any other way!

Get started building serverless functions on Pulumi today!

kubespy trace: a real-time view into of a Kubernetes Service

Alex Clemmer — Wed, 26 Sep 2018 00:00:00 +0000

This post is part 3 in a series on the Kubernetes API. Earlier, Part 1 focused on the lifecycle of a Pod, and later Part 3 details how Kubernetes deployments work.

Why isn’t my Pod getting any traffic?

An experienced ops team running on GKE might assemble the following checklist to help answer this question:

Does a Service exist? Does that service have a .spec.selector that matches some number of Pods?
Are the Pods alive and has their readiness probe passed?
Did the Service create an Endpoints object that specifies one or more Pods to direct traffic to?
Is the Service reachable via DNS? When you kubectl ``exec into a Pod and you use curl to poke the Service hostname, do you get a response? (If not, does any Service have a DNS entry?)
Is the Service reachable via IP? When you SSH into a Node and you use curl to poke the Service IP, do you get a response?
Is kube-proxy up? Is it writing iptables rules? Is it proxying to the Service?

This question might have the highest complexity-to-sentence-length ratio of any question in the Kubernetes ecosystem. Unfortunately, it’s also a question that every user finds themselves asking at some point. And when they do, it usually means their app is down.

To help answer questions like this, we’ve been developing a small diagnostic tool, kubespy. In this post we’ll look at the new kubespy trace command, which is broadly aimed at automating questions 1, 2, 3, and providing “hints” about 4 and 5.

Below is a gif demonstrating the CLI experience. You can watch in real-time as the Service comes online, finds pods to target, and finally is allocated a public IP address:

What is kubespy, again?

kubespy is a simple, standalone diagnostic tool, meant to make it easy to introspect on Kubernetes resources in real time.

Before we begin, it’s worth noting that this kubespy actually re-packages the machinery we developed for Kubernetes support in Pulumi.

One of our major goals in this work was to make deploying an application to Kubernetes as simple as possible, by presenting a concise summary of this information in the CLI experience. See my tweetstorm on the subject, or try it out for yourself!

A real-time view of a Service’s life

The kubespy repository contains the simple trace example we use in this demo. The README contains detailed installation instructions, as well as explaining how to run the app (using either kubetl or pulumi though of course we hope you will try Pulumi).

Essentially: running kubespy trace service nginx will cause kubespy to sit and wait for you to deploy a Service called nginx. When you run this example, it will do just this: creating a Deployment which replicates an nginx Pod 3 times and exposes it publicly to the Internet with a Service, also called nginx.

Let’s break down the kubespy trace gif above to show that there are actually several distinct steps in the process of booting up a Service.

First: Service is created, the Service controller creates an Endpoints object of the same name. The Endpoints object is to specify which Pods get traffic — their IPs, which ports to direct traffic to, and so on. In this case, there are no Pods to target, which kubespy trace tells us:

Second: Pods that match the Service’s .spec.selector are created; their readiness probes immediately pass. The Endpoints object is updated to reflect this. As we will see below, if the Pods failed the readiness probes, kubespy trace would note this.

Third: Service is allocated a public IP address. The Service has .spec.type set to LoadBalancer, which on most cloud platforms means that a public IP address should be allocated for it.

Exercise: Other Service types, watching rollouts, deleting Services!

kubespy trace supports all the other Service types, including ExternalName and ClusterIP. Try both of those, and you’ll see slightly different output. Try them! It’s also worth watching what happens when a Service is deleted.

You can also use kubespy trace to watch an unhealthy deployment become healthy. In the following gif, we see a bunch of Pods that are failing readiness checks become healthy as a new version is rolled out:

Conclusions

Confession time. Last time we told you we’d dig more into the lifecycle of a Pod. And we will, at some point. But we ended up deciding that it would be easier to explain with a cohesive trace command.

And, while this is a good start, it is only the beginning. trace currently supports only Service. In our next post, we’ll extend trace to Deployment (or perhaps ReplicaSet), and from there, we will have enough tools to really dig into what is happening when you roll out your app.

In the mean time, if you enjoyed this post, or are curious to see how this lifecycle is baked into the Pulumi CLI, give it a spin! We’d love to have your feedback.

Advanced TypeScript type FTW!

Cyrus Najmabadi — Wed, 19 Sep 2018 00:00:00 +0000

We at Pulumi love TypeScript for cloud apps and infrastructure, because of its rich type system and great ahead-of-time typechecking – making for a more productive inner loop and helping to find errors sooner. The typesystem magic behind how this works for infrastructure as code can be fascinating!

A core part of the Pulumi programming model is that we allow people to express complex dependency data that may eventually be available. Traditional JavaScript programming might expose that as a Promise, but we’ve taken that one step further by introducing a type we call:

type Input<T> = T | Promise<T> | Output<T>;
// Like a Promise<T>, but also keeps track of dependency information so we can figure
// out what the upstream/downstream impact is whenever any values change in the system
interface Output<T> { /* ... */ }

This complex type can often show up as inputs to all Pulumi objects like so:

interface VpcInput {
ipAddresses: Input<Input<string>[]>;
}

Breaking that down, it’s basically saying “I’ll take an array of strings. But those actual string values can individually be strings, promises-of-strings, or strings-carrying-dependency-information.” It even goes one step beyond that, because the entire array itself could be just an array-of-those-things, or a promise-of-array-of-those-things, or an array-of-those-things-with-dependency-information. Whew…!

This approach is great for people producing the data. As a producer I can do:

new Vpc({ ipAddresses: ["an-address"] }); // or even
new Vpc({ ipAddresses: ["an-address", computeAddressAsynchronously() ]}); // or even
new Vpc({ ipAddresses: ["an-address", computeAddressAsynchronously(), someDependency.address ]}); // or even
new Vpc({ ipAddresses: computeAnArrayAnyOfTheAboveAsynchrousnously() }); // etc. etc.

It’s super flexible, and let’s people naturally produce the values in whatever way they want without having to work too hard to make it conform to some very narrow type.

However, while easy to produce, this can sometimes be difficult to consume. For example, if you wanted to take in that Input<Input<string>[]> and then check if there was a very specific value in it, you might have to go through and do a bunch of work. You’d have to first deal with potentially getting a Input<string>[] or a Promise<Input<string>[]> or even an Output<Input<string>[]>. Then, once you even got to the underlying array, you’d have to deal with each potential element in it being a string… or a Promise<string>… or a… well… you get the idea. It’s not fun 😃.

Up until now, we’d supplied some simple helpers to make this more manageable. Our helpers would effectively ‘unwrap’ one layer of this sort of structure allowing you take the external value and deal with an internal value of a known shape. So it effectively worked like this:

function unwrap<T>(input: Input<T>): T { /* ... */ }
// so, you could then write:
const ipAddresses: Input<string>[] = unwrap(vpc.ipAddresses);

This helped things somewhat. But it still meant that as you processed that array you had to then unwrap each and every element. As our data model has grown, and we’ve tackled more and more complex domains, we started seeing code start to form the programming Pyramid of Doom, where you just kept on going deeper and deeper, unwrapping all the way. We started seeing this happen so often that we wanted to see if we there was anything we could do to make life much nicer for people using our APIs and using these helpers.

Our core goal was to change our ‘unwrap’ function from just peeling away one layer of this onion, to peeling away all layers. So, for example, if you started with an Input<Input<string>[]> you would just get back a plain old string[] that you could use completely naturally in all the ways you’d expect. We were initially uncertain if we could make this work, but thanks to the power of JavaScript and the expressivity of TypeScript, we think we found a great solution here.

We knew that providing the implementation such a function itself was not going to the true hard part. After all, we already had the code to peel back one layer. So peeling back all the layers was more a matter of just having the function call itself recursively when encountering things like arrays (or other objects that themselves contained Inputs).

The real hard part came in when were trying figure out how would we even express this in a type-safe manner TypeScript. After all, what’s the type signature when you effectively want to say:

function unwrap(input: Input<...MaybeHasInputsOrPromisesOrOutputsArbitrarilyDeep...>): ThatThingWithAllTheDeepInputsPromisesAndOutputsErased;

This was a big head scratcher for a while. But after exploring some of the powerful new features that TypeScript has been bringing about in recent releases, we finally found a great way to manage it. In the end, it ends up looking something like this:

type primitive = string | number | boolean | undefined | null;
type Unwrap<T> =
T extends Promise<infer U> ? UnwrapSimple<U> :
T extends Output<infer U> ? UnwrapSimple<U> :
UnwrapSimple<T>;
type UnwrapSimple<T> =
T extends primitive ? T :
T extends Array<infer U> ? UnwrappedArray<U> :
T extends object ? UnwrappedObject<T> :
never;
interface UnwrappedArray<T> extends Array<Unwrap<T>> {}
type UnwrappedObject<T> = {
[P in keyof T]: Unwrap<T[P]>;
};
function unwrap<T>(val: T): Unwrap<T>;

It’s a bit of a doozy containing a bunch of type definitions and the final function declaration that uses them. It makes use of many advanced TypeScript typing features. Including ‘Union Types’ (the ‘primitive’ type), ‘Mapped Types’ (which ‘UnwrappedObject’ is an example of), ‘Conditional Types’ (the types with ? and : in them) and ‘Inferred Types’ (the use of infer U in several places). You can read more about these in the Advanced Types documentation.

Put together, the Unwrap<T> itself is effectively a recursively defined structural pattern matching type. You can see it reference itself once you walk into UnwrappedArray<T> and UnwrappedObject<T> types. As it recurses, it matches the type it has and then ‘peels away’ the wrappings that have been caused by Promise<T> and Output<T>. As it hits Arrays and Objects, it dives deeper, unwrapping along the way. This is all manageable in TypeScript because it does this sort of matching and unwrapping in a lazy fashion, allowing us to express what is effectively an infinitely large pattern-matching type, without it having any trouble understanding it or applying it!

It amazes us how well this works and how expressive TypeScript is here. As you can see here, TypeScript completely understands what’s going on and will give you all the great type-checking support and tooling that we know and love:

Since adding support for this, we’ve been able to start using this new function across our own codebase and we’ve seen things get dramatically simpler. The pyramid-of-doom goes away and code becomes much cleaner and easier to read and maintain. JavaScript helped us write this system easily, but thanks to TypeScript we can express these complex intents and have the safety and understanding to do this sort of thing with confidence.

A special thanks to @DanielRosenwasser over from the TypeScript team for putting up with my constant pestering on how to work through this and deal with some especially thorny areas!

Kubespy, and the lifecycle of a Pod, in 4 images

Alex Clemmer — Tue, 18 Sep 2018 00:00:00 +0000

This post is the first part in a series on the Kubernetes API. Future installments include Part 2 focused on the lifecycle of a Service and Part 3 details how Kubernetes deployments work.

One of the most popular features of the recent v0.15.2 release of Pulumi is fine-grained status updates for Kubernetes resources. On the CLI they look like this:

But wait — how does this work exactly? What is actually happening when you deploy a Pod to a cluster? How does a Service go from uninitialized, to being allocated a public IP address? How often is my Deployment’s status changing?

To answer these questions and others like them, today we’re releasing a small tool, kubespy, purpose-built to help you answer questions like this, by displaying the changes made to a Kubernetes object in real time. For example, in the following gif, we’re running kubespy status v1 Pod nginx to watch the changes to a Pod’s status as it is booted up.

We will spend the rest of this short post using kubespy to take a closer look at what happens to a Pod when it’s deployed to the cluster.

What happens when you boot up a Pod?

kubespy comes with a simple example that deploys nginx using a naked Pod. (The example works with both the Pulumi CLI and kubectl, but we hope you’ll give Pulumi a shot!)

The essential gist is that kubespy will sit and watch for the example Pod to be created or changed; once you deploy the Pod (either pulumi up or kubectl apply), you will see something like the gif above.

If you do this yourself, you will see that there are 4 updates reported by kubespy, each of which individually gives us a fairly clear picture of what Kubernetes is doing internally to try to run the Pod:

First: Acknowledging the Pod definition is written to etcd. The API server receives the Pod definition and begins trying to schedule it.

Second: Scheduling the Pod. The scheduler successfully schedules the Pod to run on a node.

Third: Creating the Pod. The kubelet running on the node receives the Pod definition and begins “creating” the Pod. This involves pulling the container image, adding volume mounts, etc. At the end of all of this, it will attempt to run the container.

Fourth: Marking the Pod as Running. The kubelet has now resolved the image tag to a specific SHA, pulled it to the node, has successfully applied the required configuration, and has successfully started the nginx container. In this case, the Pod contains only one container to run, and once it’s initialized, the whole Pod gets marked running.

Exercises:

Try inducing an error and see what kubespy outputs! Make the image tag nonsense, add a volume that doesn’t exist, and so on.
Try booting up a service! See what kubespy displays under various configurations!

Conclusions, and the subtleties of the Pod API.

This post shows a fairly basic, happy-path use of Pods, and shows that we can kubespy status to see exactly what our containers are up to.

But wait — what does Running mean, precisely? If a Pod is Running, does that mean it’s healthy? If a Pod has multiple containers, and one crashes, does it still get marked Running? Are there any other settings that can change when a Pod is marked Running?

Unfortunately, though the Pod is the atomic compute abstraction of Kubernetes (or perhaps because of this), the answer to these questions turns out to be fairly subtle. But, because Pods are so important, we should endeavor to answer them anyway!

In the next post, we will kubespy to observe some of the subtleties of the Pod lifecycle, and especially the effect of restartPolicy on the semantics of the Pod object. We will also talk about some of the best practices we’ve accumulated, and describe how we solved the problem of repeatable Pod deployments in Pulumi.

If you enjoyed this post, or are curious to see how this lifecycle is baked into the Pulumi CLI, give us a shot! We’d love to have your feedback.

How we use Pulumi to build Pulumi

Luke Hoban — Tue, 26 Jun 2018 00:00:00 +0000

Here at Pulumi we are (perhaps unsurprisingly!) huge fans of using Pulumi to manage our cloud infrastructure and services. We author our infrastructure in strongly-typed programming languages, which allows us to to benefit from rich tooling - documenting and factoring our infrastructure using the same software engineering practices we apply to our application code. This also allows us to create reusable abstractions which accelerate our ability to deliver new features and services, and our ability to standardize and refactor infrastructure patterns across our services with relative ease.

Like other users, we use Pulumi at a variety of levels of abstraction. We use Pulumi for raw infrastructure provisioning, defining the core networking layer for our AWS-based backend infrastructure. And we use Pulumi to define how our application services are deployed into ECS using just a few lines of code. Pulumi hosts and manages static content for www.pulumi.com and get.pulumi.com. We use Pulumi to define the CloudWatch dashboards connected to our infrastructure. And for monitoring, Pulumi defines metrics and notifications/alarms in PagerDuty and Slack.

Best of all, we’ve been able to take things we’ve learned from these use cases, and others we’ve worked with beta users on over the last few months (thank you!), and factor common patterns out into reusable libraries like @pulumi/aws-infra and @pulumi/cloud for ourselves and others to build upon.

In this post, we’ll do a deeper dive into each of these use cases, highlighting unique aspects of how we use Pulumi itself, and some of our engineering processes around how we integrate Pulumi into the rest of our toolchain.

Infrastructure

Our backend infrastructure that supports app.pulumi.com and the Pulumi CLI is fairly typical of a cloud service/application building on AWS. Those who have done this before will be familiar with the combination of AWS VPC, EC2, ASG, ALB, ECS, S3, Aurora, CloudFront and Route53 that powers the core of many AWS-based web applications.

We started off building this infrastructure last year - before we even had completed support for managing AWS resources in Pulumi! As a result, we brought this up initially using CloudFormation. Moving this to Pulumi was an amazing experience. We immediately brought more engineering rigour to the codebase, factoring and decomposing functionality in clearer ways, extracting out useful patterns into named functions, using loops and conditionals instead of copy/paste, and getting completion lists and on-the-fly error checking from within our development tools.

Equally important, authoring our infrastructure in Pulumi gave us the ability to factor out reusable abstractions. We found that our own VPC definition was something that many early customers we worked with also needed - so we factored it out into new awsinfra.Network. We found that standing up an ECS cluster was another big pile of AWS resources that are commonly needed, but can be configured with a much smaller space of configuration values in the common case, so we factored out new awsinfra.Cluster.

Defining infrastructure in Pulumi made it easy to stand up many instances of our service, including per-developer stacks, with just pulumi stack init my-developer-stack and pulumi update. The ability to build reusable components which can be multi-instantiated into the same target cloud is critical here, and something that Pulumi takes care of by default. Within minutes, a developer can have an isolated copy of our full infrastructure stood up, which can be used to test out and quickly iterate on changes. This includes not just the basic infrastructure, but also provisioning of DNS entries in Route53 and creation of custom CloudWatch dashboards (see more on this below).

We have three shared environments - development, staging and production - each an isolated environment with separate configuration. For these shared environments we drive deployments of the Pulumi programs that define the infrastructure above via our CI/CD pipeline. Pulumi can be integrated into any existing CI/CD pipeline or toolchain, via direct use of pulumi preview and pulumi update as part of build and deployment stages. This allows us to reliably deliver a known desired state for each target environment, and to promote changes between environments with just a git merge.

API and App Services

Most of our backend application services are developed in Go, and we deploy them onto our production infrastructure as container images. However, instead of building separate CI pipelines for our container images and the infrastructure they need to function correctly (ECS Service, Application Load Balancer, ECR Repository, etc.), we drive the deployment of both infrastructure and application code via a single Pulumi program. This allows our application code to version along with its supporting infrastructure - when we need to expose a new port for example, we can make the changes to both application code and load balancer as part of a single commit, and promote it through CI as an atomic action. We have found this to significantly simplify the delivery of containerized services on AWS, as well as making it easy to stand up new services into our existing shared networking and cluster infrastructure.

Again, we found this to be a common pattern as we worked with other early customers, and factored out a library to simplify deployment of containerized services into a production deployment on AWS as part of the cloud.Service API in @pulumi/cloud. Here’s an example of using an application service defined in the ./airflow-container folder (with a Dockerfile) into an ECS cluster. This one declaration will provision a scaled-out ECS Service for running the container images, a ECR repository to store the image, optionally a load balancer if ports were exposed, and each pulumi update will build the container image, push it to the private repository, and turn-over each service task to use the new image. All of that workflow complexity is abstracted away behind a simple API and a single CLI command.

Metrics, Alarms and Dashboards

We also use Pulumi to define a few less traditional parts of our infrastructure - including our metrics, dashboards and alarming infrastructure on top of AWS CloudWatch.

Defining and versioning this along with our infrastructure has significant benefits: new features are built with metrics, dashboards and alarms lit up from the get-go, developers get a full set of dashboards as part of their personal isolated development environments, and promotions from development to staging to production can verify not just physical infrastructure, but also that alarms and dashboards are behaving as expected prior to pushing code to production. Here’s a sample of our CloudWatch dashboards:

We build up AWS resources representing CloudWatch dashboards using a compositional framework shown below. As this example shows, bringing software engineering approaches to infrastructure as code can lead to new approaches to how code is authored and factored - and ultimately to the creation of higher level frameworks for building domain-specific cloud infrastructure - in this case, CloudWatch dashboards.

The same metrics that power these dashboards are also used to power notifications and alarms. In these cases, we want to define both the infrastructure for the alarm - the SNS topics and registration of those topics with CloudWatch - as well as the small piece of code that fires off Pagerduty and Slack notifications on notifications being pushed to that SNS topic - all in one codebase. This ensures that we can version these together, and that we can deploy both consistently as part of updates. To do this, we include the event handlers for these infrastructure events directly inline in our Pulumi programs. Note in the code snippet below how we define the infrastructure for a Topic, and then conditionally wire up event handlers on that topic, including the code to execute at runtime on these events directly in our Pulumi program. This code can even reach out and reference configuration used to provision our Pulumi deployment - key to enabling this to be deployed seamlessly into a variety of different development environments.

Static Content

We serve three properties as static content from S3, each described using Pulumi. One of these is https://get.pulumi.com which hosts downloads of our pulumi CLI installer. It’s a very simple program, and even better - it’s open source on GitHub. It combines AWS S3, CloudFront and Route53.

This infrastructure is deployed and managed using the Pulumi Service at app.pulumi.com, allowing us to track the history of deployments and changes to the infrastructure and quickly jump in to the resources defined by this stack in the AWS console.

Conclusion

Treating cloud infrastructure as software has changed how we think about infrastructure engineering. Using Pulumi for both traditional cloud infrastructure as well as to get application code and supporting service infrastructure into production provides a simple workflow and the ability to create simple high-level APIs to make new services easy to define and deploy.

Try out Pulumi yourself today!

Pulumi Blog: Engineering

How We Eliminated Long-Lived CI Secrets Across 70+ Repos

The problem with static CI secrets

Our approach: zero static secrets

What the change looks like

Scale: 70+ repos, zero static secrets

Auditability and centralized control

What happens if a GitHub Action is compromised

Get started

How We Built a Distributed Work Scheduling System for Pulumi Cloud

Where we started

Why not use an off-the-shelf queue?

Design constraints

The background activity

The state machine

Leases: distributed execution without coordination

How it works

Routing work to the right runner pool

Dependencies and multi-step workflows

Two execution modes, one interface

Putting it all together

Retries and scheduling

Observability

What we learned

Wrapping it up

How We Built Platybot: An AI-Powered Analytics Assistant

The problem with throwing AI at your Data Warehouse

Why we built a semantic layer first

Adding the AI layer

Meeting users where they are

The web UI

Slack

MCP: Making Platybot usable by Agents

Deploying with Pulumi

Results

What we learned

Speeding up Pulumi Operations by up to 20x

Benchmarks

Introduction to snapshotting

Why is it slow?

Fast, but lacking data integrity?

Enter journaling

Rollout

What’s next

A Recipe for a Better AI-based Code Generator

Using RAG for code generation

The anatomy of Pulumi Copilot RAG

Assessing search quality using recall and precision

Practical concerns

Semantic search with vector embeddings

Full text vs semantic search

Pruning the results

Prompt generation

Self-debugging

From kitchen to table: testing what works

Building confidence through testing

Monitoring quality in production

Wrapping up

AI Engineering Lessons from Building Pulumi Copilot

Engineering for Reality: Prompt Engineering vs Software Engineering

Copilot in Action: Real-World Dogfooding

Skillful Slicing: Modular Mastery

Debug Dispatch

The Illusion of Correctness

Testing the Untestable: Validating LLM Outputs

LLMs think like humans (sort of)

Next-level IaC: Bridging the Declarative Gap

Pulumi’s architecture and flow of operations

Inputs and Outputs as asynchronous futures

Realizing values using the apply function

Example: Using apply to Base64 encode an Output

Next steps

Pulumi Cloud

Details of the Pulumi Outage on October 6, 2023

Converting Bicep code to Pulumi

Installing and using the plugin

Early days for the converter

How does it work?

Bonus converter: ARM to Pulumi

Building your own Pulumi language converter

Realizing values using the `apply` function

Example: Using `apply` to Base64 encode an Output

Setting up `pkg`

2 - Set `bin` in `package.json`