Pulumi Blog: Insights

Connect Your Cloud Accounts to Pulumi in Minutes

Levi Blackstone — Wed, 15 Jul 2026 00:00:00 +0000

Pulumi Insights gives you visibility and governance across your entire cloud footprint, but that visibility is only as complete as the set of accounts you’ve connected. Until now, connecting an account meant repeating a manual setup for each one: OIDC configuration, hand-written Pulumi ESC environments, and per-account scan and policy setup. For an organization with dozens or hundreds of AWS accounts, Azure subscriptions, or Google Cloud projects, that per-account friction was the biggest obstacle to getting a complete picture. Today, the new Connect cloud accounts wizard removes it: discover every account in your cloud organization, select the ones you want, and onboard them all in a single guided flow.

From hours of setup to a single flow

The time savings are real: connecting a batch of accounts across AWS, Azure, and Google Cloud takes about three minutes end to end. Setting up those same accounts with the manual per-account workflow would take hours to days.

The wizard handles the entire onboarding lifecycle for AWS, Microsoft Azure, and Google Cloud:

Bulk discovery: Authenticate once with your cloud organization and the wizard lists every account, subscription, or project you can access. Accounts that are already connected are recognized and skipped.
Automatic OIDC setup: The recommended flows configure each account with short-lived credentials based on OpenID Connect (OIDC) and workload identity federation. No long-lived cloud secrets are stored in Pulumi Cloud.
ESC environments, created for you: The wizard generates Pulumi ESC environments that follow the best practices from the manual OIDC guides — environments that previously had to be authored one by one.
Scans and policies from day one: By default, scheduled discovery scans and a pre-built policy pack are applied to every account as part of setup: the Pulumi Best Practices pack on the Team and Enterprise editions, or a compliance pack (CIS, or NIST 800-53 for AWS and Google Cloud) on Business Critical.

How it works

The wizard walks you through five steps: choose a provider, authenticate, select accounts, configure discovery and policy, and review the results. You can open it from Management > Accounts in the Pulumi Cloud console, or from the Get to know Pulumi card on the home dashboard.

Authentication uses each provider’s native federation mechanism: IAM Identity Center (SSO) for AWS, Microsoft Entra ID workload identity federation for Azure, and Workload Identity Federation for Google Cloud. After you sign in, the wizard discovers the accounts in your organization and pre-selects everything that isn’t already connected. You can search, select all, or toggle individual accounts.

Choose the right access level for your security posture

Not every team wants to grant write access on day one. The wizard offers two access levels, and you can set them per account:

Build & Manage (read and write): Enables the full platform: Pulumi Neo, infrastructure as code, deployments, and policies that remediate issues automatically (Business Critical).
Discovery & Policy (read-only): Limited to discovery scanning and inventory. Pulumi can’t modify your infrastructure.

If your security review requires it, start everything read-only and raise access for specific accounts later.

Everything set up, nothing hidden

When setup completes, the summary shows exactly what was created: the ESC environments grouped by access level, the state of discovery scanning, and the policy pack applied. If any account fails to connect, the summary lists it with the specific error so you can fix the cause and re-run the wizard. Accounts that connected successfully are skipped on retry.

For security reviewers, the docs include a full accounting of what the wizard creates in your cloud provider and in Pulumi Cloud: the IAM roles, app registrations, and service accounts on the cloud side, and the ESC environments and Insights accounts on the Pulumi side.

Get started

The Connect cloud accounts wizard is available now for all Pulumi Cloud organizations. To connect your first accounts:

Navigate to Management > Accounts in the Pulumi Cloud console and select Connect cloud accounts.
Follow the guided flow for AWS, Azure, or Google Cloud.
Explore your discovered resources and policy findings.

To learn more:

Connect cloud accounts documentation — prerequisites, each wizard step in detail, and troubleshooting
Insights & Governance overview — full documentation for discovery and policy capabilities
Pulumi ESC — how the generated environments manage cloud credentials with OIDC

Scan AWS GovCloud and more partitions with Pulumi Insights

Alejandro Cotroneo — Tue, 14 Apr 2026 09:00:00 +0000

Pulumi Insights account scanning now supports every AWS partition. If your workloads run in GovCloud, China, the European Sovereign Cloud, or one of the ISO intelligence-community clouds, you can get the same resource discovery, cross-account search, and AI-assisted insights that commercial accounts already have.

Supported partitions

AWS Standard (Commercial)
AWS GovCloud (US)
AWS ISO (US)
AWS ISOB (US)
AWS ISOF (US)
AWS ISOE (Europe)
AWS European Sovereign Cloud
AWS China

You can also exclude specific regions from discovery — useful when regions are disabled by SCPs or fall outside an audit’s scope.

Set it up

In the Pulumi Cloud console:

Go to Accounts → Create account.
Select AWS as the provider.
Under Add your configuration, pick the target partition.
Supply credentials via a Pulumi ESC environment. The OIDC trust policy uses the partition-appropriate ARN prefix (arn:aws-us-gov:, arn:aws-cn:, etc.).

For IAM and ESC setup, see the Insights accounts docs. Log in to Pulumi Cloud to get started.

From Kubernetes Gatekeeper to Full-Stack Governance with OPA

Levi Blackstone — Thu, 19 Mar 2026 00:00:00 +0000

Pulumi’s OPA (Open Policy Agent) support is now stable. The v1.1.0 release of pulumi-policy-opa makes OPA/Rego a first-class policy language for Pulumi with full feature parity alongside the native TypeScript and Python policy SDKs. Write Rego policies that validate any resource Pulumi manages, across AWS, Azure, GCP, Kubernetes, and the rest of the provider ecosystem. If you already have Kubernetes Gatekeeper constraint templates, a new compatibility mode lets you drop those .rego files directly into a Pulumi policy pack and enforce them against your Kubernetes resources without modification.

What’s in the stable release

OPA/Rego is now fully supported as a policy language for Pulumi Insights, with the same capabilities as the TypeScript and Python SDKs:

Resource and stack-level policies: Validate individual resources with deny and warn rules, or evaluate your entire stack at once with stack_deny and stack_warn for cross-resource checks like relationship validation and resource count limits.
Enforcement levels: Control how violations are handled. mandatory blocks deployments, advisory surfaces warnings, and disabled turns rules off without removing them. Enforcement levels can be overridden per policy without modifying Rego source.
Policy configuration: Pass custom parameters to policies via configuration files, with optional JSON schema validation. Configuration values are accessible in Rego as data.config.<policy_name>.<key>.
OPA metadata annotations: Use standard OPA # METADATA comments to provide titles, descriptions, and messages for your policies. These populate the policy metadata displayed in Pulumi Cloud.
Preventative and audit evaluation: OPA policies work with both preventative enforcement during pulumi up and audit policy scans for continuous compliance monitoring.

You can choose whichever language best fits your team. Organizations already using OPA across their toolchain can standardize on Rego for Pulumi policies, while teams preferring TypeScript or Python can continue to use those. All three languages work side by side in the same policy groups.

Kubernetes Gatekeeper compatibility

The headline feature of this release is native support for Kubernetes Gatekeeper constraint template rules. If you’re running Gatekeeper as an admission controller in your clusters, you likely have a library of .rego policies that enforce security and operational standards at admission time. With v1.1.0, those same rules can now run as Pulumi policies, catching violations during pulumi preview before resources ever reach the cluster.

To enable Gatekeeper compatibility, set inputFormat: kubernetes-admission in your PulumiPolicy.yaml:

description: Kubernetes Gatekeeper Policy Pack
runtime: opa
inputFormat: kubernetes-admission

With this setting, Pulumi automatically wraps Kubernetes resources in the Gatekeeper AdmissionReview structure (input.review.object, input.review.kind, etc.), so your existing rules work without modification. Non-Kubernetes resources are silently skipped.

Here’s an example that reuses standard Gatekeeper-style rules, requiring an app label and prohibiting the latest image tag:

package gatekeeper

import rego.v1

# METADATA
# title: Require App Label
# description: All Kubernetes resources must have an "app" label.
violation contains {"msg": msg} if {
 not input.review.object.metadata.labels["app"]
 msg := sprintf("%s '%s' is missing required label: app",
 [input.review.kind.kind, input.review.name])
}

# METADATA
# title: Disallow Latest Tag
# description: Container images must not use the "latest" tag.
deny contains msg if {
 container := input.review.object.spec.template.spec.containers[_]
 endswith(container.image, ":latest")
 msg := sprintf("container '%s' uses the 'latest' tag -- pin to a specific version",
 [container.name])
}

These rules are identical to what you’d write for Gatekeeper. Both rule head formats are supported and can coexist: the violation[{"msg": msg}] map format and the deny[msg] string format. Per-policy configuration via input.parameters also works as expected. You can take a .rego file from your Gatekeeper constraint templates, drop it into a Pulumi policy pack, and publish it to Pulumi Cloud to enforce automatically across your stacks.

This shifts policy enforcement left. Instead of waiting for the Kubernetes API server to reject a resource at admission time, you catch the violation during pulumi preview, before anything is deployed.

Walkthrough: Reusing policies from the gatekeeper-library

The OPA Gatekeeper Library is a community-maintained collection of constraint templates covering common Kubernetes guardrails like pod security, image provenance, and resource limits. You can use these policies directly with Pulumi. Here’s an end-to-end example using the allowedrepos policy to restrict which container image registries your deployments can use.

Create a new Kubernetes OPA policy pack:
```
pulumi policy new kubernetes-opa
```

Copy the Rego source from gatekeeper-library into your policy pack as allowedrepos.rego. No modifications are needed:

package k8sallowedrepos

violation[{"msg": msg}] {
 container := input.review.object.spec.containers[_]
 not strings.any_prefix_match(container.image, input.parameters.repos)
 msg := sprintf("container <%v> has an invalid image repo <%v>, allowed repos are %v",
 [container.name, container.image, input.parameters.repos])
}

violation[{"msg": msg}] {
 container := input.review.object.spec.initContainers[_]
 not strings.any_prefix_match(container.image, input.parameters.repos)
 msg := sprintf("initContainer <%v> has an invalid image repo <%v>, allowed repos are %v",
 [container.name, container.image, input.parameters.repos])
}

violation[{"msg": msg}] {
 container := input.review.object.spec.ephemeralContainers[_]
 not strings.any_prefix_match(container.image, input.parameters.repos)
 msg := sprintf("ephemeralContainer <%v> has an invalid image repo <%v>, allowed repos are %v",
 [container.name, container.image, input.parameters.repos])
}

Verify that your PulumiPolicy.yaml has Gatekeeper compatibility enabled:

description: Kubernetes Gatekeeper Policy Pack
runtime: opa
inputFormat: kubernetes-admission

Configure the allowed registries. Create a policy-config.json file to pass the repos parameter:

{
 "k8sallowedrepos": {
 "repos": ["gcr.io/my-company/", "docker.io/library/"]
 }
}

Test the policy locally against a stack:
```
pulumi preview --policy-pack . --policy-pack-config policy-config.json
```
Any Kubernetes deployment using an image outside the allowed registries will produce a violation at preview time, before it reaches the cluster.
Publish the pack and add it to a policy group to enforce it across your organization:
```
pulumi policy publish
```

The same approach works for any policy in the gatekeeper-library: containerlimits, requiredlabels, disallowedtags, and others. Copy the Rego, configure parameters, and publish.

Part of the Pulumi Insights governance story

OPA policy support is part of the broader Pulumi Insights governance platform. Insights gives you visibility and compliance across your entire cloud footprint, and OPA policies plug directly into that:

Audit policy scans continuously evaluate OPA policies against your Pulumi stacks and discovered cloud resources, providing a compliance baseline without redeploying anything.
Self-hosted execution lets you run policy evaluations on your own infrastructure using customer-managed workflow runners, keeping credentials and data within your network.
Pre-built compliance packs for CIS, NIST, PCI DSS, and other frameworks are available alongside your custom OPA policies in the same policy groups.

Whether you’re enforcing policy at deployment time, scanning existing infrastructure for drift, or running continuous compliance checks, OPA policies are a native participant.

Frequently asked questions

Do I need to modify my existing Gatekeeper `.rego` files?

No. Set inputFormat: kubernetes-admission in your PulumiPolicy.yaml and your existing Gatekeeper constraint template rules work as-is. Pulumi handles the AdmissionReview wrapping automatically.

What happens with non-Kubernetes resources?

When using inputFormat: kubernetes-admission, non-Kubernetes resources are silently skipped during evaluation. Your Gatekeeper rules only run against Kubernetes resources.

Do I need OPA installed locally?

No. The pulumi-policy-opa analyzer plugin embeds the OPA evaluation engine and is installed automatically by the Pulumi CLI (v3.227.0+). The standalone OPA CLI is only needed if you want to run opa test against your policies independently.

When should I use OPA vs. TypeScript or Python for policies?

If your team already writes Rego for other tools like Gatekeeper, writing Pulumi policies in Rego keeps your policy language consistent. If your team is more comfortable with general-purpose languages or needs auto-remediation, use the TypeScript or Python SDKs.

Gatekeeper constraint templates can be reused directly via the kubernetes-admission input format, but other OPA integrations use different input structures, so those policies would need to be adapted to Pulumi’s resource model. All three languages work together in the same policy groups.

Get started

Templates are available for kubernetes-opa, aws-opa, azure-opa, and gcp-opa via pulumi policy new. For more details, see the policy authoring guide and the Policy as Code overview.

Get started with OPA policies

github.com/pulumi/pulumi-policy-opa

Run Pulumi Insights on Your Own Infrastructure

Levi Blackstone — Mon, 02 Mar 2026 00:06:00 -0700

Pulumi Insights gives you visibility and governance across your entire cloud footprint: discovery scans catalog every resource in your cloud accounts, and policy evaluations continuously enforce compliance against those resources. Until now, Insights workflows ran exclusively on Pulumi-hosted infrastructure. That works well for many teams, but enterprises with strict data residency requirements, private network constraints, or regulatory obligations need to run this work in their own environments. Today, Pulumi Insights supports customer-managed workflow runners for both SaaS Pulumi Cloud and self-hosted Pulumi Cloud installations.

Insights at a glance

Insights provides two complementary capabilities that together form a governance lifecycle for your cloud infrastructure.

Discovery scans cloud accounts across AWS, Azure, GCP, and more to catalog every resource regardless of how it was provisioned: Pulumi, Terraform, CloudFormation, or manual creation. Once cataloged, you can search, filter, group, and export your resource data. You can also import unmanaged resources into Pulumi to bring them under IaC management.

Policy enforces compliance with policy-as-code written in TypeScript or Python. Pulumi ships pre-built compliance packs for CIS, NIST, PCI DSS, HITRUST, and other frameworks so you can start evaluating without writing any code. Audit policy groups continuously evaluate all discovered resources and IaC stacks, while preventative policies block non-compliant deployments before they reach production.

This enables you to map out your cloud estate, evaluate compliance, and then remediate any issues uncovered by policy.

Why self-hosted?

Running Insights on your own infrastructure with customer-managed workflow runners gives you:

Data residency: Scan execution and policy evaluation run entirely within your private network.
Private infrastructure access: Scan resources in VPCs and environments that are not accessible from the public internet.
Compliance: Cloud provider credentials can stay internal to your network, meeting regulatory requirements for credential handling.
Flexible hosting: Run workflow runners on any environment that meets your needs, including Linux, macOS, Docker, and Kubernetes.

How it works

Customer-managed workflow runners are lightweight agents that poll Pulumi Cloud for pending work, execute it locally, and report results back. You can configure runners to handle specific workflow types: discovery scans, policy evaluations, deployments, or all three.

This works identically whether you use SaaS Pulumi Cloud or a self-hosted installation. The runner communicates with the Pulumi Cloud API over HTTPS, so no inbound connectivity is required, making it well suited to run in restricted network environments.

Under the hood, this is powered by a distributed work scheduling system that routes activities to the right runner pool, handles lease-based execution, and recovers automatically from failures. For a deep dive on the architecture, see How We Built a Distributed Work Scheduling System for Pulumi Cloud.

If your team already uses customer-managed workflow runners for Pulumi Deployments, your existing runner pools can handle Insights workflows with no additional infrastructure.

Get started

Self-hosted Insights is available on the Business Critical edition of Pulumi Cloud. To learn more or get set up:

Self-hosted Insights documentation — configuration and setup for discovery scans and audit policy evaluations on your own infrastructure
Customer-managed workflow runners — runner installation, configuration reference, and pool management
Insights & Governance overview — full documentation for discovery and policy capabilities
Contact sales to enable self-hosted Insights for your organization

How We Built a Distributed Work Scheduling System for Pulumi Cloud

Levi Blackstone — Thu, 26 Feb 2026 00:00:00 +0000

Pulumi Cloud orchestrates a growing number of workflow types: Deployments, Insights discovery scans, and policy evaluations. Some of that work runs on Pulumi’s infrastructure, and some of it runs on yours via customer-managed workflow runners. We needed a scheduling system that could handle all of these workflow types reliably across both environments. In this post, we’ll take a look at the system we built.

Where we started

For our first workflow integration, Deployments, scheduling wasn’t too complicated. A deployment was queued, a worker picked it up, and it ran. The queue was purpose-built for deployments, and it worked well for that single use case. Over time, we added more sophisticated logic to handle retries, ordering, rate limiting, observability, and more.

With the launch of Insights, the number of workflow types grew. Now Pulumi Cloud manages discovery scans to catalog cloud resources and runs audit policy evaluations to continuously verify compliance. While these workflows share similarities, each type needed its own scheduling, retry logic, and failure handling.

Later we added the option for customers to run workflows on their own infrastructure using customer-managed workflow runners. As the complexity of these requirements grew, we knew that our initial approach for Deployments wasn’t going to scale. We needed a single system that could schedule any type of work, route it to the right place, and handle the messy reality of distributed execution: crashes, network failures, rate limits, and retries.

We call this the background activity system.

Why not use an off-the-shelf queue?

Why build this instead of using Amazon SQS, RabbitMQ, or one of the many existing queue libraries? We considered these options but chose to build our own for a few reasons.

Pulumi Cloud supports self-hosted installations, including air-gapped environments. We intentionally minimize external dependencies so that self-hosted customers don’t have to stand up additional infrastructure. A system built on an external queue works fine for our hosted service, but it means self-hosted customers would need to provide a compatible backend. By building on top of the database we already require, we avoid adding another system to maintain.

More importantly, queueing is only part of the problem. What we actually need is scheduling with durability. This means ensuring that remote workers don’t lose activities on restart, priority so that urgent work gets compute resources first, constraints like “only so many scans per org at a time,” structured logging for observability, and checkpointing so that long-running operations can resume after a failure.

These features can be layered onto a generic queue library but can require more code than implementing them directly. For example, priority queues are often implemented with multiple ranked queues, but this breaks single-activity-at-a-time constraints. A second queue wouldn’t see a job already running in the first one. There’s no way for producers in a distributed system to coordinate across the queues without support in the queuing system itself.

Capacity management is another area where generic queues fall short. Distributed systems need to respond dynamically to slowdowns, network interruptions, and rate limits from downstream services. These are common low-level details that every workflow type needs, and building them into the scheduling layer means individual handlers don’t have to solve them independently.

We also need structured logging that works everywhere, including on customer-managed runners behind firewalls where centralized logging services aren’t accessible.

Building this ourselves gave us a system that works with existing infrastructure and handles these requirements natively.

Design constraints

With that context, here are some of the constraints that shaped the design:

Pull-only agents. Customer-managed workflow runners live behind NATs, corporate proxies, and air-gapped networks. They can’t accept inbound connections, so all communication has to be agent-initiated.
Mixed execution environments. The same system needs to work for Pulumi-hosted workers (with direct access to internal systems) and customer-managed runners (communicating entirely over REST). We didn’t want to maintain two separate code paths.
Different workflow types. Deployments, Insights scans, and audit policy evaluations have different payloads and execution semantics, but they all need the same scheduling guarantees: exactly-once execution, automatic retries, failure recovery, and observability.
Automatic fault tolerance. Agents crash, networks drop, and machines get recycled by autoscalers. The system needs to detect these failures and recover without needing a person to step in.
Extensibility. We knew we’d keep adding workflow types. Adding a new one should mean writing a handler and registering it, not building new infrastructure.

The background activity

At the center of the system is the background activity, a persistent, typed work unit. Each activity includes:

A type discriminator that identifies what kind of work it represents (e.g., “insights-discovery” or “policy-evaluation”)
A payload specific to that type, containing whatever data the handler needs
A routing context that determines which runner pool should execute it
Scheduling metadata like priority, activation time, and retry configuration
A status tracking where the activity is in its lifecycle

The type discriminator makes this system polymorphic. The scheduling engine doesn’t need to know what’s inside the payload. It moves activities through their lifecycle and delegates the actual work to a type-specific handler.

The state machine

Every activity follows the same lifecycle regardless of type:

---
config:
flowchart:
curve: linear
---
graph LR
Start(( )) -->|Created| Ready
Ready -->|Leased| Pending
Pending -->|Started| Executing
Executing -->|Success| Completed
Completed --> End(( ))
Executing -->|Error| Failed
Failed --> End
Executing -->|Canceled| Canceled
Canceled --> End
Executing -->|Dependencies| Waiting
Waiting -->|Unblocked| Ready
Pending -->|Lease expired| Restarting
Executing -->|Lease expired| Restarting
Restarting -->|Re-lease| Ready
style Start fill:#000,stroke:#000,color:#000
style End fill:#000,stroke:#000,color:#000

The states fall into two groups:

Running states (work is in flight or can be resumed):

Ready: queued and eligible to be claimed by a worker
Pending: claimed by a worker, execution about to start
Executing: actively running on a worker
Waiting: parked, blocked on one or more dependency activities
Restarting: recovered after a worker failure, ready to be re-claimed

Terminal states (work is done):

Completed, Failed, Canceled

New workflow types get these features automatically: scheduling, retries, dependency management, and observability.

Leases: distributed execution without coordination

A central challenge of any distributed work queue is preventing double-execution. If two agents try to execute the same activity simultaneously, you get duplicate work and data corruption. A central coordinator can solve this, but it becomes a single point of failure.

We use lease-based optimistic concurrency instead. This is a well-known pattern, adapted here for long-running, stateful workflows.

How it works

When an agent is ready for new work, it asks the service to lease an activity. The service atomically selects the highest-priority ready activity, assigns a lease token with an expiration time, and transitions the activity to Pending. No other agent can claim the same activity.

sequenceDiagram
participant Agent
participant Service
Agent->>Service: Poll for work
Note right of Service: Select highest-priority<br/>Ready activity
Note right of Service: Atomically set lease<br/>token + expiration
Service->>Agent: Lease (token, expiration)
Agent->>Service: Begin execution
Note right of Service: Transition to Executing
Note over Agent: Work in progress...
Agent->>Service: Renew lease
Service->>Agent: New expiration
Note over Agent: Work continues...
Agent->>Service: Complete (token, result)
Note right of Service: Transition to Completed<br/>Archive activity
Service->>Agent: Acknowledged

While executing, the agent periodically renews its lease to signal that it’s still working. If the agent crashes, loses network connectivity, or is terminated, it stops renewing. Once the lease expires, the service transitions the activity to Restarting, making it available for another agent to claim.

sequenceDiagram
participant A as Agent A
participant S as Service
participant B as Agent B
A->>S: Lease activity
S->>A: Token + expiration
Note over A: Executing...
A--xS: Agent A crashes
Note right of S: Lease expires
Note right of S: Transition → Restarting
B->>S: Poll for work
Note right of S: Lease to Agent B<br/>Transition → Pending
S->>B: Token + expiration
Note over B: Agent B continues execution

The service doesn’t need to explicitly coordinate between workers because leases are acquired using atomic database operations. The lease expiration is the failure detector; if a lease expires, then the work needs to be rescheduled.

Routing work to the right runner pool

Pulumi Cloud supports multiple workflow runner pools. An organization might have one pool for production in us-east-1, another for staging in eu-west-1, and use Pulumi-hosted runners for development. Work needs to reach the right pool.

Each activity carries a routing context that identifies which runner pool should execute it. When a runner polls for work, it filters by its own pool identifier so that it only sees activities meant for it.

We use prefix matching for this filtering. A runner matches activities whose context starts with its pool’s identifier. This means the service can use hierarchical contexts (e.g., pool-abc/insights/scan-123) and runners will still match on the pool prefix. Cleanup is also straightforward; when a runner pool is deleted, all activities with that context prefix are bulk-canceled.

This routing mechanism works the same way regardless of workflow type, and adding a new workflow type doesn’t require changes to the routing layer.

Dependencies and multi-step workflows

Some workflows are naturally multi-step. An Insights discovery scan might discover resources that then need policy evaluation. Rather than building a separate orchestration engine, we built dependency management into the activity system.

An activity can declare a dependency set: a list of other activities that must complete before it can run. A dependent activity enters the Waiting state when created. As its dependencies complete, the system checks whether all prerequisites are satisfied. When the last one finishes, the waiting activity transitions to Ready and enters the scheduling queue.

graph TD
A["Insights Discovery<br/><b>Executing</b>"] -->|depends on| B["Policy Evaluation<br/><b>Waiting</b>"]
A -->|completes| C["Insights Discovery<br/><b>Completed</b>"]
C -->|triggers| D["Policy Evaluation<br/><b>Ready</b><br/>(auto-scheduled)"]

This gives us a lightweight DAG of work without requiring a separate workflow engine. Dependent activities get the same guarantees as any other activity: lease-based execution, automatic recovery, and observability.

Two execution modes, one interface

This is where the design really pays off for customer-managed runners. The system supports two execution modes:

Direct mode runs in-process alongside the Pulumi Cloud service. Workers have low-latency access to internal systems and can process activities with minimal overhead. This is what Pulumi-hosted runners use.
Remote mode communicates over REST APIs. The runner polls for activities, leases them, executes work locally, and reports results back over HTTP. This is what customer-managed runners use. No database access, no internal network access, no inbound connectivity required.

Both modes share the same handler interface so that a workflow handler doesn’t need to know where it’s running. Whether it’s running on Pulumi’s hosted infrastructure or on a customer’s Kubernetes cluster, the handler simply processes the payload and reports a result.

Putting it all together

Let’s walk through a concrete example. A user wants to run an Insights discovery scan on an AWS account using a customer-managed workflow runner.

sequenceDiagram
participant User
participant PC as Pulumi Cloud
participant Runner as Workflow Runner<br/>(Customer Infrastructure)
User->>PC: 1. Configure Insights scan<br/>for runner pool
Note right of PC: 2. Create background activity<br/>type: insights-discovery<br/>context: runner-pool-xyz<br/>status: Ready
Runner->>PC: 3. Poll for work (filtered by pool)
PC->>Runner: 4. Lease activity (token + expiration)
Runner->>PC: 5. Initialize workflow
PC->>Runner: Return cloud credentials + job token
Note over Runner: 6. Execute scan locally<br/>(talks directly to cloud APIs —<br/>credentials are used only on<br/>the runner)
Runner->>PC: 7. Renew lease
Note right of PC: Extend expiration
Runner->>PC: 8. Report completion
Note right of PC: 9. Mark completed<br/>Archive activity
Note right of PC: 10. Unblock dependent activities<br/>(e.g., policy evaluation)

A user configures an AWS account for Insights scanning in Pulumi Cloud and assigns it to a workflow runner pool.
Pulumi Cloud creates a background activity with the type set to insights discovery, the routing context set to the runner pool, and the payload containing the account configuration.
A customer-managed workflow runner polling that pool detects new work.
The runner leases the activity, acquiring an exclusive lock via the lease token.
The runner initializes the workflow, receiving any required cloud provider credentials (e.g., resolved from Pulumi ESC (Environments, Secrets, and Configuration)) and a job token from Pulumi Cloud.
The runner executes the scan locally on the customer’s infrastructure, talking directly to the cloud provider APIs.
During execution, the runner periodically renews its lease to signal liveness.
The scan completes, and the runner reports the result back to Pulumi Cloud.
The service marks the activity as completed and archives it.
If a dependent policy evaluation activity was waiting on this scan, it automatically transitions to Ready and enters the scheduling queue, where another runner in the pool can pick it up.

This flow works the same way whether the runner is hosted by Pulumi or by the customer. The only difference is whether the execution mode is direct or remote.

Retries and scheduling

Failures are expected in distributed systems. The background activity system handles them at several levels:

Lease expiration covers hard failures like agent crashes, network partitions, and machine terminations. If a lease expires, the activity moves to Restarting, and is available for another agent to pick up.
Handler-controlled retries cover soft failures like transient API errors and rate limits. A handler can request a reschedule with a delay, putting the activity back in Ready with a future activation time.
Automatic retries provide a configurable retry budget per activity. Each activity can specify how many times it should be retried and the delay between attempts, preventing runaway retry loops.
Priority scheduling ensures urgent work gets processed first. Higher-priority activities are leased before lower-priority ones, even if the lower-priority activity has been waiting longer.
Lease renewal during slowdowns keeps the activity alive without blocking other work, even if a downstream service is slow. The agent continues renewing its lease while it waits, and the scheduler remains free to assign other activities to other agents.

Observability

Every activity generates a structured log of its execution, including timestamps, severity levels, and code context. Logs are stored with the activity record and are accessible via an API and admin tooling.

This is especially useful for customer-managed runners, where the service can’t directly observe the execution environment. The structured log gives operators visibility into the execution context, even when the runner is behind a firewall. Handlers can also use these logs as a progress journal, encoding checkpoints that allow a restarted activity to pick up where it left off rather than starting from scratch.

Retention policies are configurable per organization and per workflow type. Completed activities can be retained for auditing or purged to manage storage, and failed activities are typically retained longer for debugging.

What we learned

A generic system pays off quickly. Our initial instinct was to build targeted solutions for each workflow type. Investing in a generic activity system required more upfront design work, but now adding a new workflow type requires a fraction of the effort it would take otherwise. New workflows ship with full scheduling, retry, and observability support from day one.

Leases handle many failure modes. We evaluated several approaches for distributed work coordination, including message queues with explicit acknowledgment and coordinator-based assignment. The lease model works well because all failure modes are handled through timeouts. If an agent is running as expected, it renews. If it isn’t, the lease expires.

Keeping the execution paths symmetric requires discipline. Making the hosted and self-hosted paths share the same handler interface was a deliberate choice. It would be easy to add shortcuts for the hosted path that bypass the remote API, but resisting that temptation means that features work for both cases automatically.

The hard part isn’t running the work. Running a scan or a deployment is straightforward once you have the right credentials. The real complexity is in everything around the execution: scheduling, routing, leasing, retrying, resolving dependencies, and cleaning up. These operational concerns aren’t visible to users, but they are essential to providing a reliable experience.

Wrapping it up

Today this system powers deployments, Insights discovery scans, and policy evaluations across both Pulumi Cloud and customer-managed infrastructure. The architecture is general enough that every new workflow type we add inherits the full scheduling, routing, retry, and observability stack without additional plumbing.

If you’re interested in running workflows on your own infrastructure, check out customer-managed workflow runners. To see how Insights can help you understand and manage your cloud infrastructure, get started with Pulumi Insights.

Unified Resources in Pulumi Cloud

Pulumi Insights Team — Mon, 08 Sep 2025 09:00:00 -0800

We’re excited to announce unified resources in Pulumi Cloud. This powerful new feature automatically consolidates resources from multiple sources into single, comprehensive views. When the same AWS S3 bucket, Azure VM, or Google Cloud database appears in both your IaC stacks and Pulumi Insights, you’ll now see one unified entry instead of duplicates cluttering your search results.

Why We Built This

Cloud resources naturally exist across multiple contexts in modern organizations. The same S3 bucket might be imported into a Pulumi stack, discovered by Pulumi Insights, and referenced in multiple projects. Previously, this created fragmented views with duplicate search results that made it difficult to understand your true infrastructure footprint.

Teams told us they needed a way to see the complete picture of each resource without manually correlating information across different sources. They wanted accurate resource counts, clear visibility into management states, and faster access to all relevant information when troubleshooting issues.

Unified resources solve these challenges by automatically detecting when the same cloud resource appears across multiple sources and consolidating them intelligently.

How It Works

When you search for resources in Pulumi Cloud, our system now automatically identifies duplicates across all sources including:

Resources managed by Pulumi IaC
Resources imported into Pulumi stacks
Resources discovered through Pulumi Insights including resources managed outside of Pulumi IaC

Resources with multiple sources are marked with a distinctive icon in search results. This visual indicator lets you immediately identify which resources have rich, multi-source information available.

The consolidation happens automatically. Your existing workflows continue unchanged, but now with cleaner, more actionable results.

The New Unified Resource Details Page

The centerpiece of this release is the completely redesigned resource details page. Previously, clicking on a resource from different sources led to separate, disconnected views. Now, every resource has a single, unified details page that brings together all available information.

Complete Source Visibility

The new details page shows every source where a resource appears:

Pulumi stacks: See which stacks manage or import this resource
Insights accounts: View all cloud accounts where the resource was discovered
Source metadata: Access provider details, last update times, and source-specific details

Comprehensive Property Views

Compare resource properties across all sources to:

Detect configuration changes between IaC definitions and actual cloud state
View complete metadata from both IaC and cloud discovery
Track property changes over time

Key Benefits for Your Team

Accurate Infrastructure Inventory Get an accurate view of your resources without manual deduplication. Understand exactly what’s running in your cloud accounts and how resources are distributed across regions and projects.

Enhanced Compliance Visibility Instantly identify which resources are managed by IaC versus those created outside your governance processes. Track your progress in bringing unmanaged resources under control.

Faster Incident Response When investigating issues, find all relevant information about a resource in one place. No more switching between multiple views or manually correlating data from different sources.

Learn More

Unified resources are available now for all Pulumi Cloud users. Ready to see unified resources in action? Check out these resources:

Resource search documentation for search tips and filtering options
API reference for programmatic access
Pulumi Insights overview for the complete platform capabilities

We’d love to hear your feedback as you explore unified resources. Share your thoughts in the Pulumi Community Slack or connect with your customer success team.

Start exploring your unified resource inventory today in Pulumi Cloud.

Introducing Google Cloud to Insights Account Discovery

Pulumi Insights Team — Thu, 24 Jul 2025 12:38:18 -0500

We’re excited to introduce Google Cloud support to Insights Account Discovery. This new capability expands Pulumi’s resource discovery to help you uncover infrastructure in Google Cloud, regardless of how those resources are managed. With full visibility into all of your cloud resources, you can monitor, search, and manage your environments more effectively.

Scan an Entire Google Cloud Project in Minutes

With Pulumi Insights, gaining visibility into your Google Cloud environment is fast and frictionless. Once configured, Pulumi scans your selected project and uncovers resource data across services like Compute Engine, Cloud Storage, IAM, networking, and more. There’s no need to install agents or write custom scripts—just link your account and let Pulumi do the rest.

Within minutes, you’ll have a comprehensive view of your cloud infrastructure, including resources not managed by infrastructure-as-code. From there, you can explore your project, search for specific configurations, and start identifying security gaps, cost inefficiencies, and compliance risks.

Visibility Beyond Infrastructure as Code

While infrastructure as code provides control and repeatability, the reality of most cloud environments includes resources created manually or by third-party automation. Pulumi Insights bridges that gap by discovering all of your resources—regardless of how they were created.

With this unified view, you can:

Identify drift from your intended infrastructure state
Spot resources that may introduce security or cost risks
Enforce compliance across all resources using Pulumi Crossguard
Bring unmanaged resources under Pulumi IaC using Pulumi Interactive Import

This makes Pulumi Insights an essential tool not just for monitoring, but also for cloud governance, compliance, and modernization.

Getting Started

Get started today by creating a Google Cloud Insights Account and start uncovering your hidden resources.

Go from Cloud to Code in Minutes with Visual Import

Pulumi Insights Team — Thu, 08 May 2025 14:41:53 -0500

Note: This post discusses Pulumi Copilot, which Pulumi Neo has replaced. Learn about Neo →

Pulumi is excited to announce the release of Visual Import, a new feature that transforms the way platform teams onboard existing cloud infrastructure into Pulumi. With Visual Import, you can discover unmanaged cloud resources across your organization, organize them into logical units, and instantly generate high-quality Pulumi code—with AI enhancements built in.

Visual Import is now available for all Team, Enterprise and Business Critical customers with Pulumi Insights enabled.

Modern Infrastructure, Codified Faster

One of the biggest challenges platform teams face when adopting infrastructure as code is migrating existing cloud deployments. Legacy scripts, manual configurations, and scattered resource ownership all slow down progress. Visual Import solves this by making the journey from cloud to code seamless and intuitive.

Instead of manually digging through consoles and writing boilerplate code by hand, Visual Import enables teams to:

Discover all cloud resources—across any number of accounts—via Pulumi Insights.
Select and group resources into meaningful stacks, services, or projects.
Generate production-ready Pulumi code in your language of choice.
Enhance that code using Pulumi Copilot, which refines formatting, naming, references, and documentation.
Integrate with your development workflows by downloading, editing, and importing Pulumi code into your existing repositories.

Whether you’re modernizing infrastructure, unifying multi-account environments, or enabling developers to adopt infrastructure as code more easily, Visual Import gives you a fast, repeatable onramp to scalable, consistent cloud management.

How It Works

Visual Import is a guided three-step workflow within the Pulumi Cloud Console:

Select Resources: Use powerful filters and grouping tools to find the resources you want to codify. Whether you’re capturing a full service or just a few related components, Visual Import gives you visibility and precision.
Review References: Automatically discover and select dependent resources—ensuring your code captures the right context and connectivity between infrastructure elements.
Generate Code: Choose your preferred language and preview or download fully-formed Pulumi code. You can refine it manually or invoke Pulumi Copilot to improve naming, structure, and comments.

Throughout the workflow, you maintain full control. Select what you need, tweak it as required, and integrate on your terms. You can find full documentation and usage details in the Visual Import docs.

Key Benefits

Accelerated Migration

Visual Import dramatically shortens the time it takes to transition existing cloud infrastructure into a managed, version-controlled state. What previously required weeks of manual discovery, scripting, and trial-and-error can now be accomplished in a matter of hours.

Whether you’re consolidating legacy environments, auditing infrastructure before a cloud cost review, or preparing for a compliance milestone, Visual Import automates much of the busywork—so your team can focus on what matters.

Increased Developer Productivity

Manually transitioning existing infrastructure to code often pulls developers away from their primary focus—building and improving services. By automating code generation and applying intelligent AI enhancements, Visual Import lets teams spend less time writing boilerplate code and fighting resource drift, and more time shipping meaningful changes.

Pulumi Copilot adds a layer of polish that typically requires hands-on review and iteration, saving your team valuable time while helping new contributors write clear, consistent infrastructure code from the start.

Consistent, Reliable Infrastructure

Unifying your cloud environments under a single IaC framework brings consistency, accountability, and repeatability to operations. Pulumi’s typed IaC model makes configurations explicit and auditable, while policies and CI/CD integrations enforce best practices from the start.

By codifying your infrastructure, you reduce the risk of configuration drift, simplify onboarding, and ensure reproducibility across staging and production.

From Insights to IaC—Seamlessly

Visual Import builds on Pulumi Insights, providing a natural path from discovery to codification. If you’re already using Insights to monitor and visualize your infrastructure, you can now convert those resources into managed, versioned code in just a few clicks.

And with built-in support for dependency analysis and cross-account visibility, you can finally treat your entire infrastructure portfolio as a single, coherent system—even if it wasn’t built that way.

Leverage the Full Power of Pulumi Cloud

Once resources are in code, you can take full advantage of Pulumi Cloud’s full suite of features: define and enforce policies with CrossGuard, gain visibility into your entire infrastructure through Pulumi Insights, manage state with ESC’s secrets management, and automate workflows through Pulumi Deployments. Together, these features turn your infrastructure into a governed, observable, and continuously deployable system built for modern cloud operations.

Try It Today

Visual Import is now available in the Pulumi Cloud Console for Team, Enterprise and Business Critical users. If you’re ready to modernize your infrastructure and empower your teams with a fast path to Infrastructure as Code, there’s no better place to start.

Announcing Public Preview of Insights Account Discovery

Craig Symonds — Mon, 02 Dec 2024 09:10:00 +0000

Note: This post discusses Pulumi Copilot, which Pulumi Neo has replaced. Learn about Neo →

Enterprise cloud infrastructures are complex environments that are evolved over time and made up of thousands of different kinds of resources. Enabling customers to wrap their arms around this complexity and get a complete understanding of the scope and structure is the goal of the Pulumi Insights 2.0 product.

We are excited to announce the public preview of Insights Account Discovery that makes it easy for you to gain visibility of your entire infrastructure regardless of how it is managed. Insights provides the tools to find, group and drill into your resources as needed to make sense of all aspects of your cloud infrastructure. In addition Copilot AI gives you the ability to get answers about your infrastructure that can be challenging to get in other ways, dramatically simplifying the work needed to move your infrastructure forward.

Account Management

To get started, Insights Account Discovery provides a simple UI approach to setting up all of your infrastructure accounts to be scanned and read into the Insights supergraph. Once discovered, all of your infrastructure resources can then be seen in the Resource Explorer and accessed by Pulumi Copilot and Resource Search, providing you the tools needed to deliver on your strategic infrastructure goals.

Account Discovery starts with the new Accounts page to provide a list of all accounts created along with the current status of the latest account scans. The Accounts page is the single place to go to manage all aspects of the Insights Account Discovery process.

Account Creation

From this page you can create new Accounts to provide the configuration and credentials needed for Insights to regularly scan and manage your infrastructure. Once you create a new top level Account, Insights will automatically create child accounts, based on the underlying platform model, for each group you enabled. These child accounts enable you to control the discovery behavior for each group separately. For example, AWS enables you to divide you infrastructure into regions and Insights will create separate child accounts for each region you specify.

Account Discovery leverages Pulumi ESC to enter and manage the credentials needed for the Discovery process to find and read all of the infrastructure resources. By relying on ESC, Insights aligns with enterprise best practices for creating and handling application secrets.

Understanding Your Infrastructure

Once each of your Accounts have started scanning, your resources will be displayed in the Insights Resources Explorer along with all Infrastructure as Code resources that you have imported. The Resources Explorer and Resources Search enables you to ask questions and factor your resources into logical groupings that significantly improves the process for managing your infrastructure and getting answers to key questions needed for your projects. For example, you can ask the AI Resource Search to ‘Find all public IP addresses’ and any other question you might have about your infrastructure.

More to Come

We are excited about the potential of Pulumi Insights and are working hard to bring additional capabilities to continue to simplify the experience of managing today’s complex cloud environments. Watch for additional features like Pulumi Crossguard Policy support for Discovered resources, along with additional platform support, and more, coming soon.

Insights Account Discovery is free while in public preview, with per-resource pricing for Team, Enterprise and Business Critical tiers coming in Q1 2025.

Conclusion

The addition of Account Discovery significantly expands the scope of Pulumi Insights. You can now leverage the capabilities of Insights 2.0, not just for your Pulumi IaC managed resources, but for all resources in your infrastructure regardless of how they are managed. With Pulumi Copilot and Resource Search, you are able to gain insights and ask questions about your infrastructure that would otherwise be challenging to answer, saving time and providing the critical information needed.

For additional information about Pulumi Insights please refer to: Pulumi Insights

Introducing the new Resources view

Craig Symonds — Wed, 02 Oct 2024 14:35:35 -0700

Pulumi Insights gives you the tools to stay informed about your cloud infrastructure. Our Resource explorer provides advanced search and filtering too find what you need. Today, we are excited to release an update that adds new ways to factor your resource data, and share those views with other users in your organization!

Factoring the Data

Being able find and group your resources using different fields and properties is the key to gaining the insights needed to understand the status of your infrastructure and determine any future projects needed. The new Resources view provides several updated tools for doing this factoring.

Groups - Resources can be grouped into logical categories, including nested groups. Group resources by properties like stack, modified time, or category to explore your infrastructure from different angles. Relevant aggregations are included for each grouping to help you understand the information at a glance.
Filters - Columns provide type-specific filters to narrow your search. Filters work in conjunction with the free-form search bar, giving you the control to build focused views for different aspects of your infrastructure.
Sorting - Each column supports sorting, and multi-sort support allows multiple columns to be included in the ordering.

Customizing Columns

The updated Resources page enables you to customize the view of your resources by enabling you to control the columns of the table. With the table actions menu you can add and remove columns as well as auto size each of the columns in the table to fit the current content. Like previous versions of the Resources page, this menu also enables you to export the Resources as a CSV file for integration into external tools.

Pinning Favorite Views

Once you have configured the resources view to provide the insights needed, you can now save that view as a pinned view under the Resource tab in the left navigation panel. These pinned views enable the Resources page to act as a dashboard to the most important views and insights into your infrastructure. The views are also defined in the url for the page which means you can freely copy the url and send it to a colleague, or add the URL to a work tracking system to provide context in the task definition.

Overall, the new Resources page provides the tools needed to enable you to gain additional insights and track and manage those insights to ensure your infrastructure is delivering on the goals you have set.

Pulumi - Automate, Secure, and Manage Everything You Run in the Cloud

Joe Duffy — Wed, 18 Sep 2024 09:00:00 +0000

Note: This post discusses Pulumi Copilot, which Pulumi Neo has replaced. Learn about Neo →

At our fourth annual PulumiUP conference, we unveiled a new vision for Pulumi, going beyond infrastructure as code to a comprehensive product suite. Pulumi’s platform now consists of three core products: Pulumi IaC is open source infrastructure as code in any programming language; Pulumi ESC is centralized secrets management that scales; and Pulumi Insights delivers instant visibility, intelligence, and control over all of your infrastructure. Each product can be adopted independently but is better together. In this post, we will share more about how Pulumi is your trusted partner to automate, secure, and manage everything you run in the cloud.

Pulumi - One Unified Platform For All Your Infrastructure Needs

The Pulumi platform solves a broad set of problems that engineering leaders face in the modern cloud era. This spans cloud automation, security, and management, which aligns to our three products: Pulumi IaC, Pulumi ESC, and Pulumi Insights. Our unified approach ensures developers, infrastructure experts, and security teams can collaborate closely and ship faster with high confidence.

In addition to our three products, Pulumi’s platform incorporates three core capabilities that span the entire product suite. Pulumi CrossGuard is our policy as code engine which ensures security, compliance, and best practices; Pulumi Deployments is our deployment workflow technology that lets us orchestrate infrastructure-oriented tasks; and Pulumi Copilot is our generative AI technology that deeply understands the cloud and infrastructure management and operations.

You can see this visually depicted here:

We’ve now worked with 3,000 customers to help with their cloud journey and have listened to and solved the problems we regularly hear about. That includes Nvidia who is at the forefront of the AI revolution as well as BMW who is reimagining their entire engineering mindset in a cloud-first world, both leveraging Pulumi. Unity Games now ships 5x faster from code to cloud, and Tivity Health saved 71% of their cloud spend, automating everything and saving $6.5M annually.

Now let’s dive into the three products and what we announced at PulumiUP:

Pulumi IaC - Infrastructure as Code in Any Programming Language

Pulumi IaC is our flagship open source technology, and it powers virtually everything we do, even the new products. We have seen accelerated adoption of Pulumi IaC. We’ve had over 100M downloads, are about to crack 21K GitHub stars (give us a star and bump us over!), and have seen 167% the contributions of Terraform this past month and 300% of OpenTofu. Not only is Pulumi the most powerful IaC technology, it is becoming more popular and vibrant too.

We are always improving Pulumi IaC and today is no different. We launched a number of new and exciting IaC features. That includes using any Terraform provider, beyond the nearly 200 providers that Pulumi already supports. We also shipped a new Pulumi VS Code Extension that enables rich debugging and easier editing of your IaC programs. Pulumi Copilot can now diagnose and fix your IaC errors. And finally, we’ve improved our Kubernetes support with powerful controls over await logic for more robust cloud native deployments. And we’re not slowing down anytime soon.

Pulumi is open source, of course, but works best with Pulumi Cloud, the easiest, most secure, and most reliable way to adopt Pulumi IaC. Starburst saved 100 days per year by moving from DIY state management to Pulumi Cloud. By using Pulumi Cloud, you also gain access to the other products in this post. Try IaC now and choose the option that suits you best.

Pulumi ESC - Centralized Secrets Management and Orchestration that Scales

Pulumi ESC is a new approach to secrets and configuration management that is scalable, secure automatically with easy dynamic credentials, and tames sprawl with engineering best practices. ESC is not just a feature of our IaC product — it can be adopted entirely independently of it for application and infrastructure secrets. We initially launched ESC in preview nearly a year ago, and since then have had hundreds of customers and thousands of users try it out – many at production scale – who have given us amazing feedback. We are excited to announce that ESC is generally available.

Being generally available is a big deal to us, and means it is ready for robust and secure enterprise-grade workloads. New capabilities available at GA include a new projects system which lets you group your environments into familiar hierarchies. We have added versioning, tagging, and pinning, so that you can institute more robust change management practices. ESC can manage encrypted secrets directly, but we have always supported pulling secrets from other secret stores, like AWS Secrets Manager, Azure KeyVault, Google Secret Manager, and HashiCorp Vault. We have now added 1Password to the list. We now also support secrets sync, which lets you still edit your secrets straight from Pulumi ESC, but have them automatically mirrored back to the backing secrets store. Flexible webhooks now support custom workflows triggered by environment changes. The new Pulumi VS Code Extension helps you edit your environments in code. We’ve launched application SDKs for Node.js, Python, and Go, as well as a Kubernetes Operator to inject secrets into Kubernetes workloads.

We aim for you to “fall into the pit of success” with security, so that it is built-in and automatic, and ESC is a major enabler of that. Try out ESC now, or read more in the launch post.

Pulumi Insights - Visibility, Intelligence, and Controls Over All Infrastructure

Pulumi Insights already delivers visibility, intelligence, and controls over your infrastructure resources. We began with Insights a year and a half ago as a collection of IaC features. We added resource search with structured and natural language search syntax, to help you find your IaC infrastructure no matter where it is running. We added the ability to export your infrastructure data and slice and dice it from data tools like Snowflake. We also added new dashboards to help visualize your infrastructure at scale.

The big news with Pulumi Insights 2.0 is that we’re bringing these capabilities to all of your cloud infrastructure, no matter where it is, and no matter how it was created. You point Pulumi at your cloud accounts for any of nearly 200 supported clouds, and you get an instant asset inventory of everything you have. This is true even for resources created by Terraform, CloudFormation, Azure Resource Manager, cloud CLIs and SDKs, or even by clicking in your cloud console. An improved resource view gives you pivot-table-like capabilities so you can understand everything you have, and new graph visualizations with inferred dependencies help you to make sense of how things relate. Search and Pulumi Copilot work over discovered cloud resources, letting you explore and ask questions interactively. Pulumi Insights leverages Pulumi CrossGuard, our policy as code technology, and can tap into any existing IaC policies to find issues pertaining to security, compliance, cost, and best practices. Pulumi’s compliance-ready policy packs just work. Insights can also leverage CrossGuard’s ability to automate remediations so that Insights not only tells you about your issues, but helps you fix them with the click of a button. Finally, Insights is better together with Pulumi IaC, and enables you to migrate any resources in the cloud to Pulumi IaC – improving the robustness and built-in security of your operations – with a fully interactive import experience.

This marks the start of an ambitious long-term vision for Pulumi Insights. Some of what was announced is actively shipping while other pieces will ship in the coming weeks. Learn more in the launch blog and, get started with Pulumi Insights 2.0.

Thank You

I am incredibly thankful to our community, 3,000 customers, and the partners and speakers who made PulumiUP 2024 a reality. Your feedback and continued belief in Pulumi has fueled the innovations we’re shipping today and into the future.

All of the talks are streaming live here and will be up on YouTube afterwards.

We can’t wait to hear what you think about the new vision for Pulumi’s platform as well as the new IaC, ESC, and Insights capabilities announced at PulumiUP 2024. As always, you can try Pulumi for free here.

Happy cloud spelunking.

-Joe

Introducing Pulumi Insights 2.0

Craig Symonds — Wed, 18 Sep 2024 08:57:00 +0000

Note: This post discusses Pulumi Copilot, which Pulumi Neo has replaced. Learn about Neo →

Pulumi Insights 2.0 brings the power of Insights to all of your cloud infrastructure, not just the infrastructure managed by Pulumi IaC. Insights 2.0 also brings an exciting new suite of tools that make it easy to discover your entire infrastructure, assess security and efficiency of it, and bring it under management of IaC for best-in-class infrastructure management practices.

Today we are outlining our vision for Pulumi Insights 2.0, and the features that are part of Insights 2.0 will be released over the coming weeks and months. Get started with Pulumi Insights 2.0.

Since the initial launch of Pulumi Insights, customers have used these tools to gain valuable awareness into their IaC platform resources through the flexibility of Resource Search and the power of Pulumi CrossGuard. Insights 2.0 builds on this previous generation of tools to provide a range of new capabilities:

Bring visibility to your entire infrastructure by enabling you to scan and import all of your resources
Add rich new visual explorers, structured graphs and dashboards for understanding and managing your entire infrastructure
Extends Pulumi CrossGuard to provide direct remediation of discovered policy violations inside the Pulumi Cloud
Integrates Pulumi IaC to make it easy to incrementally bring your infrastructure under management with Pulumi IaC in just a few clicks
Brings Pulumi Copilot to your entire infrastructure for intelligent insight discovery and analysis

Resources 2.0

The updated Resources page provides a customizable experience for grouping, filtering, sorting and visualizing your resources to help you manage scope and discover new insights about your infrastructure.

Grouping: To group by a particular column, drag and drop the column header to the grouping control in the top left of the grid. You can group by multiple columns and can control the order of the grouping by reordering them in the grouping control. Grouping enables you to factor your resources in different ways to help you understand the scope and uncover issues within your infrastructure.

Filtering: Adding filters to each column can be done by selecting the filter icon in the column header. A filter dialog will be displayed based on the type of data present in the column. Adding a column filter will add the specific filter to the Resource Search query enabling you to use the grid as a form of query builder.

Sorting: To sort your resources by a particular column, click once on the column header to sort in ascending order and click a second time to sort in descending order.

Column Selection: To add or remove columns from the grid, select the ‘Choose Columns’ menu item from the column menu selector. This presents a pop up to allow you to add, remove or rearrange the order of columns in the grid.

Column Resize: You can resize the width of a column by clicking and dragging on the column boundary line on the right side of the column header. To automatically resize the column to the width of the widest cell contents, double click on the boundary line.

Copy Contents: You can copy the contents of a cell to the clipboard by right clicking on the cell and selecting Copy. This is particularly useful for resource values you need to use in other contexts like the Name or URN.

Policy Violations

The Policy Violations page provides a comprehensive view of all policy violations across your organization, helping you maintain visibility and control over your infrastructure. See the Policy Violations blog post for more information.

Resource Structure Graph

The new resource structure graph provides a visual representation of the relationships for a selected resource. This provides the additional context needed to understand the structure around the resource and what actions might need to be taken.

Within this view you can navigate the relationships by clicking on the related resources. When a resource is clicked, the Resource Detail view navigates to the selected resource to give you the details needed to understand that context.

Policy violations are indicated on the Resource Structure graph to quickly highlight when an insight has been detected and what related nodes are affected.

Goto Definition

For any resource that is Pulumi IaC managed, the Resource Details page will contain a link to the file and line of code in your source code provider that defines the given resource.

Selecting this link will take you to the file and line making it easy for you to quickly see and edit the IaC code for the given resource.

User Customizable Dashboard

Pulumi Cloud will provide a rich, user customizable dashboard that enables you to define views of any aspect of your infrastructure. This provides a powerful overview of your infrastructure and lets you easily track progress of any project.

The dashboard supports defining multiple pages so you can create separate dashboards for each application, each department, or any configuration that maps to your configuration.

Each dashboard page enables you to add, remove, rearrange and create new cards. Each new card can be customized to show the aggregate results of a resource field or property including related policy violation fields. The card can be configured to use a variety of different chart and graph visualizations that best suit the associated data.

Infrastructure Account Scanning

We are excited to introduce Infrastructure Account Scanning (IAS). IAS fundamentally transforms Pulumi Cloud into an Intelligent Cloud Management platform that brings all of the Pulumi Insights capabilities to your entire infrastructure. With IAS, Pulumi can scan your entire infrastructure, even resources that are not IaC managed. This enables you to leverage all of the new tools like, the Resource Explorer with Resource Structure Graphs, Pulumi CrossGruard with Auto Remediation and the Policy Violations Explorer, and the User Customizable Dashboard to manage all of your cloud resources.

IAS leverages the rich ecosystem of Pulumi Platform Providers to discover and read your resources independently of whether they have been moved to IaC.

With IAS Account management you can quickly configure how Pulumi Cloud is enabled to discover new or changed resources keeping your customized Resources, Policy Violations and Dashboard views up to date with your physical infrastructure.

Auto Remediation

Insights 2.0 provides you the tools needed to understand your infrastructure and uncover issues to ensure you can maintain a secure and efficient cloud platform. In addition to these tools, Pulumi Cloud will also give you the context and tools needed to remediate the issues.

When a policy violation occurs for a given resource and that policy is an Auto Remediation policy, a ‘Remediate’ button will be displayed in the Resource Details page. When selected, the Remediate button will run the remediation and show a diff of the infrastructure changes that will be applied in order to remediate targeted insight. If approved, Pulumi Cloud will apply the changes defined by the diff to your infrastructure using the associated Pulumi Providers.

Any change made through the Remediation flow will be tracked and auditable using the Pulumi Audit tools to give you confidence and tracking of the changes made to your infrastructure. In addition, this Remediation flow will provide a warning if the Resource is IaC managed as this change would cause drift with the underlying IaC code. For IaC managed resources it is recommended that you use the new Goto Definition feature to jump to the IaC code for the resource to apply the necessary remediation in the code.

CLI Resource Query Import

Being able to browse your entire infrastructure and discover new opportunities to move your platform forward enables you to understand and build plans to move your infrastructure forward. However, leveraging Pulumi IaC for managing your resources gives you the best in class tools to ensure your infrastructure is delivering on your strategic goals. Insights 2.0 significantly simplifies the process of migrating your resources to IaC management by making it easy to find the resources needed and create resource queries to define those groups.

The Pulumi cli import command will now support passing a Resource Query as a parameter, simplifying the process of finding, grouping and importing the resources you want. The results of this query will be used to get the resource IDs and types needed to be able to import the resources into your IaC program. The result of the command is the source code needed to construct each of the resources defined by the query.

pulumi import \-q ‘.volumeType:Standard’
...
import * as pulumi from "@pulumi/pulumi";
import * as aws_native from "@pulumi/aws-native";

const EC2Volumevol0a789ea1874bfac61 = new aws_native.ec2.Volume("EC2Volumevol0a789ea1874bfac61", {
 autoEnableIo: true,
 availabilityZone: "us-west-2b",
 encrypted: false,
 multiAttachEnabled: false,
 size: 8,
 snapshotId: "snap-0d017523dfd31e801",
 tags: [],
 volumeType: "standard",
}, {
 protect: true,
});
...

Visual Resource Import

In addition to the powerful command line tools for simplifying the process of importing your resources into Pulumi IaC, Insights 2.0 will provide an integrated UI workflow for finding and importing resources into IaC. The workflow leverages the rich Insights explorer tools for finding and grouping your resources, then enabling you to automatically create a PR with the generated IaC code for the resource group, all with a click of a button.

Conclusion

Insights 2.0 provides you the set of tools needed to understand the chaos of modern enterprise clouds and to bring that chaos under control. The ability to scan your entire infrastructure enables you to leverage the rich visualization, remediation, and IaC migrations tools for all aspects of your infrastructure and gives you the insights needed to enable you to deliver on your strategic objectives.

Get started with Pulumi Insights 2.0.

Introducing Pulumi Copilot: Intelligent Cloud Management

Luke Hoban — Wed, 12 Jun 2024 04:00:00 -0700

Note: This post discusses Pulumi Copilot, which Pulumi Neo has replaced. Learn about Neo →

AI is transforming how users interact with every category of technology today, and cloud infrastructure is no exception. Last year we launched Pulumi AI to combine generative AI with Pulumi’s knowledge of cloud infrastructure, helping users solve complex cloud development problems using Infrastructure-as-Code. Pulumi AI has seen rapid adoption and engagement, with tens of thousands of users leveraging Pulumi AI over the last year.

Today, we’re excited to introduce Pulumi Copilot, a huge leap forward in applying AI to cloud infrastructure management. Pulumi Copilot is a new conversational chat interface integrated throughout Pulumi Cloud, enabling Pulumi Cloud users to quickly accomplish a variety of cloud infrastructure management tasks by leveraging the power of large language models plus the rich capabilities of Pulumi Cloud.

Through Pulumi Copilot, you can explore your cloud infrastructure and gain insights across an incredible breadth of use cases, including:

Pulumi Copilot Key Features

Access any data in Pulumi Cloud

The state of every resource you are managing with Pulumi across any Cloud, any account, and any region. With Pulumi Insights’ Cloud Supergraph support for 160+ cloud providers, this offers an unprecedented breadth of cloud infrastructure data to explore and interrogate with Pulumi Copilot.
Pulumi stacks, projects, updates, deployments, environments, policies, audit logs and more - enabling historical understanding of what happened when, by who, and why across all of your cloud engineering systems managed by Pulumi.

Pulumi IaC Authoring and Deployment

The same great Pulumi AI features for authoring IaC are now available inside Pulumi Copilot as well, enabling you to quickly solve new IaC problems within Pulumi Cloud, and even deploy code directly from Pulumi Copilot.

Access cloud metadata from the clouds themselves

Through the use of new skills, Pulumi Copilot can access cloud metadata in real time in AWS, Azure, Kubernetes, and more, allowing it to join Pulumi’s IaC world view with information about usage, costs, and more – as well as infrastructure not yet under the management of Pulumi.

Pulumi Copilot combines the best generative AI models available in the industry today with Pulumi Copilot “skills” which enable Pulumi Copilot to access the data and actions from within Pulumi Cloud needed to help you with your questions and explorations. Pulumi Copilot also incorporates the context of where the user is in the Pulumi Cloud console to easily answer questions about “this stack”, or “the latest update”, offering an even more natural, conversational and persistent experience across Pulumi Cloud.

Pulumi Copilot is Enterprise-Ready on day one - it adheres to all the same identity and role based access control (RBAC) rules that exist across Pulumi Cloud, ensuring that users only have access to stacks, environments and resources that they have permissions for within Pulumi Cloud. This means that each organization gets AI generated responses based solely on their own organizational data.

Pulumi Copilot is available now in public beta, and is free while in beta for all organizations in Pulumi Cloud. Organization administrators can turn on Pulumi Copilot for their organization by going to Settings > Access Management > Pulumi Copilot within the Pulumi Cloud console.

Try Pulumi Copilot now at https://app.pulumi.com/.

Walkthrough of Pulumi Copilot

Let’s jump in and take Pulumi Copilot for a spin!

We can start off simple - it’s a conversational chat, so we can introduce ourselves. Pulumi Copilot knows who we are from context and we get a light hearted programming related joke.

Perhaps we need to gain access to additional infrastructure within Pulumi Cloud and need to contact an admin, we can ask who the admins are.

We get a hyperlink at the end, enabling us to jump directly to the members page, where we can see all of the details about the members and admins of the organization. Pulumi Copilot helps to not just get information, but then navigate within the Pulumi Cloud console to continue to explore.

Pulumi Copilot has access to everything that Pulumi Cloud manages, including Pulumi ESC environments. We can find all of the environments we have that manage access to GitHub through a natural language query, and then ask a follow up question to get a direct link to where each environment is defined.

If we do a Pulumi update, and it fails, we often get a complex error message, caused by some combination of a coding problem or a cloud provider configuration problem or a transient failure in cloud providers backing service. We can use Pulumi Copilot to ask why an update failed, and get a plain language explanation, leveraging the AI’s existing knowledge of programming languages and cloud providers, combined with access to all of the update logs from within Pulumi Cloud. Notably, we can ask why “this” update failed, taking advantage of the context of the page we are on within Pulumi Cloud to understand which update we are referring to.

Perhaps we received an email from our compliance team telling us there was an issue with the VPC with id vpc-04a116f7075fb7ca5, but we have no idea where that is. Given we have hundreds of AWS accounts and dozens of regions in each of these accounts, finding this resource isn’t simple. We can just ask Pulumi Copilot, and it immediately tells us what AWS account it is in. We can follow this up with a question about which Pulumi project and stack is managing it, and get a link to go directly to the stack to learn more about how it is configured.

All of the capabilities of Pulumi AI for writing Pulumi IaC code are also available in Pulumi Copilot, so we can ask how to write a program for a new use case. But we can also have Pulumi Copilot incorporate information from an existing stack, leveraging multiple Pulumi Copilot skills to help with this task.

If we are thinking about removing some infrastructure we may want to know what other infrastructure potentially depends on it. We can get a list of stacks which depend on our stack, with hyperlinks directly to those stacks so we can explore their dependencies and take action to update them.

If we’re getting started on compliance with a new compliance framework, we can get guidance on areas to focus on for compliance review, ahead of applying more formal compliance tools like Pulumi Crossguard Compliance Ready Policies.

And we can get insights into what infrastructure is exposed to the internet which we may need to look deeper into with a quick question:

And our questions don’t have to be limited to English. Early Spanish-speaking users have found themselves working with Pulumi Copilot in their native language, getting responses automatically in the same language of their question.

All of these examples are just scratching the surface of what Pulumi Copilot can already do today. One of the most exciting things about conversational chat interfaces is their wonderfully wide surface area of potential applications - almost any question you might have related to Cloud Infrastructure managed within Pulumi Cloud is something Pulumi Copilot can help with!

Future of Pulumi Copilot

All of the examples above are things you can do today, but we’re not stopping there. There’s a lot more coming up, and we wanted to give a sneak peek into some of the directions we’re taking Pulumi Copilot based on early customer feedback.

Taking Action

In the initial Beta, Pulumi Copilot is restricted to get data and answer questions, but cannot take action directly, beyond guiding users to a place within the Pulumi Cloud console to take the action themselves. In the near future, we will be extending Pulumi Copilot with the ability to propose actions and then act on them with user approval.

Check out pulumi/cloud-requests#414 for more details.

Pulumi CLI Integration

One of the most common requests we have gotten from early Pulumi Copilot users has been for integration into the Pulumi CLI where many users spend much of their time day-to-day. We have started experimenting with CLI integration to help with diagnosing Pulumi update failures, and will be adding this to the CLI in the near future.

Check out pulumi/cloud-requests#415 for more details.

Pulumi Docs Website Integration

Pulumi Copilot is initially offered as part of Pulumi Cloud. But we are working to extend Pulumi Copilot across the rest of pulumi.com, including integration with the Pulumi Docs website. A new Pulumi Docs skill will allow Pulumi Copilot to help users learn Pulumi concepts and chat with Copilot about these to go deeper on concepts than what is already provided directly within the Pulumi Docs website.

Check out pulumi/cloud-requests#417 for more details.

Cloud CLI Skills

While Pulumi Copilot already has access to all the Cloud Infrastructure you are managing with Pulumi Cloud, there are many cases where it is useful to reach out to AWS directly to ask questions, get operational metrics, and take action. We are already working on extending Pulumi Copilot with the ability to run aws and kubectl CLI commands, using short-lived credentials from one of your Pulumi ESC environments. A sneak peek of that in action - showing how the AWS skill, combined with the ability to get details about resources and ESC environments from within Pulumi Cloud can enable collecting operational metrics from AWS and presenting them for analysis.

Check out pulumi/cloud-requests#416 for more details.

Conclusion

We’re excited to introduce Pulumi Copilot, the first conversational AI for general multi-cloud infrastructure management. Pulumi Copilot is the next big step toward reimagining cloud infrastructure management via AI-based user experiences. We’re just getting started, with many improvements and new capabilities coming soon on top of the core Pulumi Copilot foundation. Pulumi Copilot is brand new and improving quickly in response to user feedback. If you have questions or suggestions:

Get started with Pulumi Copilot in Pulumi Cloud.

Pulumi Insights and AI in the Pulumi CLI

Luke Hoban — Wed, 13 Sep 2023 10:00:00 -0800

Earlier this year we introduced Pulumi Insights, a collection of features that bring intelligence to cloud infrastructure using Pulumi. Two key components of that launch were Pulumi AI, a generative AI assistant purpose-built to create cloud infrastructure using natural language, and Pulumi Resource Search, multi-cloud search and analytics across every cloud resource and environment in your organization.

Today, we are excited to bring Pulumi Insights into the pulumi CLI with the new pulumi org search and pulumi ai commands. These commands put AI and resource search at your finger tips right where Pulumi users spend most of their time, in the terminal iterating on their cloud infrastructure.

This release is the latest of a long list of improvements and enhancement to Pulumi Insights over the last few months, including AI Assist for resource search, fine-grained resource property search, and registry support in Pulumi AI. We’ve seen incredible adoption and usage of all of these features over the last few months, and they are now easier than ever to access via the pulumi CLI.

Pulumi Resource Search in the CLI

With the new pulumi org search command, you can search across all of the infrastructure managed by your organization (or individual account) using the same rich search syntax supported by Pulumi Resource Search in the Pulumi Console.

pulumi org search -q "type:aws:kinesis/stream:Stream modified:>=09/01/2023" --org my-org

If you don’t know the exact search syntax for what you want to ask, you can use AI Assist, which was recently opened up to all Pulumi organizations. Just use pulumi org search ai instead!

pulumi org search ai -q "all Kinesis streams modified since September 1st 2023" --org my-org

Notice that AI Assist comes up with the same query and runs it to get the same results we saw above.

These queries can also include complex logic which relies on looking at properties of resources, such as their tags. For example:

pulumi org search ai -q "All untagged lambda functions with memory > 2 GB" --org my-org

AI Assist determines it should run the query type:"aws:lambda:Function" .memorySize:>2048 -.tags: and shows the results of that query:

The combination of rich query syntax, AI Assist and property search makes it easy and exciting to interactively find the information you are looking for about resources in your stack.

And of course, you can jump straight into the console to drill down even further into your data.

Pulumi AI in the CLI

We’ve also taken our first step towards integrating Pulumi AI into the CLI. With the pulumi ai web command, you can type a query in the CLI and have it launch a question into a browser window.

pulumi ai web "deploy metabase on AWS with a managed postgres database"

This opens a browser to a guided Pulumi AI conversation that walks through all the steps required to accomplish this task.

Conclusion

With these new CLI commands, Pulumi Insights is available at your fingertips in the pulumi CLI. These are just the first steps to bring search and AI into the CLI, expect to see much more in the coming months. To learn more about the various Pulumi Insights features discussed here, check out the resources below:

Pulumi Blog: Insights

Connect Your Cloud Accounts to Pulumi in Minutes

From hours of setup to a single flow

How it works

Choose the right access level for your security posture

Everything set up, nothing hidden

Get started

Scan AWS GovCloud and more partitions with Pulumi Insights

Supported partitions

Set it up

From Kubernetes Gatekeeper to Full-Stack Governance with OPA

What’s in the stable release

Kubernetes Gatekeeper compatibility

Walkthrough: Reusing policies from the gatekeeper-library

Part of the Pulumi Insights governance story

Frequently asked questions

Do I need to modify my existing Gatekeeper .rego files?

What happens with non-Kubernetes resources?

Do I need OPA installed locally?

When should I use OPA vs. TypeScript or Python for policies?

Get started

Run Pulumi Insights on Your Own Infrastructure

Insights at a glance

Why self-hosted?

How it works

Get started

How We Built a Distributed Work Scheduling System for Pulumi Cloud

Where we started

Why not use an off-the-shelf queue?

Design constraints

The background activity

The state machine

Leases: distributed execution without coordination

How it works

Routing work to the right runner pool

Dependencies and multi-step workflows

Two execution modes, one interface

Putting it all together

Retries and scheduling

Observability

What we learned

Wrapping it up

Unified Resources in Pulumi Cloud

Why We Built This

How It Works

The New Unified Resource Details Page

Complete Source Visibility

Comprehensive Property Views

Key Benefits for Your Team

Learn More

Introducing Google Cloud to Insights Account Discovery

Scan an Entire Google Cloud Project in Minutes

Visibility Beyond Infrastructure as Code

Getting Started

Go from Cloud to Code in Minutes with Visual Import

Modern Infrastructure, Codified Faster

How It Works

Key Benefits

Accelerated Migration

Increased Developer Productivity

Consistent, Reliable Infrastructure

From Insights to IaC—Seamlessly

Leverage the Full Power of Pulumi Cloud

Try It Today

Announcing Public Preview of Insights Account Discovery

Account Management

Account Creation

Understanding Your Infrastructure

More to Come

Conclusion

Introducing the new Resources view

Factoring the Data

Customizing Columns

Pinning Favorite Views

Pulumi - Automate, Secure, and Manage Everything You Run in the Cloud

Pulumi - One Unified Platform For All Your Infrastructure Needs

Pulumi IaC - Infrastructure as Code in Any Programming Language

Pulumi ESC - Centralized Secrets Management and Orchestration that Scales

Pulumi Insights - Visibility, Intelligence, and Controls Over All Infrastructure

Thank You

Do I need to modify my existing Gatekeeper `.rego` files?