<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0"><channel><title>Pulumi Blog: Derek Schaller</title><link>https://www.pulumi.com/blog/authors/derek-schaller/</link><description>Pulumi blog posts: Derek Schaller.</description><language>en-us</language><pubDate>Fri, 25 Sep 2026 12:00:00 -0700</pubDate><item><title>AI agents need continuity, not just context</title><link>https://www.pulumi.com/blog/neo-kopia-workspace-snapshots/</link><pubDate>Fri, 25 Sep 2026 12:00:00 -0700</pubDate><guid>https://www.pulumi.com/blog/neo-kopia-workspace-snapshots/</guid><description>
&lt;img src="https://www.pulumi.com/images/generated/blog/neo-kopia-workspace-snapshots/index.png" /&gt;
&lt;p&gt;Pulumi Neo works on infrastructure the way an engineer does: it clones repositories, edits files, installs dependencies, runs previews, and produces intermediate work along the way. A task is not only a conversation with a model. It is also a working directory that has to survive long enough for the agent to keep making progress.&lt;/p&gt;
&lt;p&gt;Imagine asking an agent to upgrade a Pulumi provider version across a repository, regenerate SDK code, run previews and tests, inspect the failures, and open a pull request. Halfway through, the runtime restarts. Conversation replay can recover what the agent said and which tools it called, but it does not recover the generated files, dependency state, local commits, or scratch output the next step may depend on. Without workspace continuity, the agent can wake up remembering the plan but missing the room it was working in.&lt;/p&gt;
&lt;p&gt;For early hosted Neo tasks, workspace recovery started with Git. We persisted enough information to reconstruct repositories later: remotes, branches, commits, and local diffs. This was about the hosted runtime workspace, not a user&amp;rsquo;s local checkout when running &lt;code&gt;pulumi neo&lt;/code&gt;, where local filesystem and shell tools execute on the user&amp;rsquo;s machine. The Git-based approach worked for simple hosted-runtime cases, but it was the wrong abstraction for long-running agent work. We needed to recover the workspace Neo had actually used.&lt;/p&gt;
&lt;p&gt;This post walks through how we tackled workspace continuity for long-running agent tasks, what we learned along the way, and how the same techniques can help you build more resilient agentic systems of your own. The core primitive we chose was &lt;a href="https://kopia.io/"&gt;Kopia&lt;/a&gt;, an open source backup tool that creates encrypted, incremental, content-addressed snapshots and stores them in backends such as object storage.&lt;/p&gt;
&lt;h2 id="git-was-the-wrong-recovery-boundary"&gt;Git was the wrong recovery boundary&lt;/h2&gt;
&lt;p&gt;The original persistence model was repository-oriented. After a turn, the hosted agent runtime scanned its own working directory for Git repositories, recorded each remote, branch, commit, and local diff, and stored that representation through our task state API. On a cold start, a new runtime fetched that state, cloned the repositories again, checked out the saved revisions, and reapplied patches.&lt;/p&gt;
&lt;p&gt;That was a reasonable first design because most infrastructure work starts in Git. It also kept persisted state relatively small when local changes were simple.&lt;/p&gt;
&lt;p&gt;But agent workspaces are not only Git repositories. They contain generated files, scratch files, local commits, package-manager state, tool output, and sometimes files outside any repository. The reconstruction path also depended on everything still being fetchable later: the remote repository had to exist, credentials had to work, the saved revision had to be available, and the patch still had to apply cleanly.&lt;/p&gt;
&lt;p&gt;In practice, that showed up in a few painful ways:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Resumed runtimes could start with incomplete working directories even though the conversation history replayed successfully.&lt;/li&gt;
&lt;li&gt;Tasks could fail to resume with clone errors when a repository was deleted after the task started and no longer existed at its original remote.&lt;/li&gt;
&lt;li&gt;Large diffs could outgrow the payload limit for persisted task state.&lt;/li&gt;
&lt;li&gt;Intentionally skipped directories, such as dependency caches and build output, sometimes contained context the agent had just created.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The distinction that finally clarified the design was simple:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Replay events to restore what Neo remembers. Restore a snapshot to recover what Neo changed.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;Conversation memory and workspace state are separate continuity planes. Event replay rebuilds messages, tool calls, approvals, and compaction state. Filesystem snapshots rebuild the workspace.&lt;/p&gt;
&lt;h2 id="the-thing-we-were-not-willing-to-make-customers-run"&gt;The thing we were not willing to make customers run&lt;/h2&gt;
&lt;p&gt;The deciding question was not &amp;ldquo;what can back up a directory?&amp;rdquo; Lots of things can back up a directory. The deciding question was &amp;ldquo;what are we willing to make every Pulumi Cloud and self-hosted customer operate so Neo can resume a task?&amp;rdquo;&lt;/p&gt;
&lt;p&gt;That ruled out more than you might expect.&lt;/p&gt;
&lt;p&gt;EFS plus AWS Backup is lovely if your answer to every deployment question is &amp;ldquo;be on AWS.&amp;rdquo; Ours is not. Self-hosted Pulumi has to work in customer environments, and those environments already have a portable storage requirement: blob storage. The snapshot system needed to ride on that, not introduce an AWS-shaped trapdoor.&lt;/p&gt;
&lt;p&gt;A Git server had a similar problem. Neo workspaces often include Git repositories, but the workspace is not itself a Git repository. Turning recovery into &amp;ldquo;run a Git service, maybe with many repositories per task, plus enough conventions to capture non-Git files&amp;rdquo; would have given us another product to operate and another thing for self-hosted customers to debug.&lt;/p&gt;
&lt;p&gt;Direct object storage sync looked better from the portability angle, but it was still the wrong abstraction. A sync can copy files, but it does not naturally give us point-in-time restore, efficient rename-heavy history, retention, or a clean recovery model after partially failed writes. Block-volume snapshots had the opposite problem: strong recovery semantics, but the wrong deployment and cost model for task-scoped workspaces.&lt;/p&gt;
&lt;p&gt;That left us looking for a boringly useful shape: an encrypted, task-scoped snapshot repository over generic object storage.&lt;/p&gt;
&lt;p&gt;Kopia gave us the closest match to the unit of recovery: the task workspace. It supports incremental snapshots, point-in-time restore, encryption, content-addressed storage, object storage backends, and practical cleanup. Renames and moves generally change metadata rather than rewriting all bytes. Frequent temporary-file churn still matters, which is why ignores and retention policy are part of the design rather than afterthoughts.&lt;/p&gt;
&lt;p&gt;The repository layout is intentionally task-scoped:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-text" data-lang="text"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;s3://&amp;lt;task-backup-bucket&amp;gt;/
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; tasks/
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; &amp;lt;task-id&amp;gt;/
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; kopia/
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; ...encrypted repository data...
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;That gives up cross-task deduplication, but it aligns the storage boundary with the security and lifecycle boundary. A task ID names the storage prefix; it is not the access-control boundary by itself. The service still has to authenticate the task before minting temporary credentials, those credentials are scoped to that task&amp;rsquo;s prefix, and the Kopia repository is encrypted with a per-task password. One task&amp;rsquo;s runtime must never be able to read another task&amp;rsquo;s snapshot history.&lt;/p&gt;
&lt;h2 id="split-the-control-plane-from-the-data-plane"&gt;Split the control plane from the data plane&lt;/h2&gt;
&lt;p&gt;The next decision was where the bytes should move. We wanted the service to authorize recovery, not become the storage proxy for every dependency tree, generated artifact, and scratch file an agent might create.&lt;/p&gt;
&lt;p&gt;Our service remains the control plane. It authenticates the task, decides whether snapshotting is &lt;code&gt;disabled&lt;/code&gt;, &lt;code&gt;shadow&lt;/code&gt;, or &lt;code&gt;enabled&lt;/code&gt;, and mints temporary credentials scoped to that task&amp;rsquo;s storage prefix. The runtime remains the data plane for its own local workspace: it restores and checkpoints directly against object storage instead of streaming large filesystem payloads through the service.&lt;/p&gt;
&lt;p&gt;That split is also the multi-tenant security boundary. The runtime gets only the storage authority for the task it is currently serving, and the encrypted repository it opens is protected by that task&amp;rsquo;s repository password. A different task would need both a different storage grant and a different repository password to read its snapshots.&lt;/p&gt;
&lt;pre class="mermaid"&gt;
flowchart LR
Events[Persisted task events] --&amp;gt;|replay memory| Runtime[Neo runtime]
Service[Service control plane] --&amp;gt;|snapshot mode| Runtime
Service --&amp;gt;|short-lived task-scoped credentials| Runtime
Runtime --&amp;gt;|restore and checkpoint| Kopia[Kopia repository for one task]
Kopia --&amp;gt; S3[S3 task prefix]
Cleanup[Trusted cleanup path] --&amp;gt;|expire and maintain| Kopia
&lt;/pre&gt;
&lt;p&gt;This split matters. If the service were the only storage principal, every snapshot and restore would have to flow through the application service. That would turn the service into a high-volume data path for dependency trees, generated artifacts, and arbitrary workspace state. Instead, the service authorizes access, while the runtime moves bytes directly to and from storage.&lt;/p&gt;
&lt;p&gt;Isolation has several layers:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;The service only issues credentials after authenticating the task.&lt;/li&gt;
&lt;li&gt;Credentials are short-lived and scoped to a single task prefix, not just a bucket.&lt;/li&gt;
&lt;li&gt;Runtime credentials can read and write only that task&amp;rsquo;s snapshots, but cannot delete durable history.&lt;/li&gt;
&lt;li&gt;Each task has its own Kopia repository password, so object access and repository decryption are separate checks.&lt;/li&gt;
&lt;li&gt;Cleanup runs through a trusted path with delete authority.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Kopia&amp;rsquo;s retention model also helped us keep delete permission out of the runtime. Snapshot expiration removes references, and maintenance later reclaims unreferenced encrypted blobs. The agent-controlled runtime can create recoverable history without being able to erase it.&lt;/p&gt;
&lt;h2 id="roll-out-the-escape-hatch-first"&gt;Roll out the escape hatch first&lt;/h2&gt;
&lt;p&gt;We did not want a single switch that made a new persistence layer authoritative on day one. The rollout used three modes:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Mode&lt;/th&gt;
&lt;th style="text-align: right"&gt;Legacy repo reconstruction&lt;/th&gt;
&lt;th style="text-align: right"&gt;Snapshot writes&lt;/th&gt;
&lt;th style="text-align: right"&gt;Snapshot restores&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;disabled&lt;/code&gt;&lt;/td&gt;
&lt;td style="text-align: right"&gt;Authoritative&lt;/td&gt;
&lt;td style="text-align: right"&gt;No&lt;/td&gt;
&lt;td style="text-align: right"&gt;No&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;shadow&lt;/code&gt;&lt;/td&gt;
&lt;td style="text-align: right"&gt;Authoritative&lt;/td&gt;
&lt;td style="text-align: right"&gt;Yes&lt;/td&gt;
&lt;td style="text-align: right"&gt;No&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;enabled&lt;/code&gt;&lt;/td&gt;
&lt;td style="text-align: right"&gt;Fallback&lt;/td&gt;
&lt;td style="text-align: right"&gt;Yes&lt;/td&gt;
&lt;td style="text-align: right"&gt;Yes&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;&lt;code&gt;shadow&lt;/code&gt; mode was the key. It let us write snapshots, observe latency and storage behavior, and validate that repositories could be initialized and updated without changing task recovery. Only after that did &lt;code&gt;enabled&lt;/code&gt; make snapshots the primary cold-start restore path.&lt;/p&gt;
&lt;p&gt;Even in &lt;code&gt;enabled&lt;/code&gt; mode, we continued writing the legacy Git repository state. That gave us a real rollback path instead of a ceremonial one. If we needed to turn snapshot restores back off, existing tasks would still have the old recovery data available, so rollback would not degrade into &amp;ldquo;resume, but with half the workspace missing.&amp;rdquo;&lt;/p&gt;
&lt;p&gt;The restore path is deliberately asymmetric:&lt;/p&gt;
&lt;pre class="mermaid"&gt;
flowchart TD
Start[Cold start] --&amp;gt; Mode{Snapshot restore enabled?}
Mode --&amp;gt;|No| Legacy[Clone repositories and apply saved patches]
Mode --&amp;gt;|Yes| Restore[Attempt latest Kopia snapshot]
Restore --&amp;gt;|Success| Workspace[Use restored workspace]
Restore --&amp;gt;|Missing or failed| Legacy
Workspace --&amp;gt; Replay[Replay persisted events]
Legacy --&amp;gt; Replay
Replay --&amp;gt; Ready[Resume task]
&lt;/pre&gt;
&lt;p&gt;If a snapshot restore fails, Neo can still fall back to the old repository reconstruction path while that path exists. Snapshot failure is observable, but it does not automatically make a task unrecoverable.&lt;/p&gt;
&lt;p&gt;Checkpointing is also best effort. The runtime returns the final response for the turn, then starts a background checkpoint. If that checkpoint fails, the completed response is still completed. The tradeoff is a bounded durability gap: the conversation may contain a completed turn whose latest filesystem changes are newer than the latest successful snapshot. On the next cold start, Neo restores the latest known-good snapshot rather than pretending a partial checkpoint exists.&lt;/p&gt;
&lt;h2 id="snapshot-the-work-not-the-noise"&gt;Snapshot the work, not the noise&lt;/h2&gt;
&lt;p&gt;An agent workspace has both signal and noise. The signal is user-relevant work: source changes, generated configuration, local commits, tool output, and files the agent may need on the next turn. The noise is everything a package manager, compiler, or test runner can recreate.&lt;/p&gt;
&lt;p&gt;The runtime snapshots existing workspace roots such as:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;/workspace&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;/tmp-workspace&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;/var-workspace&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Regenerable caches and short-lived dependency or build trees are ignored where appropriate. This is an engineering tradeoff for the team operating the agent. Snapshot too much and storage plus request volume become noisy. Snapshot too little and the workspace stops representing what the agent actually saw.&lt;/p&gt;
&lt;p&gt;The boundary we cared about was user-relevant continuity. Source changes, generated configuration, local commits, and tool output should be there after recovery. A dependency cache or build directory that can be recreated may not be. In practice, that means a recovered task might spend extra time reinstalling packages or rebuilding generated artifacts, but it should not lose the work the agent produced or the files it needs to decide what to do next.&lt;/p&gt;
&lt;h2 id="pay-for-isolation-on-purpose"&gt;Pay for isolation on purpose&lt;/h2&gt;
&lt;p&gt;The big cost decision was choosing one repository per task. A shared repository might deduplicate identical content across tasks, but it would also create the wrong cleanup and isolation model. We chose to pay some extra storage cost so the security and lifecycle boundary matched the product boundary: one task, one encrypted snapshot repository.&lt;/p&gt;
&lt;p&gt;With a task-scoped repository, deletion, retention, credential scope, and auditability all line up.&lt;/p&gt;
&lt;p&gt;That means cost management depends on four practical controls:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Incremental snapshots.&lt;/strong&gt; Kopia avoids re-uploading unchanged content inside a task repository.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Ignore policy.&lt;/strong&gt; High-churn generated trees should not dominate every checkpoint.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Retention windows.&lt;/strong&gt; The initial design assumed short default retention for completed tasks, with room for longer retention by policy.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Trusted maintenance.&lt;/strong&gt; Expiring snapshots removes references; repository maintenance later reclaims unreferenced physical data.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;In the planning model, blob-backed Kopia repositories were dramatically cheaper than direct object storage sync for our benchmark workload. The reason was request shape more than raw storage. A sync-based mirror looks simple, but rename-heavy and small-file-heavy workloads turn into large numbers of write, list, and delete operations. Snapshot repositories have their own metadata and maintenance costs, but they preserve history and behave much better under repeated workspace churn.&lt;/p&gt;
&lt;h2 id="make-failures-diagnosable-not-mysterious"&gt;Make failures diagnosable, not mysterious&lt;/h2&gt;
&lt;p&gt;Snapshotting added a new storage system, so we instrumented recovery as a sequence of named phases instead of one &amp;ldquo;persistence failed&amp;rdquo; bucket. For each task, we wanted the logs and metrics to answer:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Was the invocation warm or a cold start?&lt;/li&gt;
&lt;li&gt;Was the task in &lt;code&gt;disabled&lt;/code&gt;, &lt;code&gt;shadow&lt;/code&gt;, or &lt;code&gt;enabled&lt;/code&gt; mode?&lt;/li&gt;
&lt;li&gt;Did task authorization and credential issuance succeed?&lt;/li&gt;
&lt;li&gt;Did Kopia connect to the expected task repository?&lt;/li&gt;
&lt;li&gt;Which workspace roots existed, and which completed?&lt;/li&gt;
&lt;li&gt;Did restore fall back to repository reconstruction?&lt;/li&gt;
&lt;li&gt;Was the restored snapshot older than the latest completed turn?&lt;/li&gt;
&lt;li&gt;Did event replay succeed independently of filesystem recovery?&lt;/li&gt;
&lt;li&gt;Did cleanup expire snapshots or reclaim physical blobs?&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The takeaway is to make recovery observable at the same boundaries where it can fail. A task can restore files but fail to replay model history. It can replay history but fall back to a stale or reconstructed workspace. Those are different incidents with different remediations, so the logs and metrics need to preserve the distinction.&lt;/p&gt;
&lt;h2 id="workers-made-ordering-matter"&gt;Workers made ordering matter&lt;/h2&gt;
&lt;p&gt;Our initial Kopia snapshot path recovered a task when a runtime cold-started. The next step was making the same continuity model work as Neo moved tool execution into managed workers.&lt;/p&gt;
&lt;p&gt;Worker mode was a separate change to how Neo executes tools: instead of every tool running inside the runtime process, a managed worker can own the workspace and execute the tool calls. Because recovery had already moved from &amp;ldquo;rebuild Git repositories&amp;rdquo; to &amp;ldquo;restore the task workspace,&amp;rdquo; the migration did not need a second persistence model. The main new problem was ordering: making sure the tool result and the snapshot pointer advanced together.&lt;/p&gt;
&lt;p&gt;In worker mode, a separate process owns the task workspace, runs tool calls against that workspace, snapshots it after each completed tool call, and reports both the tool result and the latest recovery pointer back to the service.&lt;/p&gt;
&lt;p&gt;A recovery pointer is small metadata: effectively the Kopia snapshot object ID for one logical workspace root. We store the pointer in the service database, while the encrypted snapshot data stays in object storage.&lt;/p&gt;
&lt;p&gt;The important invariant is:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Only the current worker can commit the tool result and the snapshot pointer that becomes the next restore source.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;The service uses a worker lease to keep snapshot pointers ordered. At any point, one worker lease is current for the task. If that worker stalls, dies, or stops heartbeating, the service can assign a replacement worker with a newer lease. If the old worker later wakes up and tries to report a tool result with an older snapshot pointer, the stale lease prevents that pointer from becoming the next restore source.&lt;/p&gt;
&lt;p&gt;That matters because the result and the snapshot describe the same moment in the task. If Neo records &amp;ldquo;the command finished&amp;rdquo; but restores the next worker from an older snapshot, the agent can continue from a workspace that does not contain the files produced by the completed tool call. The lease check makes the persisted tool result and the authoritative recovery pointer advance together.&lt;/p&gt;
&lt;p&gt;Worker snapshots also have failure fuses. A restore failure resets the workspace and marks the next result so the agent knows local filesystem continuity was lost. Repeated snapshot failures disable further snapshot attempts for that worker process rather than blocking tool execution.&lt;/p&gt;
&lt;h2 id="what-users-got-back"&gt;What users got back&lt;/h2&gt;
&lt;p&gt;The user-visible goal was not &amp;ldquo;backups.&amp;rdquo; It was continuity.&lt;/p&gt;
&lt;p&gt;During our internal stress testing for agent tasks, legacy recovery often took around two minutes and repeatedly reported failed repository recovery for extremely large repositories with extensive history. The same tasks restored from Kopia snapshots in roughly half that time with a 100% success rate. More importantly, the snapshot path avoided failure modes the old model could not solve, such as deleted remotes, oversized diffs, and non-repository workspace state.&lt;/p&gt;
&lt;p&gt;Restores became faster and much more stable, but the write path did get slower. Kopia checkpoints cost seconds, not milliseconds, because they preserve the workspace rather than a reconstruction recipe. We mitigated that by making turn-level checkpoints non-blocking: the task can return its completed response, then checkpoint the workspace in the background. For our tasks, a few seconds of asynchronous durability work was the right tradeoff for reliable recovery.&lt;/p&gt;
&lt;h2 id="whats-worth-stealing-from-this-design"&gt;What&amp;rsquo;s worth stealing from this design&lt;/h2&gt;
&lt;p&gt;If you are building long-running agents, the storage question is easy to underestimate. It can look like an implementation detail: clone the repository again, replay the conversation, rehydrate some state, and keep going. But agents do not only produce messages. They produce working directories.&lt;/p&gt;
&lt;p&gt;The most useful lesson for us was to persist the abstraction the product depends on. Neo does not resume into a Git diff. It resumes into a workspace. Once we made the workspace the persistence boundary, the rest of the design became clearer: task-scoped repositories, short-lived credentials, runtime-owned data transfer, trusted cleanup, and an incremental rollout.&lt;/p&gt;
&lt;p&gt;The second lesson was to separate memory from files. Event replay and filesystem restore are both required for a healthy resume, but they are not the same recovery operation. A task can remember the conversation and lose local filesystem continuity. It can restore files and still fail to rebuild model history. Treating those as separate planes made the system easier to reason about, easier to observe, and easier to repair.&lt;/p&gt;
&lt;p&gt;The final lesson was to make rollback part of the design, not an emergency plan. &lt;code&gt;shadow&lt;/code&gt; mode let us learn before relying on snapshots. Fallback kept early restore issues from becoming task-ending incidents. Continuing to write legacy Git state meant &lt;code&gt;enabled&lt;/code&gt; mode was still reversible. The storage engine mattered, but the rollout shape mattered just as much.&lt;/p&gt;
&lt;p&gt;The durable takeaway is this: agent continuity is a product feature, not a backup feature. Users do not care whether a checkpoint succeeded in the abstract. They care whether the agent comes back with the same context, the same files, and the same ability to keep working. Kopia gave us the snapshot primitive, but the real design was choosing the right boundary for recovery.&lt;/p&gt;</description><author>Derek Schaller</author><category>pulumi-neo</category><category>ai-agents</category><category>infrastructure-as-code</category></item><item><title>Pulumi + Gitlab: Better Than Ever</title><link>https://www.pulumi.com/blog/gitlab-better-than-ever/</link><pubDate>Fri, 14 Mar 2025 00:00:00 +0000</pubDate><guid>https://www.pulumi.com/blog/gitlab-better-than-ever/</guid><description>
&lt;img src="https://www.pulumi.com/images/generated/blog/gitlab-better-than-ever/index.png" /&gt;
&lt;p&gt;Pulumi&amp;rsquo;s integration with GitLab has reached new heights with enhancements designed to streamline your infrastructure as code workflows. Today, we&amp;rsquo;re excited to announce several significant improvements to our GitLab integration that make managing cloud infrastructure with Pulumi and GitLab more seamless than ever before: GitLab as a first-class VCS in Pulumi Cloud, enhanced merge request comments, organizational templates in GitLab, and later this year, Pulumi Deployments for GitLab.&lt;/p&gt;
&lt;p&gt;Pulumi and GitLab have long enjoyed a productive partnership. From supporting GitLab sign-in to enabling GitLab CI/CD pipelines for infrastructure deployments, we&amp;rsquo;ve consistently worked to ensure GitLab users can leverage Pulumi&amp;rsquo;s powerful infrastructure as code capabilities within their existing workflows.&lt;/p&gt;
&lt;p&gt;Our GitLab Merge Request integration allows teams to visualize infrastructure changes directly within merge requests, making code reviews more effective and infrastructure changes more transparent.&lt;/p&gt;
&lt;h2 id="whats-new-in-our-gitlab-integration"&gt;What&amp;rsquo;s New in Our GitLab Integration&lt;/h2&gt;
&lt;h3 id="enhanced-merge-request-comments"&gt;Enhanced Merge Request Comments&lt;/h3&gt;
&lt;p&gt;We&amp;rsquo;ve revamped how Pulumi interacts with GitLab by building a Gitlab application to handle the authentication. Previously, only customers using Gitlab as an identity provider to Pulumi Cloud could leverage it. Now, Pulumi provides even more detailed and actionable comments on your merge requests, giving reviewers comprehensive insights into proposed infrastructure changes.&lt;/p&gt;
&lt;p&gt;These comments include:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Streamlined discoverability of infrastructure changes&lt;/li&gt;
&lt;li&gt;Resource-by-resource breakdown of changes&lt;/li&gt;
&lt;li&gt;Direct links to the Pulumi Console for deeper analysis&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;img src="diff-comment.png" alt="Enhanced GitLab Merge Request Comments"&gt;&lt;/p&gt;
&lt;p&gt;To configure the GitLab integration:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Navigate to &lt;strong&gt;Organization Settings → Integrations&lt;/strong&gt; in Pulumi Cloud&lt;/li&gt;
&lt;li&gt;Select GitLab and enter your access token&lt;/li&gt;
&lt;li&gt;Choose which projects or groups should receive Pulumi comments&lt;/li&gt;
&lt;/ol&gt;
&lt;h3 id="gitlab-as-a-first-class-vcs-in-pulumi-cloud"&gt;GitLab as a First-Class VCS in Pulumi Cloud&lt;/h3&gt;
&lt;p&gt;Version control systems are a first-class, configurable concept in Pulumi Cloud. When you select GitLab as your VCS, the Pulumi Cloud interface adapts to show only relevant options and terminology, creating a more intuitive experience for GitLab users.&lt;/p&gt;
&lt;p&gt;This integration enables:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Direct linking to GitLab repositories, branches, and commits&lt;/li&gt;
&lt;li&gt;Automatic detection of Pulumi projects in GitLab repositories&lt;/li&gt;
&lt;li&gt;Streamlined setup for GitLab-based CI/CD workflows, using the CI/CD wizard on a stack&amp;rsquo;s page&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id="organizational-templates-in-gitlab"&gt;Organizational Templates in GitLab&lt;/h3&gt;
&lt;p&gt;Pulumi Cloud now fully supports &lt;a href="https://www.pulumi.com/docs/idp/concepts/organization-templates/#using-organization-templates-within-your-pulumi-organization"&gt;organizational templates&lt;/a&gt; stored in GitLab repositories. This powerful feature allows platform teams to define standardized infrastructure templates that developers across the organization can easily discover and use to create new projects.&lt;/p&gt;
&lt;p&gt;Once a GitLab repository is registered as a template source, Pulumi automatically detects &lt;code&gt;pulumi.yaml&lt;/code&gt; files and surfaces them in the template gallery. These templates can be:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Public or private repositories&lt;/li&gt;
&lt;li&gt;Subject to Pulumi&amp;rsquo;s role-based access control (RBAC)&lt;/li&gt;
&lt;li&gt;Available in both the UI and CLI via &lt;code&gt;pulumi new&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;This integration makes it easier than ever to standardize infrastructure patterns across your organization while leveraging GitLab for version control and collaboration.&lt;/p&gt;
&lt;h2 id="coming-soon-pulumi-deployments-for-gitlab"&gt;Coming Soon: Pulumi Deployments for GitLab&lt;/h2&gt;
&lt;p&gt;&lt;a href="https://www.pulumi.com/docs/deployments/concepts/"&gt;Pulumi Deployments&lt;/a&gt; will also soon support GitLab, enabling fully automated infrastructure provisioning directly from your GitLab pipelines. This integration will provide:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Fully managed compute for running Pulumi operations&lt;/li&gt;
&lt;li&gt;Click to deploy self service workflows in the Pulumi Cloud UI&lt;/li&gt;
&lt;li&gt;Rich new out of the box functionality like Time to Live Stacks, Drift Detection, Click to Deploy, Git Push to Deploy, Review Stacks and Scheduled Deployments.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;All this functionality is available today by specifying the raw git credentials, but we look forward to making the integration frictionless in the future. To learn more about Pulumi Deployments read our &lt;a href="https://www.pulumi.com/docs/deployments/concepts/"&gt;Pulumi Deployments documentation&lt;/a&gt;.&lt;/p&gt;
&lt;h2 id="getting-started"&gt;Getting Started&lt;/h2&gt;
&lt;p&gt;Ready to try these new GitLab integrations? Here&amp;rsquo;s how to get started:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;If you&amp;rsquo;re new to Pulumi, &lt;a href="https://app.pulumi.com/signup"&gt;sign up for a free account&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;For existing users, navigate to your organization settings to configure the GitLab integration&lt;/li&gt;
&lt;li&gt;Check out our updated documentation for detailed setup instructions&lt;/li&gt;
&lt;/ol&gt;
&lt;h2 id="conclusion"&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;With these enhancements, Pulumi&amp;rsquo;s GitLab integration is better than ever, providing a seamless experience for managing infrastructure as code within your GitLab workflows. Learn more in the &lt;a href="https://www.pulumi.com/docs/iac/operations/continuous-delivery/gitlab-ci/"&gt;Pulumi GitLab documentation&lt;/a&gt;. We&amp;rsquo;re committed to continuing to improve this integration based on your feedback.&lt;/p&gt;
&lt;p&gt;Have questions or feedback? Join our &lt;a href="https://slack.pulumi.com/"&gt;Community Slack&lt;/a&gt; to connect with the Pulumi team and other users, or open an issue on our &lt;a href="https://github.com/pulumi/pulumi"&gt;GitHub repository&lt;/a&gt;.&lt;/p&gt;</description><author>Meagan Cojocar</author><author>Derek Schaller</author><category>gitlab</category><category>integrations</category><category>pulumi-cloud</category><category>releases</category><category>features</category></item><item><title>Introducing Pulumi ESC Projects and Environment Tags</title><link>https://www.pulumi.com/blog/esc-projects-environment-tags-launch/</link><pubDate>Thu, 12 Sep 2024 13:00:00 -0400</pubDate><guid>https://www.pulumi.com/blog/esc-projects-environment-tags-launch/</guid><description>
&lt;img src="https://www.pulumi.com/images/generated/blog/esc-projects-environment-tags-launch/index.png" /&gt;
&lt;p&gt;We&amp;rsquo;re thrilled to unveil two new features that will empower you to organize your collection of &lt;a href="https://www.pulumi.com/docs/esc/"&gt;Pulumi ESC&lt;/a&gt; Environments: &lt;b&gt;Projects&lt;/b&gt; and &lt;b&gt;Environment Tags&lt;/b&gt;. Projects offer a structured way to group related environments and Environment Tags allow you to add contextual information to each environment. Together, they offer a powerful way for you to manage, navigate, and collaborate on your secrets and configurations.&lt;/p&gt;
&lt;h2 id="overview-of-pulumi-esc"&gt;Overview of Pulumi ESC&lt;/h2&gt;
&lt;p&gt;Pulumi ESC is a developer-first platform designed to simplify the management of secrets and configurations into collections called &lt;i&gt;&lt;a href="https://www.pulumi.com/docs/esc/concepts/environments/"&gt;environments&lt;/a&gt;&lt;/i&gt;. As a fully managed solution, it offers &lt;a href="https://www.pulumi.com/docs/esc/providers/login/"&gt;dynamic cloud provider credential&lt;/a&gt; resolution, a rich set of &lt;a href="https://www.pulumi.com/docs/esc/providers/"&gt;providers&lt;/a&gt; to retrieve secrets from external platforms, and the ability to use the secrets and configurations you have defined across any surface, including your applications and infrastructure via &lt;a href="https://www.pulumi.com/docs/esc/languages-sdks/"&gt;Multi-language SDKs&lt;/a&gt;, &lt;a href="https://www.pulumi.com/docs/reference/cloud-rest-api/#environments"&gt;REST APIs&lt;/a&gt;, &lt;a href="https://www.pulumi.com/docs/iac/cli/commands/pulumi_env/"&gt;CLI&lt;/a&gt;, &lt;a href="https://www.pulumi.com/registry/packages/pulumiservice/api-docs/environment/"&gt;Pulumi-Service Provider&lt;/a&gt;, and &lt;a href="https://www.pulumi.com/blog/esc-automation-api-pulumi-service-provider-launch/#streamline-automated-workflows-with-automation-api-enhancements"&gt;Automation API&lt;/a&gt;. Like all Pulumi functionality, Pulumi ESC focuses on enhancing both agility and security in modern cloud development.&lt;/p&gt;
&lt;p&gt;As developers have embraced Pulumi ESC to manage their collections of secrets and configurations across many teams and projects, the demand for more sophisticated organizational tools has increased. To address this growing need for more organized and efficient management of cloud environments, we are introducing Projects and Environment Tags in Pulumi ESC.&lt;/p&gt;
&lt;h2 id="introducing-projects"&gt;Introducing Projects&lt;/h2&gt;
&lt;p&gt;Projects offer a structured way to organize environments into logical groupings. Each Project has a distinct name, providing context for the environments within it, and can contain multiple environments with variations, each having a specific purpose. Additionally, Projects help large organizations simplify environment management by mitigating possible environment name conflicts and ensuring changes are made in isolation to the correct environments.&lt;/p&gt;
&lt;p&gt;For example, your organization could have a project named &amp;ldquo;payments-app&amp;rdquo; containing &lt;code&gt;dev&lt;/code&gt;, &lt;code&gt;staging&lt;/code&gt;, and &lt;code&gt;prod&lt;/code&gt; environments within it. Your organization could also have a project called &amp;ldquo;dev-credentials&amp;rdquo; which contains different environments used to provide temporary runtime access to credentials for developers while working.&lt;/p&gt;
&lt;p&gt;We have two additional features as part of this launch:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Grouping and Search&lt;/strong&gt;: You can group environments by either Project name or by Environment Tags, as well as search by them, making it simple to locate and manage your environments.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Environment Clone&lt;/strong&gt;: Using the built-in cloning functionality, you can securely copy environments with sensitive information, preserving either the entire version history or just the latest environment state, depending on your needs.&lt;/li&gt;
&lt;/ul&gt;
&lt;div class="note note-info"&gt;
&lt;div class="icon-and-line"&gt;
&lt;svg xmlns="http://www.w3.org/2000/svg" class="ph-icon ph-icon--fill" fill="currentColor" aria-hidden="true" focusable="false"&gt;&lt;use href="https://www.pulumi.com/icons/sprite.930341d891823cfdd478d54625fe6a3208720ef3ac597420c81d818e3555671d.svg#p-info-fill"/&gt;&lt;/svg&gt;
&lt;div class="line"&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;div class="content"&gt;To use Projects and Environment Tags, please update your &lt;a href="https://www.pulumi.com/docs/iac/cli/"&gt;Pulumi CLI&lt;/a&gt; and &lt;a href="https://www.pulumi.com/docs/esc/languages-sdks/"&gt;SDK&lt;/a&gt; to the latest.&lt;/div&gt;
&lt;/div&gt;
&lt;div class="note note-info"&gt;
&lt;div class="icon-and-line"&gt;
&lt;svg xmlns="http://www.w3.org/2000/svg" class="ph-icon ph-icon--fill" fill="currentColor" aria-hidden="true" focusable="false"&gt;&lt;use href="https://www.pulumi.com/icons/sprite.930341d891823cfdd478d54625fe6a3208720ef3ac597420c81d818e3555671d.svg#p-info-fill"/&gt;&lt;/svg&gt;
&lt;div class="line"&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;div class="content"&gt;&lt;p&gt;Environments will now need to be referenced by specifying the project as part of the environment identifier. Any existing environments within your organization will be available in the &lt;code&gt;default&lt;/code&gt; project. Projects are fully backward compatible - all your existing programs, imports and CLI commands will work without requiring any changes at the moment. Note that only environments within the &lt;code&gt;default&lt;/code&gt; project will be backwards compatibly and resolve without a project prefix.&lt;/p&gt;
&lt;p&gt;Users will be able to create new environments in the default project until &lt;b&gt;Nov 30, 2024&lt;/b&gt;. Starting Dec 1, 2024, users will not be able to create new environments in the default project.&lt;/p&gt;
&lt;p&gt;Starting &lt;b&gt;Apr 1, 2025&lt;/b&gt;, users will not be able to edit their environments within the ‘default’ Project. We recommend customers migrate their existing environments to user-created projects at the earliest to take advantage of new Pulumi ESC capabilities.&lt;/p&gt;
&lt;p&gt;Moving environments out of the &lt;code&gt;default&lt;/code&gt; project is as easy as cloning the environment into a new project and updating any references to it. An example clone command that preserves all environment history, tags, and team access looks as follows&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-bash" data-lang="bash"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; $ pulumi env clone default/dev your-project/dev --preserve-history --preserve-env-tags --preserve-rev-tags --preserve-access
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;/div&gt;
&lt;/div&gt;
&lt;p&gt;We offer numerous ways for you to get started with Projects via the Pulumi Web Console, &lt;a href="https://www.pulumi.com/docs/esc/languages-sdks/"&gt;SDK&lt;/a&gt;, &lt;a href="https://www.pulumi.com/docs/reference/cloud-rest-api/#environments"&gt;REST API&lt;/a&gt;, &lt;a href="https://www.pulumi.com/docs/iac/cli/commands/pulumi_env/"&gt;CLI&lt;/a&gt; and &lt;a href="https://www.pulumi.com/registry/packages/pulumiservice/api-docs/environment/"&gt;Pulumi-Service Provider&lt;/a&gt; - choose what best fits your workflow. Here is one example using the Pulumi CLI to create and list environments within a project:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-bash" data-lang="bash"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; $ pulumi env init cloud-pe/demo
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; Environment created: dschaller/cloud-pe/demo
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; $ pulumi env ls -p cloud-pe
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; cloud-pe/demo
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt; pulumi/cloud-pe/demo
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;h3 id="breaking-changes"&gt;Breaking Changes&lt;/h3&gt;
&lt;p&gt;With the introduction of Projects, all existing environments part of the &lt;code&gt;default&lt;/code&gt; project will continue to work without requiring any changes. However, when moving existing environments out of the default project you may notice some differences that we have outlined below.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;If your environment was interpolating the &lt;code&gt;context.currentEnvironment.name&lt;/code&gt; or &lt;code&gt;context.rootEnvironment.name&lt;/code&gt; from &lt;a href="https://www.pulumi.com/docs/esc/concepts/environments/#pulumi-contextual-information"&gt;Contextual information&lt;/a&gt;, this value will now include the project for any environments outside of the &lt;code&gt;default&lt;/code&gt; project.&lt;/li&gt;
&lt;li&gt;If you were using any of the &lt;a href="https://www.pulumi.com/docs/esc/concepts/environments/#using-secrets-providers-and-oidc"&gt;secrets providers with OIDC&lt;/a&gt;, you will need to update both the subject and audience if a trust relationship has been set up. The subject will be either &lt;code&gt;pulumi:environments:org:&amp;lt;organization name&amp;gt;:env:&amp;lt;project name&amp;gt;/&amp;lt;environment name&amp;gt;&lt;/code&gt; if no &lt;code&gt;subjectAttributes&lt;/code&gt; are specified or if the &lt;code&gt;currentEnvironment.name&lt;/code&gt; subject attribute is specified it will now resolve to &lt;code&gt;currentEnvironment.name:&amp;lt;project name&amp;gt;/&amp;lt;environment name&amp;gt;&lt;/code&gt;. The audience will be the provider&amp;rsquo;s platform name (&lt;code&gt;aws&lt;/code&gt;, &lt;code&gt;azure&lt;/code&gt;, or &lt;code&gt;gcp&lt;/code&gt;) followed by the org name of the environment (e.g. &lt;code&gt;aws:pulumi&lt;/code&gt;).&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="introducing-environment-tags"&gt;Introducing Environment Tags&lt;/h2&gt;
&lt;p&gt;Using Environment Tags, teams can now assign any number of custom tags to each environment within Pulumi ESC. These tags serve as contextual identifiers that can streamline workflows by enabling you to group and search across environments based on specific criteria such as teams, project stages, deployment environments, compliance requirements, or geographic locations. This multi-dimensional approach provides unparalleled flexibility and control, ensuring your Pulumi ESC environments can scale seamlessly with your organizations evolving needs.&lt;/p&gt;
&lt;p&gt;For example, suppose your organization has multiple environments across different geographic regions. You can use Environment Tags like &amp;lsquo;region: us-east&amp;rsquo; or &amp;lsquo;region: eu-central-1&amp;rsquo; to quickly filter environments by region. Similarly, for security audits, you might tag environments with ‘Compliance: SOC2’ or ‘Compliance: GDPR’ to easily find relevant environments and ensure they are compliant.&lt;/p&gt;
&lt;p&gt;You can use the Pulumi Web Console and the &lt;a href="https://www.pulumi.com/docs/iac/cli/commands/pulumi_env/"&gt;CLI&lt;/a&gt; to create and manage Environment Tags. Here is an example of using the CLI to add and list Environment Tags.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-bash" data-lang="bash"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;$ pulumi env tag cloud-pe/demo region us-east-1
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;Name: region
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;Value: us-east-1
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;Last updated at 2024-09-10 11:03:05.708 -0700 PDT by Derek &amp;lt;dschaller&amp;gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;$ pulumi env tag ls cloud-pe/demo
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;Name: region
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;Value: us-east-1
&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;Last updated at 2024-09-10 11:03:05.708 -0700 PDT by Derek &amp;lt;dschaller&amp;gt;
&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;h2 id="conclusion"&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;Pulumi ESC Projects and Environment Tags are powerful additions that bring order and clarity to your secrets and configuration to even the most complex infrastructure landscapes. By enabling you to organize, group, and search for your environments with ease, we&amp;rsquo;re empowering you to scale your infrastructure with high confidence and flexibility.&lt;/p&gt;
&lt;p&gt;We&amp;rsquo;re excited to see how you leverage these new capabilities. Check out the docs to learn more about Pulumi ESC Projects and Environment Tags and start organizing your environments today!&lt;/p&gt;
&lt;p&gt;As always, please share your &lt;a href="https://github.com/pulumi/esc/issues/new/choose"&gt;feedback&lt;/a&gt; on how we can further improve Pulumi ESC to suit your needs.&lt;/p&gt;</description><author>Derek Schaller</author><author>Arun Loganathan</author><category>esc</category><category>secrets</category><category>features</category></item></channel></rss>