Pulumi Blog: Engin Diri

Incident Response as Code: Managing PagerDuty with Pulumi

Engin Diri — Mon, 20 Jul 2026 00:00:00 +0000

You usually find out in the postmortem: the alarm fired, but it paged a schedule nobody was on anymore. Or the service had been running in production for three months before anyone created the matching PagerDuty service, so the first person to notice the outage was a customer. The infrastructure was code, reviewed and versioned. The incident response setup was forty clicks in a web UI, done once, by someone who has since changed teams.

PagerDuty’s own engineering team has been making the case for managing PagerDuty as code for years, and the OneUptime folks recently published a hands-on guide to the PagerDuty Terraform provider that walks the whole resource catalog. I liked both posts, and you probably know where this is going: I want the same thing in TypeScript, with the on-call setup and the infrastructure it protects living in the same Pulumi program. Teams, schedules, escalation policies, services, and the CloudWatch alarm that pages you: one pulumi up.

What you are building

The demo models a small platform team that owns a checkout service on AWS. On the PagerDuty side: a team with three members, a weekly on-call rotation, an escalation policy with two levels, and a service with a CloudWatch integration. On the AWS side: a dead-letter queue, an alarm that fires when a message lands in it, and an SNS topic that forwards the alarm to PagerDuty. The interesting part is where the two meet: the integration key PagerDuty mints flows straight into the SNS subscription without ever touching a clipboard.

flowchart LR
DLQ["SQS dead-letter queue"] --> Alarm["CloudWatch alarm"]
Alarm --> SNS["SNS topic"]
SNS -->|"HTTPS subscription<br/>with integration key"| Int["CloudWatch integration"]
Int --> Svc["PagerDuty service<br/>Checkout API"]
Svc --> EP["Escalation policy"]
EP -->|"level 1"| Sched["Weekly rotation<br/>Lena, Jonas, Mia"]
EP -->|"level 2, after 15 min"| Lead["Team lead"]

Everything on both sides of that diagram is one TypeScript program. If you would rather write Python, Go, C#, Java, or YAML, the Pulumi PagerDuty provider supports all of them. It is bridged from PagerDuty’s official Terraform provider (which started life as a community project by Alexander Hellbom before PagerDuty adopted it), so the resource coverage is identical.

Prerequisites

Before getting started, ensure you have:

Pulumi CLI installed and configured
A Pulumi Cloud account
A PagerDuty account with admin access; a free trial works fine
An AWS account for the alarm wiring in the second half
Node.js 18+ for the TypeScript program

The provider and the token

Create a new project and add the PagerDuty provider:

pulumi new aws-typescript --name pagerduty-oncall
npm install @pulumi/pagerduty

The provider authenticates with a REST API key. In PagerDuty, navigate to Integrations > API Access Keys and create a new key. This is the account-level REST API key, not a user token: user tokens are scoped to what one person can see, and you want the automation to manage the whole account.

I store the key in Pulumi ESC so it never lands in a .env file, shell history, or agent context:

pulumi env init <your-org>/pagerduty-demo/dev

Add the token to the environment definition:

values:
 pagerduty:
 token:
 fn::secret: <your-rest-api-key>
 pulumiConfig:
 pagerduty:token: ${pagerduty.token}

Then reference the environment from your stack in Pulumi.dev.yaml:

environment:
 - <your-org>/pagerduty-demo/dev

In a hurry? pulumi config set pagerduty:token --secret does the same job for a single stack, and the provider also reads the PAGERDUTY_TOKEN environment variable. If your account lives on the EU service region, additionally set pagerduty:serviceRegion to eu.

Teams, users, and memberships

Everything in PagerDuty hangs off teams and users, so that is where the program starts:

import * as pagerduty from "@pulumi/pagerduty";

const platformTeam = new pagerduty.Team("platform-team", {
 name: "Platform Engineering",
 description: "Owns the checkout path and the shared platform",
});

const lena = new pagerduty.User("lena", {
 name: "Lena Fischer",
 email: "lena@example.com",
});

const jonas = new pagerduty.User("jonas", {
 name: "Jonas Weber",
 email: "jonas@example.com",
});

const mia = new pagerduty.User("mia", {
 name: "Mia Schneider",
 email: "mia@example.com",
});

new pagerduty.TeamMembership("lena-platform", {
 userId: lena.id,
 teamId: platformTeam.id,
 role: "manager",
});

new pagerduty.TeamMembership("jonas-platform", {
 userId: jonas.id,
 teamId: platformTeam.id,
 role: "responder",
});

new pagerduty.TeamMembership("mia-platform", {
 userId: mia.id,
 teamId: platformTeam.id,
 role: "responder",
});

Three users, one team, three memberships. Reorgs stop being an afternoon of UI archaeology and become a pull request where the diff shows exactly who moved where.

If your users already come from SSO or SCIM provisioning (they usually do), don’t create them in the program. Look them up instead with the getUser data source: pagerduty.getUserOutput({ email: "lena@yourcompany.com" }). And if your account restricts sign-ups to your company’s email domain, the API enforces that too, so swap the example.com addresses for ones on your own domain when trying this out.

A schedule people can live with

The on-call schedule is where incident response meets real life, so it deserves more care than a demo usually gives it. If you have seen PagerDuty schedules in Terraform before, you probably remember the Schedule resource and its layers. That resource is deprecated in the current provider because it speaks the legacy v1 schedules API; its successor Schedulev2 uses PagerDuty’s new model built on rotations, events, and RFC 5545 recurrence rules:

const primary = new pagerduty.Schedulev2("primary", {
 name: "Platform Primary On-Call",
 description: "Weekly hand-off rotation for the platform team",
 timeZone: "Europe/Berlin",
 teams: [platformTeam.id],
 rotations: [{
 events: [{
 name: "Weekly rotation",
 startTime: "2026-08-03T09:00:00+02:00",
 endTime: "2026-08-10T09:00:00+02:00",
 effectiveSince: "2026-08-03T07:00:00Z",
 recurrences: ["RRULE:FREQ=WEEKLY;BYDAY=MO"],
 assignmentStrategies: [{
 type: "rotating_member_assignment_strategy",
 shiftsPerMember: 1,
 members: [
 { type: "user_member", userId: lena.id },
 { type: "user_member", userId: jonas.id },
 { type: "user_member", userId: mia.id },
 ],
 }],
 }],
 }],
});

Read it from the inside out. The event is one shift: it starts Monday at 09:00 Berlin time and ends the following Monday at 09:00, and the RRULE makes it recur weekly. The assignment strategy hands each occurrence to the next member in the list, so Lena, Jonas, and Mia each take a full week, round robin. effectiveSince tells PagerDuty when this configuration starts producing shifts. Business-hours-only coverage falls out of the same model: shorten the shift window and change the recurrence to RRULE:FREQ=WEEKLY;BYDAY=MO,TU,WE,TH,FR.

The escalation policy

The escalation policy reads top to bottom, the same way the incident will play out:

const escalation = new pagerduty.EscalationPolicy("platform-escalation", {
 name: "Platform Escalation Policy",
 numLoops: 2,
 teams: platformTeam.id,
 rules: [
 {
 escalationDelayInMinutes: 15,
 targets: [{
 type: "schedule_reference",
 id: primary.id,
 }],
 },
 {
 escalationDelayInMinutes: 10,
 targets: [{
 type: "user_reference",
 id: lena.id,
 }],
 },
 ],
});

Page whoever the primary schedule says is on call. If nobody acknowledges within 15 minutes, page Lena directly. If the whole chain runs dry, numLoops: 2 starts it over twice more before PagerDuty gives up. When someone asks what actually happens when this fires at 3 AM, the answer is a dozen lines of reviewed code, not a screenshot of a UI that may or may not be current.

The service

A PagerDuty service is the unit incidents attach to, and it is where the escalation policy gets wired in:

const checkoutService = new pagerduty.Service("checkout", {
 name: "Checkout API",
 escalationPolicy: escalation.id,
 autoResolveTimeout: "14400",
 acknowledgementTimeout: "600",
});

Think of a service as the thing that can break, like the checkout API or the payments queue behind it. Every incident in PagerDuty opens against exactly one service, and the service decides what happens next. The escalation policy answers who gets paged. acknowledgementTimeout re-triggers an incident that was acknowledged and then went quiet for ten minutes, so a sleepy “ack” cannot make a problem disappear. And autoResolveTimeout closes an incident that has sat open for four hours, which keeps the dashboard honest when everyone forgot to press resolve. Map services one-to-one to the things your team actually owns, and the service directory turns into an honest inventory of your system, each entry with an owner attached.

So far this is a nicer syntax for what the Terraform guides already do. Here is where it starts earning its keep. Remember the service from the top of this post that ran for three months without a pager? The place that failure usually hides is a dead-letter queue, where bad news piles up quietly. This is what it looks like when the queue, the alarm, and the pager live in one program:

import * as aws from "@pulumi/aws";
import * as pulumi from "@pulumi/pulumi";

const cloudwatch = pagerduty.getVendorOutput({ name: "Amazon CloudWatch" });

const checkoutIntegration = new pagerduty.ServiceIntegration("checkout-cloudwatch", {
 name: "Amazon CloudWatch",
 service: checkoutService.id,
 vendor: cloudwatch.id,
});

const alarmTopic = new aws.sns.Topic("checkout-alarms");

new aws.sns.TopicSubscription("checkout-alarms-to-pagerduty", {
 topic: alarmTopic.arn,
 protocol: "https",
 endpoint: pulumi.interpolate`https://events.pagerduty.com/integration/${checkoutIntegration.integrationKey}/enqueue`,
 endpointAutoConfirms: true,
});

const dlq = new aws.sqs.Queue("checkout-dlq");

new aws.cloudwatch.MetricAlarm("checkout-dlq-alarm", {
 alarmDescription: "Messages are landing in the checkout dead-letter queue",
 namespace: "AWS/SQS",
 metricName: "ApproximateNumberOfMessagesVisible",
 dimensions: {
 QueueName: dlq.name,
 },
 statistic: "Maximum",
 period: 60,
 evaluationPeriods: 1,
 threshold: 1,
 comparisonOperator: "GreaterThanOrEqualToThreshold",
 alarmActions: [alarmTopic.arn],
 okActions: [alarmTopic.arn],
});

Follow the integration key. PagerDuty mints it when the ServiceIntegration is created, and a few lines later it is interpolated into the SNS subscription endpoint. In the two-tool world, this is the step where you open the PagerDuty UI, copy the integration URL, and paste it into a variables file or a wiki page, which is exactly the step that gets skipped at 5 PM on a Friday. Here it cannot be skipped, because it is a dependency: Pulumi will not create the subscription before the integration exists.

endpointAutoConfirms: true matters because PagerDuty answers the SNS subscription handshake on its own; without it, the subscription sits in “pending confirmation” forever. And since the alarm sends its OK state to the same topic, PagerDuty resolves the incident automatically once the queue drains.

Tearing down the service? The alarm, the topic, the subscription, and the PagerDuty service all go in the same pulumi destroy, which spares you the orphaned PagerDuty service that keeps paging people about infrastructure that no longer exists. I have seen that one too.

Azure and Google Cloud, same trick

Nothing about this is AWS-specific. It is always the same two steps: PagerDuty mints a key, the monitoring side consumes it. On Google Cloud, Cloud Monitoring has a native PagerDuty channel type, and its serviceKey is the integration key of an Events API v2 integration:

import * as gcp from "@pulumi/gcp";

const gcpIntegration = new pagerduty.ServiceIntegration("checkout-gcp", {
 name: "Google Cloud Monitoring",
 service: checkoutService.id,
 type: "events_api_v2_inbound_integration",
});

new gcp.monitoring.NotificationChannel("checkout-pagerduty", {
 displayName: "PagerDuty: Checkout API",
 type: "pagerduty",
 sensitiveLabels: {
 serviceKey: gcpIntegration.integrationKey,
 },
});

On Azure, an Azure Monitor action group with a webhook receiver does the same job, pointed at the familiar enqueue URL (with resourceGroup being an azure.resources.ResourceGroup from earlier in your program):

import * as azure from "@pulumi/azure-native";

const azureVendor = pagerduty.getVendorOutput({ name: "Microsoft Azure" });

const azureIntegration = new pagerduty.ServiceIntegration("checkout-azure", {
 name: "Microsoft Azure",
 service: checkoutService.id,
 vendor: azureVendor.id,
});

new azure.monitor.ActionGroup("checkout-pagerduty", {
 resourceGroupName: resourceGroup.name,
 location: "global",
 groupShortName: "checkout",
 enabled: true,
 webhookReceivers: [{
 name: "pagerduty",
 serviceUri: pulumi.interpolate`https://events.pagerduty.com/integration/${azureIntegration.integrationKey}/enqueue`,
 useCommonAlertSchema: true,
 }],
});

The location: "global" is not a placeholder. Action groups are a global service, and Azure rejects regular regions like westeurope for them; setting it explicitly makes that intent obvious in the code.

Same program, same key handoff, different cloud. If your platform spans all three, the PagerDuty side does not care.

Breaking a Fargate payment service on purpose

Everything so far wires configuration together, but I did not want to publish wiring I had never watched fire. So I put it to the test: a companion project that deploys a small payment service on AWS Fargate together with its entire failure chain, and then breaks it on purpose.

The worker is deliberately boring: the plain AWS CLI container image running a shell loop. It receives payment messages from an SQS queue, processes anything that has an amount, and refuses everything else. A refused message goes back to the queue, and after three failed receives SQS moves it to the dead-letter queue. From there, the chain from earlier in this post takes over: DLQ alarm, SNS topic, CloudWatch integration, escalation policy, phone.

A valid payment flows through in seconds:

./scripts/send-payment.sh
# worker log: processed payment: {"orderId":"4711","amount":"49.99","currency":"EUR"}

A payment without an amount poisons the queue:

./scripts/inject-failure.sh

Here is the timeline from my test run, as the test script observed it through the AWS and PagerDuty APIs:

Elapsed	What happened
0:00	Poison payment injected
3:45	Third receive failed, message redriven to the DLQ, alarm in `ALARM`
3:49	PagerDuty incident triggered on Payment Service
3:50	Recovery: dead-letter queue purged
6:48	Alarm back to `OK`
6:49	Incident resolved

Under four minutes from a bad message to a page, and I never touched the PagerDuty UI in either direction; the incident opened and closed itself off the alarm’s state transitions. The incident timeline spells it out: automatically triggered and assigned, then resolved through the integration API.

The full program, the worker loop, and the inject and recover scripts are in the companion repo:

github.com/dirien/pulumi-pagerduty-fargate-demo

Growing without the drift: a reusable component

Whether any of this stays consistent as your organization grows comes down to defaults. If creating a properly paged service takes a checklist, people will skip items. If it takes a component, they won’t, because the component is less work than the checklist. So wrap the service, the integration, and the alarm topic into one:

export class OnCallService extends pulumi.ComponentResource {
 public readonly service: pagerduty.Service;
 public readonly alarmTopic: aws.sns.Topic;

 constructor(name: string, args: {
 displayName: string;
 escalationPolicy: pulumi.Input<string>;
 }, opts?: pulumi.ComponentResourceOptions) {
 super("acme:oncall:OnCallService", name, {}, opts);

 this.service = new pagerduty.Service(name, {
 name: args.displayName,
 escalationPolicy: args.escalationPolicy,
 autoResolveTimeout: "14400",
 acknowledgementTimeout: "600",
 }, { parent: this });

 const cloudwatch = pagerduty.getVendorOutput({ name: "Amazon CloudWatch" });

 const integration = new pagerduty.ServiceIntegration(name, {
 name: "Amazon CloudWatch",
 service: this.service.id,
 vendor: cloudwatch.id,
 }, { parent: this });

 this.alarmTopic = new aws.sns.Topic(name, {}, { parent: this });

 new aws.sns.TopicSubscription(name, {
 topic: this.alarmTopic.arn,
 protocol: "https",
 endpoint: pulumi.interpolate`https://events.pagerduty.com/integration/${integration.integrationKey}/enqueue`,
 endpointAutoConfirms: true,
 }, { parent: this });

 this.registerOutputs({});
 }
}

And now every new service ships with a pager attached. The checkout instance here replaces the hand-rolled service, integration, and topic from earlier:

const checkout = new OnCallService("checkout", {
 displayName: "Checkout API",
 escalationPolicy: escalation.id,
});

const payments = new OnCallService("payments", {
 displayName: "Payments API",
 escalationPolicy: escalation.id,
});

App teams get one contract: point your alarms at alarmTopic. The escalation policy, the timeouts, the integration, the SNS wiring, all of it becomes the paved road instead of a checklist item. This is the same components-and-templates story I keep telling for compute and networking; incident response just turns out to be another thing a platform team can pave.

Importing what you already clicked together

Nobody starts from an empty PagerDuty account. The realistic adoption path is pulumi import, resource by resource:

pulumi import pagerduty:index/team:Team platform-team PXXXXXX
pulumi import pagerduty:index/service:Service checkout PYYYYYY

The IDs are in the URL when you open a team or service in the PagerDuty web UI, and pulumi import writes the matching code for you, so bringing an existing setup under management is mostly a copy-paste loop. The import guide covers the mechanics.

One habit matters after that, and PagerDuty’s engineering post makes the same point about Terraform: once a resource is managed as code, stop editing it in the UI. Changes made in the web console are drift, and you will want to run pulumi refresh (or let drift detection do it on a schedule) to catch them before the 3 AM page tests your assumptions for you.

Conclusion

The pitch for managing PagerDuty as code has less to do with YAML versus HCL versus TypeScript than with the gap between two systems that each hold half the truth. Your infrastructure code knows a queue exists; PagerDuty knows who gets paged when it backs up. Keep them in separate tools and that gap is maintained by hand, which means it is maintained until the person who cared about it leaves.

Putting both sides in one Pulumi program closes the gap structurally. The integration key becomes a dependency edge instead of a wiki entry, and forgetting the escalation policy is not even possible, because escalationPolicy is a required property and the program will not compile without it. That is the version of “nothing gets forgotten” I am willing to rely on.

To go further:

The PagerDuty provider in the Pulumi Registry covers the full resource catalog: event orchestrations, maintenance windows, response plays, and more.
Pulumi ESC keeps the API token out of your shell history.
Coming from the Terraform provider? Your resource knowledge maps one-to-one, and the migration guide covers the rest.

If you build something with this, or if your on-call setup has a horror story that code would have prevented, come tell me in the Pulumi Community Slack.

Knowledge as Code: The Memory File Just Got a Spec

Engin Diri — Tue, 14 Jul 2026 00:00:00 +0000

Five weeks ago I wrote that the least glamorous piece of an agent loop is also the one that decides whether it compounds: memory. A markdown file outside the context window that holds what is done, what is next, and what was learned, because the model forgets all of it between runs. Write the memory file before the loop.

What I left open, because there was nothing to point at, was the format. My memory file looked nothing like yours, and neither of our agents could read the other’s.

Three days after that post went live, Google shipped an answer.

The pattern everyone copied

Andrej Karpathy published a gist in April he called the LLM wiki, and it collected thousands of stars and forks. It’s meant to be pasted straight into a coding agent. Instead of indexing your documents for RAG and re-deriving answers from raw text on every query, the agent builds a wiki and keeps it current: interlinked markdown pages, an index.md with a one-line summary per page, a log.md recording every change, entity pages that grow as sources come in. Drop in a meeting transcript and the agent reads it, updates a dozen existing pages, fixes the cross-references, and appends to the log in one pass.

It took off for the same reason wikis usually die. A knowledge base is valuable in exact proportion to the bookkeeping nobody wants to do: summarizing, linking, reconciling contradictions, pruning stale claims. Karpathy’s line, which Google now quotes back in its own announcement, is that LLMs “don’t get bored, don’t forget to update a cross-reference, and can touch 15 files in one pass.” The human curates sources and asks questions. The agent does the janitorial work that made every previous wiki rot.

A wiki only your agent can read

Then everyone built one, and every one is a dialect. Mine links related pages in the frontmatter; yours links them at the bottom of the body. Mine has a tags field; yours calls it categories. None of this matters while the wiki serves one person, because the schema lives in your CLAUDE.md and your agent reads it on every run.

It matters the moment the wiki has to travel. Hand your knowledge base to a teammate and their agent starts guessing at your conventions, or misses half the structure entirely. A platform team that wants one shared wiki, queried independently by everyone’s agents, has no format to agree on. A knowledge base you can’t hand to someone else’s agent is a silo of one. And a team’s collective knowledge should outlive any single person’s markdown habits.

What Google actually shipped: the Open Knowledge Format

On June 12, two tech leads in Google Cloud’s data analytics engineering org announced the Open Knowledge Format, with a spec and reference tooling on GitHub. Strip the branding and OKF v0.1 is a formalization of the Karpathy pattern:

A bundle is a directory tree of markdown files. Two filenames are reserved: index.md for progressive disclosure and log.md for update history. Everything else is a concept document.
Every concept carries YAML frontmatter with exactly one required field: type. Five more are recommended: title, description, resource, tags, timestamp. Producers can add anything; consumers must preserve what they don’t understand.
A markdown link from one concept to another asserts a relationship. The prose around the link says what kind.

The whole spec fits on a page, and that’s deliberate. The announcement names three principles: minimally opinionated; producers and consumers independently swappable; a format rather than a platform. No required SDK, no compression scheme, no new runtime. Reading a bundle is cat; distributing one is git clone. The repo backs it up with reference tooling: an enrichment agent that drafts concept docs from BigQuery datasets, and a static visualizer that renders a bundle as a graph. Three sample bundles, built by that same agent, give you something to copy from.

The critique is the feature

The pushback writes itself: there is almost nothing here. Folder indexes and five suggested frontmatter fields, on top of a pattern the community already had. True. Also the point.

The standards that stick are embarrassingly small. MCP didn’t model your tools; it standardized the socket they plug into. AGENTS.md fixed nothing but a filename, and that was enough for Neo to read the same conventions file as every other agent. Agent Skills gave procedure a place to live, and docs sites settled on /llms.txt for published content. Knowledge was the layer still missing its boring little standard. Whether OKF specifically is the one that survives matters less than the shape it commits to: markdown, frontmatter, git. Every other layer of the agent stack already landed there, so that bet is safe.

Knowledge as code

Platform teams should care about this before anyone else. Infrastructure as code captures the what: the resources, the config, the dependency graph. It has never captured the why. The runbook for rotating credentials, the decision record explaining the pinned CNI version, who owns the cluster, what the incident in November taught you. That knowledge lives in a wiki behind an API, a shared drive, a Slack thread, and the heads of two senior engineers. Google’s announcement frames this fragmentation as the problem: agents reassembling answers from systems that each speak a proprietary format.

OKF’s answer is a move platform engineers will recognize, because it’s the same one that produced IaC: put it in git, make it diffable, review changes in pull requests. Knowledge as code, sitting next to the infrastructure as code it explains:

platform/
├── index.md
├── log.md
├── services/
│ ├── index.md
│ ├── checkout-api.md
│ └── payments-worker.md
├── runbooks/
│ ├── index.md
│ └── rotate-database-credentials.md
└── decisions/
├── index.md
└── why-we-pin-the-vpc-cni-version.md

The recommended resource field takes a URI naming the asset a concept describes, and a Pulumi URN is exactly that:

---
type: runbook
title: Rotate the payments database credentials
description: Zero-downtime credential rotation for the payments Postgres instance.
resource: urn:pulumi:prod::payments::aws:rds/instance:Instance::payments-db
tags: [payments, postgres]
timestamp: 2026-07-01T09:30:00Z
---

Now the runbook names the exact resource it operates on. An agent planning a change to that database can walk from the URN to the runbook to the decision record that explains the constraint, before it proposes anything. In the loop post I said skills are intent written down, the conventions an agent reads instead of guessing. A knowledge bundle is the other half: experience written down. Tools give the loop hands, skills give it habits, and the bundle is what it gets to remember.

Before writing any of that down, I ran the test. A fresh agent got two inputs, the spec and this bundle, and one question: which services touch the payments database, and how do I rotate its credentials without downtime? It read six files and answered correctly, down to why rotating a second time too soon is the dangerous move. What sold me was the file it didn’t read. It left the CNI decision record closed, because the cross-links mark that page as a cluster concern, not a database one.

The bundle is public, exactly the tree above. Clone it, hand your agent the spec, and ask your own questions:

github.com/dirien/pulumi-platform-okf-bundle

What a standard won’t do for you

Three caveats, so nobody mistakes this for magic.

It formats knowledge; it doesn’t create it. The warning from the loop post applies with no edits: memory is what lets a loop compound, and slop compounds right alongside it. A wrong runbook in a beautifully conformant bundle is still a wrong runbook, now served to every agent on the team with confidence. The leverage isn’t the format. It’s that knowledge changes finally go through review like code changes.

v0.1 is a draft, and the name may not survive. Google says so itself: the announcement calls v0.1 “a starting point, not a finished standard.” OKF could lose to a better spec next year. I’d start anyway, because converting markdown with frontmatter into markdown with slightly different frontmatter is the cheapest migration you’ll ever run, and it’s precisely the kind of mechanical pass an agent finishes in an afternoon. The shape is the commitment. The name is a detail.

Conformance is a floor, not hygiene. The spec requires consumers to tolerate broken links, missing indexes, and unknown types. That tolerance keeps readers working, and it also means nothing forces your bundle to stay healthy. Karpathy’s gist included a lint pass for a reason: contradictions, orphaned pages, stale claims. Keep it, and run it the way you lint code, on a schedule, with findings that reach a human.

Where to start

Pick the knowledge that already hurts. The runbook nobody can find during an incident, the onboarding doc that’s wrong in ways only one person knows. One domain, not the whole org.
Let the agent do the conversion. Paste the spec into your coding agent, point it at the existing folder of docs, and have it refactor them into a bundle. This is exactly the bookkeeping LLMs are good at, and it works as well on a wiki export as on a fresh start.
Put the bundle where the code lives. Same repo or a sibling, but in git, behind pull requests. The day a knowledge change gets a review comment is the day the bundle becomes trustworthy.
Wire it into the loop. Agents read index.md first and drill down only when a concept earns it, which keeps the context window small. The memory file I told you to write before the loop now has a format, and every agent you run, in whatever harness you built, can read the same one.

The advice from June holds: write the memory file before the loop. What changed is that the file no longer has to be a private dialect. There’s now a one-page, git-native spec for the layer your agents think with, and the smallness that makes it look trivial is the property that lets it spread. The loop does the typing. The wiki does the remembering. And the remembering is now something you can put in a pull request and hand to the next agent, like everything else you ship.

Wire Pulumi context into your agents

Fully Automated AI Inference on AWS, Azure, and Google Cloud with Pulumi

Engin Diri — Tue, 30 Jun 2026 00:00:00 +0000

Putting Ollama on a cloud GPU is something I keep coming back to. A while ago I wrote up running open-source LLMs on an AWS EC2 box with Ollama and Pulumi, and the shape never really changes: a GPU instance, a model server, and a firewall rule in front. Infrastructure as code earned its place by making that kind of setup predictable and repeatable, and AI infrastructure is no exception. A GPU box serving a model is still a VM, a disk, and a firewall rule, and it should be declared like one.

Thorsten Hans made exactly that case in his Akamai post, Fully Automated AI Infrastructures with Terraform and Akamai Cloud, which stands up a single GPU instance on Linode, installs the drivers, runs Ollama, and pulls a model, with no manual steps after terraform apply.

I liked the shape of it, so this post ports the same idea to Pulumi and runs it across AWS, Azure, and Google Cloud instead of one. The result is one program shape per cloud: a single pulumi up brings up a GPU box that installs its own driver, runs Ollama, and pulls a model with no manual steps, and a single pulumi destroy takes it back down. Along the way it drops the two imperative bits the Terraform version leans on: a static access token sitting in an environment variable, and a null_resource running a shell loop to wait for the model. The first becomes an OIDC login from a Pulumi ESC environment, so no long-lived key lives anywhere. The second turns out not to be a resource at all.

What you are building

Strip away the per-cloud naming and every version of this is the same three things: a GPU virtual machine, a firewall in front of it, and a cloud-init script that turns a bare Ubuntu box into a running inference server. The model serving runs on Ollama, which exposes an HTTP API on port 11434 and keeps the model resident in GPU memory between requests.

flowchart LR
Dev([Your machine / curl]) -->|"HTTP :11434"| FW["Firewall / security group<br/>(allow 11434, optional 22)"]
FW --> VM["GPU VM (Ubuntu 24.04)<br/>NVIDIA driver + Ollama"]
VM -->|"resident in GPU memory"| Model["qwen2.5:14b"]
ESC["Pulumi ESC<br/>pulumi-idp/auth"] -.->|"OIDC login (short-lived creds)"| Pulumi["pulumi up"]
Pulumi -.->|"AWS / Azure / GCP"| VM

Component	Port	Description
GPU VM	-	Ubuntu 24.04 with an NVIDIA T4-class GPU. Installs the driver and Ollama from cloud-init, then pulls the model.
Ollama	11434	Serves the model over an HTTP API and keeps it in GPU memory between calls.
Firewall / security group	-	Allows inbound `11434` (and `22` when you ask for it). Open to the world for a demo; lock the CIDR down for anything real.
`pulumi-idp/auth` (ESC)	-	One environment that brokers an OIDC login into AWS, Azure, and GCP. No static keys in code or CI.

One detail here is worth pausing on, and it explains why the Pulumi version comes out shorter than the Terraform one. The Akamai project ends with a null_resource that runs a curl loop in local-exec to block terraform apply until the model finishes downloading. That is not infrastructure but a runtime check wearing a resource costume. Pulumi has a Command provider that would let you reproduce it line for line, and this post deliberately does not. The program provisions the box, the box pulls the model on its own, and the program prints the endpoint. Whether the model has finished downloading yet is a question you answer with a curl, not a question your IaC tool should be holding a deployment open to ask.

Prerequisites

Before getting started, ensure you have:

Pulumi CLI installed and configured
A Pulumi Cloud account (ESC and OIDC live here)
An account on at least one of AWS, Azure, or Google Cloud, with an OIDC trust set up for Pulumi (covered below)
Enough GPU quota in your target region for one T4-class instance; a fresh account often starts at zero, which is the most common reason a first deploy fails, so check it before you run pulumi up
Node.js 18+ for the TypeScript program
curl to talk to the inference endpoint once it comes up

This post serves the qwen2.5:14b model, the same one the Akamai article uses, because the 4-bit quant fits comfortably on a single 16 GB T4. The model is one config value (model), so swap in anything Ollama supports; match the GPU to the model’s memory footprint, because a model that overflows VRAM still runs but spills onto the CPU and crawls.

Credentials without the copy-paste

The Terraform version authenticates the only way a single-cloud demo can: you mint a personal access token, export it as LINODE_TOKEN, and the provider reads it from the environment. It works, but that token is long-lived, it sits in your shell history and your CI secrets, and you mint one per cloud. Across three clouds that is three static keys to rotate and worry about.

Pulumi ESC (Environments, Secrets, and Configuration) replaces it all with an OIDC login. The idea: instead of storing a cloud key, you store a trust relationship. At deploy time, ESC presents a short-lived OIDC token to AWS, Azure, or Google Cloud, and each one hands back temporary credentials scoped to a role you control. Nothing long-lived is ever written down.

I keep that wiring in one environment, pulumi-idp/auth, and every stack imports it. Here is the whole thing:

# pulumi-idp/auth: one ESC environment that brokers an OIDC login into all three
# clouds. Every stack imports it. No static cloud key lives here or anywhere else.
values:
 aws:
 login:
 fn::open::aws-login:
 oidc:
 roleArn: arn:aws:iam::123456789012:role/pulumi-esc
 sessionName: pulumi-esc
 azure:
 login:
 fn::open::azure-login:
 clientId: aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee
 tenantId: aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee
 subscriptionId: /subscriptions/00000000-0000-0000-0000-000000000000
 oidc: true
 gcp:
 login:
 fn::open::gcp-login:
 project: 123456789012 # numeric project number, not the project ID
 oidc:
 workloadPoolId: pulumi-esc
 providerId: pulumi-esc
 serviceAccount: pulumi-esc@my-project.iam.gserviceaccount.com
 environmentVariables:
 # AWS: read by the Pulumi AWS provider, the AWS SDKs, and the aws CLI
 AWS_ACCESS_KEY_ID: ${aws.login.accessKeyId}
 AWS_SECRET_ACCESS_KEY: ${aws.login.secretAccessKey}
 AWS_SESSION_TOKEN: ${aws.login.sessionToken}
 # Azure: read by the azure-native provider
 ARM_USE_OIDC: "true"
 ARM_CLIENT_ID: ${azure.login.clientId}
 ARM_TENANT_ID: ${azure.login.tenantId}
 ARM_SUBSCRIPTION_ID: ${azure.login.subscriptionId}
 ARM_OIDC_TOKEN: ${azure.login.oidc.token}
 # Google Cloud: read by the Pulumi Google Cloud provider
 GOOGLE_CLOUD_PROJECT: ${gcp.login.project}
 GOOGLE_OAUTH_ACCESS_TOKEN: ${gcp.login.accessToken}

You only need the blocks for the clouds you actually deploy to. Opening this environment makes ESC perform a live OIDC login for each cloud listed, so trim it to the one you use, or keep all three if, like me, you bounce between them.

Each cloud needs a one-time trust setup so this login is allowed at all: an IAM OIDC identity provider and role on AWS, a federated credential on an Azure app registration, and a Workload Identity Pool on Google Cloud. You do that once; after that, pulumi-idp/auth is the only thing any stack references for credentials.

A stack opts into it with one block in its stack config. The AWS stack’s Pulumi.aws.yaml:

# Pulumi.aws.yaml
environment:
 - pulumi-idp/auth

That is the entire credential story. No keys in the program, no keys in CI, nothing to rotate. The program below never mentions a secret; it only creates resources, and the ambient credentials from pulumi-idp/auth carry the request.

One cloud-init, three clouds

The part that turns a bare Ubuntu box into an inference server is identical on every cloud, so it lives in one file, cloud-init.yaml, that all three programs read. It follows the Akamai cloud-config closely, with a couple of robustness tweaks for a multi-cloud run, and it runs in a deliberate order because the GPU driver needs a reboot before Ollama can see the card:

Update packages and install the kernel headers and ubuntu-drivers.
Install the NVIDIA driver, then reboot so the kernel loads it.
On the next boot, a one-shot systemd service installs Ollama, binds it to 0.0.0.0:11434, and pulls the model.
The service disables itself, so it never runs again.

#cloud-config
# Zero-touch Ollama GPU box. Cloud-agnostic: no provider metadata, no static creds.
# The Pulumi program substitutes the model name on the `ollama pull` line below
# before this is passed as user-data (AWS/GCP) or custom-data (Azure).

write_files:
 # Make Ollama listen on every interface and never unload the model from VRAM.
 - path: /etc/systemd/system/ollama.service.d/override.conf
 content: |
 [Service]
 Environment="OLLAMA_HOST=0.0.0.0:11434"
 Environment="OLLAMA_KEEP_ALIVE=-1"

 # Runs once on the post-reboot boot, after the GPU driver is loaded.
 - path: /usr/local/bin/ollama-setup.sh
 permissions: '0755'
 content: |
 #!/usr/bin/env bash
 set -euxo pipefail

 # Install Ollama; the installer creates and starts the ollama systemd unit.
 curl -fsSL https://ollama.com/install.sh | sh

 # Pick up the OLLAMA_HOST / OLLAMA_KEEP_ALIVE override written above.
 systemctl daemon-reload
 systemctl enable ollama.service
 systemctl restart ollama.service

 # Wait for the daemon to bind its socket before pulling. This is the box
 # waiting on its OWN local daemon, not an external readiness gate.
 until curl -fsS http://127.0.0.1:11434/api/tags >/dev/null 2>&1; do
 sleep 2
 done

 # Pre-pull the model so the endpoint answers on the very first request.
 ollama pull __MODEL__

 # One-shot: never run again on future boots.
 systemctl disable ollama-setup.service

 - path: /etc/systemd/system/ollama-setup.service
 content: |
 [Unit]
 Description=One-time Ollama install and model pull
 After=network-online.target
 Wants=network-online.target

 [Service]
 Type=oneshot
 ExecStart=/usr/local/bin/ollama-setup.sh
 RemainAfterExit=true
 # The model pull runs for several minutes; without this, systemd's default
 # 90s start timeout kills the service mid-download and the model never lands.
 TimeoutStartSec=0

 [Install]
 WantedBy=multi-user.target

runcmd:
 - apt-get update
 - DEBIAN_FRONTEND=noninteractive apt-get -y upgrade
 # linux-headers-generic also covers the kernel the upgrade above may have pulled
 # in, so DKMS builds the NVIDIA module for the kernel that boots next.
 - DEBIAN_FRONTEND=noninteractive apt-get install -y "linux-headers-$(uname -r)" linux-headers-generic ubuntu-drivers-common
 # --gpgpu selects the headless server driver branch, the right one for a compute
 # GPU like the T4; bare `ubuntu-drivers install` would pull the desktop stack.
 - ubuntu-drivers install --gpgpu
 - systemctl enable ollama-setup.service

power_state:
 mode: reboot
 message: Rebooting to load the NVIDIA driver before Ollama setup
 condition: true

The model name is the literal token __MODEL__. The Pulumi program reads this file at deploy time and substitutes your model config value before handing it to the instance. Two environment settings are doing quiet but important work: OLLAMA_HOST=0.0.0.0:11434 makes Ollama listen on every interface instead of localhost alone, and OLLAMA_KEEP_ALIVE=-1 keeps the model pinned in GPU memory so only the first request pays the load cost.

The programs

Every program has the same five beats: read the config, template the cloud-init.yaml, open the inbound ports, launch the GPU instance with that cloud-init as its user data, and export the endpoint. What differs is only the dialect each cloud speaks for “GPU instance” and “firewall rule.”

The shared contract keeps the three listings legible side by side: the same model and allowSsh config keys, the same cloud-init.yaml, and the same four exports (publicIp, ollamaEndpoint, generateEndpoint, tagsEndpoint). Pick your cloud:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as fs from "fs";

const cfg = new pulumi.Config();
const model = cfg.get("model") ?? "qwen2.5:14b";
const allowSsh = cfg.getBoolean("allowSsh") ?? false;
const region = (cfg.get("region") ?? "us-east-1") as aws.Region;

// Inject the model name into cloud-init at deploy time (a file read, not a shell-out).
const userData = fs.readFileSync("cloud-init.yaml", "utf8").replace(/__MODEL__/g, model);

// Region comes from config; credentials arrive ambiently from pulumi-idp/auth.
const provider = new aws.Provider("aws", { region });

// Most recent Ubuntu 24.04 LTS (amd64, hvm, gp3) published by Canonical.
const ubuntu = aws.ec2.getAmi({
 mostRecent: true,
 owners: ["099720109477"],
 filters: [
 { name: "name", values: ["ubuntu/images/hvm-ssd-gp3/ubuntu-noble-24.04-amd64-server-*"] },
 { name: "virtualization-type", values: ["hvm"] },
 ],
}, { provider });

const ingress: aws.types.input.ec2.SecurityGroupIngress[] = [{
 description: "Ollama HTTP API. NOTE: restrict cidrBlocks to your own range in production.",
 fromPort: 11434,
 toPort: 11434,
 protocol: "tcp",
 cidrBlocks: ["0.0.0.0/0"],
}];

if (allowSsh) {
 ingress.push({
 description: "SSH",
 fromPort: 22,
 toPort: 22,
 protocol: "tcp",
 cidrBlocks: ["0.0.0.0/0"],
 });
}

const sg = new aws.ec2.SecurityGroup("ollama", {
 description: "Ollama inferencing access",
 ingress,
 egress: [{
 description: "Allow all outbound",
 fromPort: 0,
 toPort: 0,
 protocol: "-1",
 cidrBlocks: ["0.0.0.0/0"],
 }],
}, {
 provider,
});

const server = new aws.ec2.Instance("ollama", {
 ami: ubuntu.then(a => a.id),
 instanceType: "g4dn.xlarge", // 1x NVIDIA T4
 vpcSecurityGroupIds: [sg.id],
 associatePublicIpAddress: true,
 userData, // plain text; the AWS provider base64-encodes it for you
 rootBlockDevice: {
 volumeSize: 40,
 volumeType: "gp3",
 },
 tags: { Name: "ollama" },
}, {
 provider,
});

export const publicIp = server.publicIp;
export const ollamaEndpoint = pulumi.interpolate`http://${publicIp}:11434`;
export const generateEndpoint = pulumi.interpolate`http://${publicIp}:11434/api/generate`;
export const tagsEndpoint = pulumi.interpolate`http://${publicIp}:11434/api/tags`;

import * as pulumi from "@pulumi/pulumi";
import * as gcp from "@pulumi/gcp";
import * as fs from "fs";

const cfg = new pulumi.Config();
const model = cfg.get("model") ?? "qwen2.5:14b";
const allowSsh = cfg.getBoolean("allowSsh") ?? false;
const project = cfg.get("project");
const zone = cfg.get("zone") ?? "us-central1-a";

// Inject the model name into cloud-init at deploy time (a file read, not a shell-out).
const userData = fs.readFileSync("cloud-init.yaml", "utf8").replace(/__MODEL__/g, model);

// Latest Ubuntu 24.04 LTS amd64 image.
const ubuntu = gcp.compute.getImage({
 family: "ubuntu-2404-lts-amd64",
 project: "ubuntu-os-cloud",
});

// Network tag binds the firewall rule to this instance.
const networkTag = "ollama";

const firewall = new gcp.compute.Firewall("ollama-fw", {
 network: "default",
 project: project,
 // Inbound to Ollama from anywhere; restrict this CIDR for production.
 allows: allowSsh
 ? [{ protocol: "tcp", ports: ["11434"] }, { protocol: "tcp", ports: ["22"] }]
 : [{ protocol: "tcp", ports: ["11434"] }],
 sourceRanges: ["0.0.0.0/0"],
 targetTags: [networkTag],
});

const instance = new gcp.compute.Instance("ollama", {
 machineType: "n1-standard-4",
 zone: zone,
 project: project,
 tags: [networkTag],
 bootDisk: {
 initializeParams: {
 image: ubuntu.then(i => i.selfLink),
 size: 40,
 },
 },
 guestAccelerators: [{
 type: "nvidia-tesla-t4",
 count: 1,
 }],
 // GPUs cannot live-migrate, so host maintenance must terminate (and restart) the VM.
 scheduling: {
 onHostMaintenance: "TERMINATE",
 automaticRestart: true,
 },
 networkInterfaces: [{
 network: "default",
 accessConfigs: [{}], // an empty config requests an ephemeral public IP
 }],
 metadata: {
 "user-data": userData, // Ubuntu cloud-init reads this key, not startup-script
 },
}, {
 dependsOn: firewall,
});

export const publicIp = instance.networkInterfaces.apply(nics => nics[0].accessConfigs![0].natIp);
export const ollamaEndpoint = pulumi.interpolate`http://${publicIp}:11434`;
export const generateEndpoint = pulumi.interpolate`http://${publicIp}:11434/api/generate`;
export const tagsEndpoint = pulumi.interpolate`http://${publicIp}:11434/api/tags`;

import * as pulumi from "@pulumi/pulumi";
import * as resources from "@pulumi/azure-native/resources";
import * as network from "@pulumi/azure-native/network";
import * as compute from "@pulumi/azure-native/compute";
import * as random from "@pulumi/random";
import * as fs from "fs";

const cfg = new pulumi.Config();
const model = cfg.get("model") ?? "qwen2.5:14b";
const allowSsh = cfg.getBoolean("allowSsh") ?? false;
const location = cfg.get("location") ?? "eastus";

// Inject the model name into cloud-init at deploy time (a file read, not a shell-out).
const userData = fs.readFileSync("cloud-init.yaml", "utf8").replace(/__MODEL__/g, model);

const resourceGroup = new resources.ResourceGroup("ollama-rg", {
 location,
});

const vnet = new network.VirtualNetwork("ollama-vnet", {
 resourceGroupName: resourceGroup.name,
 addressSpace: { addressPrefixes: ["10.0.0.0/16"] },
});

const subnet = new network.Subnet("ollama-subnet", {
 resourceGroupName: resourceGroup.name,
 virtualNetworkName: vnet.name,
 addressPrefix: "10.0.1.0/24",
});

// Standard SKU public IPs use static allocation.
const publicIpAddress = new network.PublicIPAddress("ollama-pip", {
 resourceGroupName: resourceGroup.name,
 sku: { name: network.PublicIPAddressSkuName.Standard },
 publicIPAllocationMethod: network.IPAllocationMethod.Static,
});

// Azure permits all outbound by default, so only inbound rules are needed.
const nsg = new network.NetworkSecurityGroup("ollama-nsg", {
 resourceGroupName: resourceGroup.name,
 securityRules: [
 {
 name: "allow-ollama",
 priority: 1000,
 direction: network.SecurityRuleDirection.Inbound,
 access: network.SecurityRuleAccess.Allow,
 protocol: network.SecurityRuleProtocol.Tcp,
 sourcePortRange: "*",
 destinationPortRange: "11434",
 sourceAddressPrefix: "0.0.0.0/0", // prod: restrict to your client CIDR
 destinationAddressPrefix: "*",
 },
 ...(allowSsh ? [{
 name: "allow-ssh",
 priority: 1001,
 direction: network.SecurityRuleDirection.Inbound,
 access: network.SecurityRuleAccess.Allow,
 protocol: network.SecurityRuleProtocol.Tcp,
 sourcePortRange: "*",
 destinationPortRange: "22",
 sourceAddressPrefix: "0.0.0.0/0",
 destinationAddressPrefix: "*",
 }] : []),
 ],
});

const nic = new network.NetworkInterface("ollama-nic", {
 resourceGroupName: resourceGroup.name,
 networkSecurityGroup: { id: nsg.id },
 ipConfigurations: [{
 name: "ipconfig1",
 subnet: { id: subnet.id },
 publicIPAddress: { id: publicIpAddress.id },
 privateIPAllocationMethod: network.IPAllocationMethod.Dynamic,
 primary: true,
 }],
});

// Azure requires an admin credential even though we never log in. Generate one
// instead of hard-coding it; it stays a Pulumi secret and is never exported.
const adminPassword = new random.RandomPassword("ollama-admin-password", {
 length: 24,
 special: true,
 overrideSpecial: "!#$%*",
 minLower: 1,
 minUpper: 1,
 minNumeric: 1,
 minSpecial: 1, // Azure requires 3 of 4 character classes
});

const vm = new compute.VirtualMachine("ollama-vm", {
 resourceGroupName: resourceGroup.name,
 hardwareProfile: { vmSize: "Standard_NC4as_T4_v3" }, // 1x NVIDIA T4
 networkProfile: {
 networkInterfaces: [{ id: nic.id, primary: true }],
 },
 osProfile: {
 computerName: "ollama",
 adminUsername: "azureuser",
 adminPassword: adminPassword.result,
 customData: Buffer.from(userData).toString("base64"), // Azure wants base64
 linuxConfiguration: {
 disablePasswordAuthentication: false,
 },
 },
 storageProfile: {
 imageReference: {
 publisher: "Canonical",
 offer: "ubuntu-24_04-lts",
 sku: "server", // Gen2; NC4as_T4_v3 is a Gen2 size
 version: "latest",
 },
 osDisk: {
 name: "ollama-osdisk",
 createOption: "FromImage",
 diskSizeGB: 40,
 managedDisk: { storageAccountType: "StandardSSD_LRS" },
 },
 },
});

// A Standard/Static public IP is reserved at creation, so its address is known
// once the resource exists; no separate lookup needed.
export const publicIp = publicIpAddress.ipAddress.apply(ip => ip!);
export const ollamaEndpoint = pulumi.interpolate`http://${publicIp}:11434`;
export const generateEndpoint = pulumi.interpolate`http://${publicIp}:11434/api/generate`;
export const tagsEndpoint = pulumi.interpolate`http://${publicIp}:11434/api/tags`;

A few per-cloud details are worth calling out, since they are the places the “same program” abstraction leaks:

AWS is the shortest program, because the GPU comes with the instance shape: a g4dn.xlarge is a T4 box, so there is no separate accelerator to attach. The one thing to do before you deploy is raise the Running On-Demand G and VT instances vCPU quota in your region; a fresh account starts at zero, and pulumi up fails with VcpuLimitExceeded until you do.
Google Cloud attaches the GPU explicitly with guestAccelerators, and that brings the rule that trips people up: a GPU instance cannot live-migrate, so scheduling.onHostMaintenance must be "TERMINATE" or the apply is rejected. The cloud-init also has to ride on the user-data metadata key, not startup-script, and the empty accessConfigs: [{}] is what hands the box a public IP. T4 quota is also zero on a new project.
Azure is the longest listing, because the network is à la carte: the resource group, virtual network, subnet, public IP, security group, and NIC are each their own resource before you reach the VM. One detail surprises people: a Linux VM requires an admin credential even when you never log in, so the program generates a throwaway password with random.RandomPassword rather than committing one. The NC4as_T4_v3 is a compute GPU, so the standard server (CUDA) driver from the cloud-init is the right choice here; the GRID driver is only needed for GPU-accelerated visualization workloads.

The full programs, all three Pulumi.<cloud>.yaml files, and the shared cloud-init.yaml are in the companion repo:

github.com/dirien/fully-automated-ai-inference-pulumi

Run GPU inference across three clouds

Stand up an Ollama GPU box on AWS, Azure, or Google Cloud from one Pulumi program, with short-lived credentials brokered by Pulumi ESC and no static keys to rotate.

Get started

Deploying

Create a project, install the provider for your cloud, point the stack at pulumi-idp/auth, and deploy. For AWS:

mkdir ai-inference && cd ai-inference
pulumi new typescript
npm install @pulumi/aws

Drop the AWS listing into index.ts, copy the cloud-init.yaml shown earlier into the same directory (or grab it from the companion repo), point the stack at the auth environment, and set your region:

pulumi stack init aws
# add `environment: [pulumi-idp/auth]` to Pulumi.aws.yaml (shown above)
pulumi config set region us-east-1
pulumi config set model qwen2.5:14b
pulumi up

The other two clouds are the same flow with a different provider package and a couple of config keys: npm install @pulumi/gcp with pulumi config set project <your-project-id> and pulumi config set zone us-central1-a, or npm install @pulumi/azure-native @pulumi/random with pulumi config set location eastus.

The slow part is the GPU box: it boots, installs the driver, reboots, installs Ollama, and pulls the model, all without you. Depending on the instance, the region, and the model size, plan on roughly 10 to 15 minutes before the endpoint answers. When pulumi up finishes you get the endpoints straight back as stack outputs:

Outputs:
 generateEndpoint: "http://<public-ip>:11434/api/generate"
 ollamaEndpoint : "http://<public-ip>:11434"
 publicIp : "<public-ip>"
 tagsEndpoint : "http://<public-ip>:11434/api/tags"

Testing the inference server

pulumi up returns as soon as the infrastructure exists, which is before the model has finished downloading. This is exactly where Terraform reached for that null_resource loop, and where Pulumi hands you a URL instead. To check whether the model is ready, ask Ollama what it has loaded:

curl -s $(pulumi stack output tagsEndpoint) | grep -q "qwen2.5:14b" && echo "ready" || echo "still pulling"

Once it reports ready, send it a prompt. This is the same request the Akamai post makes, pointed at your generateEndpoint output:

curl -s $(pulumi stack output generateEndpoint) -d '{
 "model": "qwen2.5:14b",
 "system": "Answer every question with a three-line poem.",
 "prompt": "Why is the sky blue?",
 "stream": false
}'

The first call is slower, because Ollama loads the model into GPU memory before it answers. Every call after that is fast, since OLLAMA_KEEP_ALIVE=-1 keeps it resident:

{
 "model": "qwen2.5:14b",
 "response": "Sunlight scatters, short waves fly,\nBlue light paints the open sky,\nViolet fades as day drifts by.",
 "done": true
}

When you are done, take it all down with one command, so the GPU stops billing:

pulumi destroy

Cost

A GPU instance is the whole bill, and you pay by the hour whether the model is busy or idle. The numbers below are rough on-demand rates for a single T4-class box left running 24/7; in practice you spin it up, use it, and pulumi destroy it, so what you actually pay tracks the hours it stays up.

Cloud	Instance	GPU	~Hourly	~Monthly (24/7)
AWS	`g4dn.xlarge`	1× T4 (16 GB)	~$0.53	~$384
Azure	`Standard_NC4as_T4_v3`	1× T4 (16 GB)	~$0.53	~$384
Google Cloud	`n1-standard-4` + 1× T4	1× T4 (16 GB)	~$0.54	~$394

These are on-demand Linux rates in a cheap US region (us-east-1, eastus, us-central1) as of mid-2026, and the GPU dominates the bill. Spot or preemptible capacity cuts it sharply if your workload tolerates interruption (often 60 to 70 percent off), Google Cloud’s sustained-use discount trims an always-on instance on its own, and the 40 GB disk adds only a few dollars a month.

A GPU box left running overnight is the expensive mistake here. Because the whole stack is declarative, the safe habit is cheap: pulumi destroy when you stop using it, pulumi up when you need it back. The state and config are version-controlled, so standing it up again is one command, not a rebuild.

Security considerations

The architecture in this post matches the Akamai original on purpose, which means it carries the same caveat: Ollama is exposed over plain HTTP on a port open to the entire internet. That is fine for a demo on a box you tear down the same day. It is not fine for anything that outlives the afternoon, and self-hosted AI endpoints get found fast: scanners like Shodan enumerate a freshly exposed one within hours, and an open Ollama instance is free compute for whoever finds it first.

The credential side, though, is genuinely better than a static-token setup, and that is the part worth keeping. There is no long-lived cloud key in the program, in your shell, or in CI; every deploy gets short-lived credentials minted through pulumi-idp/auth and thrown away when it finishes.

Concern	Akamai/Terraform demo	This deployment
Cloud credentials	Static PAT in an env var	OIDC login, short-lived, via `pulumi-idp/auth`
Inbound exposure	Port 11434 open to `0.0.0.0/0`	Same by default; one config flag scopes the CIDR
Transport	Plain HTTP	Plain HTTP (front with TLS for real use)
Readiness wait	`null_resource` + `local-exec` shell loop	A runtime `curl`, no resource

My recommendations for taking this past a demo:

Scope the inbound rule to your own CIDR instead of 0.0.0.0/0; the allowSsh flag in the program is the pattern to copy for the Ollama port
Put a reverse proxy with TLS in front of Ollama, or keep the box off the public internet entirely and reach it over a private network or a Tailscale tailnet, the way I did for the Hermes agent
Keep credentials on OIDC through ESC; never fall back to a static key to “just get it working”
Treat the box as disposable, and pulumi destroy it when you are not using it, which shrinks both the bill and the attack window

What’s next?

The program is a foundation, and the interesting work is what you layer on once a model is a pulumi up away:

Lock it down. Scope the firewall, add TLS, or move the box behind a private network so the endpoint is not on the public internet at all.
Scale the model to the GPU. qwen2.5:14b on a T4 is the entry point. Bigger models want more memory, which means an A10G, an L4, or an A100, and the only change in the program is the instance type and the model value.
Swap the serving engine. Ollama is the simplest thing that works. For higher throughput, the same shape holds with vLLM in place of Ollama in the cloud-init.
Generate the next one. Pulumi Neo can take a target like “a GPU inference box on Azure behind a private endpoint” and produce a first draft of the program, which you then review in a PR like any other change.

Conclusion

Declaring an AI inference server as code buys you the same thing it buys for any other infrastructure: you can reproduce it on any of the three clouds, version-control it, and tear it down with a single pulumi destroy. Porting the Akamai project made the contrast sharp. The credentials moved from a static token to an OIDC login that leaves nothing behind, and the readiness wait disappeared entirely once you stop treating a runtime check as a resource.

That is the line worth carrying to the next thing you deploy. Not every step in a runbook is a resource. A GPU box and a firewall rule are; waiting for a download to finish is not. Draw that line first, and the program gets shorter on its own.

If you run into issues or have questions, drop by the Pulumi Community Slack or GitHub Discussions. New to Pulumi? Get started here.

Deploy a Private Hermes Agent on Render Securely with Pulumi, Modal, and Tailscale

Engin Diri — Wed, 24 Jun 2026 00:00:00 +0000

Personal AI agents had their breakout this year. OpenClaw crossed 100,000 GitHub stars within months of launching, and self-hosting your own assistant went from a hobbyist trick to something a lot of developers actually do. I wrote up how to deploy that lobster to AWS or Hetzner back when it was everywhere.

The one people are switching to now is Hermes, the open-source runtime from Nous Research, and it caught on just as quickly. The reason shows up in every “I ditched OpenClaw for Hermes” thread: it actually learns, building up memory and writing its own skills as it goes instead of running off a static, human-written list.

Here is the part the launch videos skip. Hermes writes and runs its own code, with no human approving the commands. A model that can write code will eventually write a bad one, and the only thing between that command and your credentials is the sandbox it runs in. That is the box you do not want on the public internet. Researchers found 175,000 exposed Ollama servers sitting open in early 2026, and attackers hijack the ones they find for compute. The fix is not a better lock on the front door. It is to have no front door at all.

So this post deploys a private Hermes agent as one Pulumi program across Render, Modal, and Tailscale. The agent and its chat UI run as Render private services with no public URL, Tailscale puts the UI on your tailnet, and Modal runs the agent’s code in throwaway sandboxes. One pulumi up to stand it up, one pulumi destroy to tear it down, around $50 a month. No Makefile, no CLI escape hatches, and an honest look at the few places “nothing on the public internet” still leaks.

What is Hermes?

Hermes is the open-source, MIT-licensed agent runtime from Nous Research. It runs continuously on a server rather than living in a browser tab, and you talk to it through a chat UI or a messaging platform like Telegram, Discord, Slack, or Signal. A few things make it more than a chatbot:

It is model-agnostic. You point it at any provider. This deployment uses OpenRouter, one key that fronts 400-plus models across providers, with automatic failover when one is down.
It remembers across restarts. Memory persists on disk, so it builds up context over time instead of resetting every session.
It writes its own skills. When it works through a task, it can save that approach as a reusable skill and reach for it next time.
It schedules itself. A built-in cron runs automations on a schedule, driven in plain language.
It runs code in a sandbox. When the agent needs to execute code, it hands that off to an isolated, ephemeral container instead of running it next to its own process.

The difference from a cloud-hosted assistant is the usual self-hosting trade: it runs on your infrastructure with your keys, and your conversation history and memory stay on your disk.

“Hermes” here is the agent runtime from Nous Research, not the Hermes family of language models that share the name. The agent is model-agnostic, and the model it talks to is a separate choice you make through OpenRouter.

Prerequisites

Before getting started, ensure you have:

Pulumi CLI installed and configured
A Pulumi Cloud account
A Render account, an API key, and your workspace owner ID
A Modal account and a token pair (token ID and secret)
An OpenRouter API key with credit on the account
A container registry you can push to; this post uses GitHub Container Registry (a GHCR username and token)
Docker running locally (the image build rides on a local buildx engine), or a Docker-enabled CI runner
A Tailscale account with HTTPS enabled and a reusable auth key
Node.js 18+ (for TypeScript) or Python 3.9+ (for Python), depending on the language you pick

This guide routes models through OpenRouter, but Hermes works with other providers too. The model is just a routing string you pass (gpt-5.5 here), so any model OpenRouter supports works by changing that one value.

Understanding the Hermes architecture

Strip away the dashboard clicks and the architecture is four pieces. One rule governs all of them: what the public internet is allowed to touch.

flowchart LR
User([Your devices]) -->|"tailnet (WireGuard)"| TS["Tailscale<br/>private mesh"]
TS -->|"open-webui.&lt;tailnet&gt;.ts.net"| WebUI["Open WebUI<br/>(Render private service,<br/>joined to your tailnet)"]
WebUI -->|"private network<br/>&lt;hermes-slug&gt;:8642"| Hermes["Hermes gateway<br/>(Render private service)"]
Hermes -->|"Modal SDK at runtime"| Modal["Modal sandbox<br/>isolated code execution"]
Hermes -->|model calls| OR["OpenRouter"]

Component	Port	Description
Hermes gateway	8642	Holds secrets and memory, talks to the model, creates Modal sandboxes. The most dangerous box, so it has no public URL.
Open WebUI	8080	The chat front end. Joins your tailnet and reaches Hermes over Render’s private network.
Modal sandbox	-	Isolated, ephemeral container for running the agent’s code.
Tailscale	-	A private WireGuard mesh that makes the UI reachable to your devices only.

The agent that can run code and read secrets is never routable from the internet, and neither is the UI in front of it. There is no inbound ingress on any box (the outbound calls to the model, to Modal, and to the image registry still cross the internet, as they have to). Expressing “which box can reach which” in code is what makes the boundary auditable: the rule lives in the program rather than in a dashboard.

The four pieces split cleanly in two: two you declare as resources, and two you hand a credential and let run.

Layer	How you provision it	Status
Render services, disks, secrets	official `render-oss` Terraform provider, bridged	Declarative, via one `pulumi package add`.
Container images	`@pulumi/docker-build`	Built and pushed during `pulumi up`. No `docker` CLI.
Modal sandboxes	nothing	No Terraform provider exists. A runtime credential.
Tailnet access (Tailscale)	nothing	No provider needed. A runtime credential, like Modal.

Two of these four layers are credentials, not resources. That distinction often gets missed, and it is worth getting right: a provider can only manage what it can provision, and spinning up a Modal sandbox is a runtime act, not a provisioned resource. We come back to it at each layer.

Setting up ESC for secrets management

Deploying Hermes means handling a pile of credentials: a Render API key, registry credentials, the OpenRouter key, the Modal token pair, and the Tailscale auth key. You don’t want these hardcoded or scattered across environment variables. Pulumi ESC (Environments, Secrets, and Configuration) stores them securely and passes them directly to your Pulumi program.

Create a new ESC environment:

pulumi env init <your-org>/hermes-secrets

Add your secrets to the environment:

values:
 renderApiKey:
 fn::secret: "rnd_xxxxx"
 renderOwnerId: "tea-xxxxx"
 ghcrUser: "your-github-username"
 ghcrToken:
 fn::secret: "ghp_xxxxx"
 openrouterApiKey:
 fn::secret: "sk-or-xxxxx"
 modalTokenId:
 fn::secret: "ak-xxxxx"
 modalTokenSecret:
 fn::secret: "as-xxxxx"
 tailscaleAuthKey:
 fn::secret: "tskey-auth-xxxxx"
 tailnetDnsName: "tailxxxxx.ts.net"
 pulumiConfig:
 renderApiKey: ${renderApiKey}
 ownerId: ${renderOwnerId}
 ghcrUser: ${ghcrUser}
 ghcrToken: ${ghcrToken}
 openrouterApiKey: ${openrouterApiKey}
 modalTokenId: ${modalTokenId}
 modalTokenSecret: ${modalTokenSecret}
 tailscaleAuthKey: ${tailscaleAuthKey}
 tailnetDnsName: ${tailnetDnsName}

To find your tailnet DNS name, go to the Tailscale admin console, look under the DNS section, and find your tailnet name (e.g., tailxxxxx.ts.net). This is the domain suffix used for all machines in your Tailscale network.

Then create a Pulumi.dev.yaml file in your project to reference the environment:

environment:
 - <your-org>/hermes-secrets

This keeps your secrets out of your codebase and hands them to the program at deploy time. The Modal token pair and the Tailscale auth key never become resources; they ride straight through as encrypted env vars on the services that need them.

Generating the Render SDK

Render has no first-party Pulumi provider. What it has is an official, maintained Terraform provider, and Pulumi can turn any Terraform provider into a typed local SDK with Any Terraform Provider:

pulumi package add terraform-provider render-oss/render 1.8.0

That one command generates a render SDK you import like any other package, in whatever language your stack is written in. There is no published @pulumi/render package and no vendor to wait on.

This is also where the provider map shows its edges. The bridge wraps a Terraform provider that already exists, so it works for Render. But Modal has no Terraform provider anywhere, so pulumi package add terraform-provider modal-labs/modal has nothing to fetch. That is not a gap to work around. Modal is not a provisioned resource here; Hermes creates sandboxes at runtime through the Modal SDK, and Pulumi’s only job is to hold the token pair and hand it over. Tailscale lands in the same bucket: a Tailscale provider exists, but it manages your tailnet’s settings, not the act of putting a container on the network, which is a runtime job done inside the image.

Securing with Tailscale

The usual way to make a self-hosted UI reachable is to give it a public hostname and put an identity gate in front: an SSO provider, an access policy, a public DNS record, and a certificate to keep renewing. Tailscale collapses all of it into nothing you declare. There is no gate to bolt in front of a public URL, because there is no public URL. Open WebUI joins your tailnet and answers only to your own devices.

This is a stronger position than a public UI behind a login. Because there is no public URL, there is no login to harden and no certificate to renew, and Open WebUI answers only to your own devices.

Tailscale lives in the Open WebUI image rather than the Pulumi program. The image adds Tailscale and an entrypoint that brings up tailscaled in userspace mode (a container has no /dev/net/tun), joins your tailnet, and runs tailscale serve to expose Open WebUI on it. serve publishes over HTTPS to authenticated members of your tailnet only, where its sibling tailscale funnel would put the same port on the public internet, the one thing this design avoids.

FROM ghcr.io/open-webui/open-webui:main-slim
RUN apt-get update && apt-get install -y --no-install-recommends curl ca-certificates \
 && curl -fsSL https://tailscale.com/install.sh | sh
COPY entrypoint.sh /usr/local/bin/ts-entrypoint
ENTRYPOINT ["/usr/local/bin/ts-entrypoint"]

#!/usr/bin/env bash
set -e
PORT="${PORT:-8080}"
mkdir -p /app/backend/data/tailscale
# Userspace networking: a container has no /dev/net/tun. Keep node state on the disk
# so it survives restarts, and accept-dns=false so Render's DNS still resolves the
# private Hermes hostname.
tailscaled --tun=userspace-networking --statedir=/app/backend/data/tailscale &
n=0
until tailscale up --authkey="${TS_AUTHKEY}" --hostname="${TS_HOSTNAME}" --accept-dns=false; do
 n=$((n + 1)); [ "${n}" -ge 30 ] && { echo "tailnet join failed; check TS_AUTHKEY"; exit 1; }
 sleep 2
done
tailscale serve --bg "${PORT}" # serve Open WebUI on the tailnet over HTTPS
cd /app/backend && exec bash start.sh

The TS_AUTHKEY env var on the service, pulled from ESC, is the only thing Pulumi contributes; everything else is the image doing its own networking.

There is one prerequisite, the same one any Tailscale node needs for serve: enable HTTPS for your tailnet once in the admin console so it can get a certificate. Use a reusable auth key, since the service re-authenticates on every redeploy, and the join loop has a ceiling so a dead key fails the deploy in a minute instead of hanging it.

To generate the reusable auth key:

Go to the Tailscale admin console.
Click Generate auth key.
Enable Reusable, since the service re-authenticates on every redeploy.
Copy the key into your ESC environment as tailscaleAuthKey.

Deploying the agent

This is the part a Makefile usually handles. Two private services, two disks, every secret an encrypted env var, and both images built and pushed during the same pulumi up that creates the services. No CLI runs anywhere, though the build still rides on a local Docker engine through buildx: the docker call is gone, the daemon it talks to is not, so pulumi up needs Docker running.

Create a new Pulumi project (pick the language you prefer; the companion repo has all three):

mkdir hermes-agent-pulumi && cd hermes-agent-pulumi
pulumi new typescript

Install the dependencies and generate the Render SDK:

npm install @pulumi/docker-build @pulumi/random
pulumi package add terraform-provider render-oss/render 1.8.0

Both services need the 2 GB Standard plan. The gateway and Open WebUI each exhaust memory on the 512 MB Starter and OOM on boot.

Two facts make this clean. Render env var values are encrypted at rest, so a secret is only an env var, with no separate resource. And @pulumi/docker-build builds and pushes an image through an embedded buildx during pulumi up, so the image pipeline is declarative too. Render then pulls the pushed image by digest.

The complete program

Hermes comes first: a private Render service with no public URL, a disk for its memory, and every secret as an encrypted env var. The shared bearer key is generated in state with random.RandomBytes, the clean replacement for openssl rand -base64 32; the Modal token pair and the OpenRouter key come from config or Pulumi ESC. Render has no port field, so Hermes binds 0.0.0.0:8642 and Open WebUI reaches it at the service’s read-only slug. Open WebUI is almost the mirror image, private too, except its image joins your tailnet (the tailscaled entrypoint from the previous section) and it carries the operational env vars that keep it lean: no local embedding models, and ENABLE_PERSISTENT_CONFIG=false so your config always wins over a stale copy in its database. One render.Provider carries your renderApiKey and ownerId into every Render resource.

Here is the whole program, copy-paste ready as your index.ts (or __main__.py, or Pulumi.yaml):

import * as pulumi from "@pulumi/pulumi";
import * as dockerbuild from "@pulumi/docker-build";
import * as random from "@pulumi/random";
// Local SDK generated by: pulumi package add terraform-provider render-oss/render 1.8.0
// (no @pulumi/render is published; the SDK is bridged from the official Render provider).
import * as render from "@pulumi/render";

const cfg = new pulumi.Config();

// ---------------------------------------------------------------------------
// Configuration. Non-secret values via `pulumi config set`, secrets via
// `pulumi config set --secret` or a Pulumi ESC environment (see README).
// The Render provider reads renderApiKey/renderOwnerId (or RENDER_API_KEY/RENDER_OWNER_ID).
// ---------------------------------------------------------------------------
const region = cfg.get("renderRegion") ?? "oregon";
const modelDefault = cfg.get("modelDefault") ?? "gpt-5.5";

// Container registry the images are pushed to and Render pulls from (GHCR here).
const ghcrUser = cfg.require("ghcrUser");
const ghcrToken = cfg.requireSecret("ghcrToken"); // GHCR PAT with write:packages

// Long-lived secrets supplied by you (dashboard-minted). Hold in config or ESC.
const modalTokenId = cfg.requireSecret("modalTokenId");
const modalTokenSecret = cfg.requireSecret("modalTokenSecret");
const openrouterApiKey = cfg.requireSecret("openrouterApiKey");

// Tailscale: the only "gateway". Open WebUI joins your tailnet and is reachable
// at https://<hostname>.<tailnet>.ts.net by your devices only — never public.
// Both values are already in your ESC. Enable HTTPS once in the Tailscale admin console.
const tailscaleAuthKey = cfg.requireSecret("tailscaleAuthKey");
const tailnetDnsName = cfg.require("tailnetDnsName"); // e.g. tailc6fb4e.ts.net
const webuiHostname = "open-webui";
const webuiTailnetUrl = `https://${webuiHostname}.${tailnetDnsName}`;

// Render provider auth (from ESC: pulumiConfig.renderApiKey / pulumiConfig.renderOwnerId).
const renderProvider = new render.Provider("render", {
 apiKey: cfg.requireSecret("renderApiKey"),
 ownerId: cfg.require("renderOwnerId"),
});

// ---------------------------------------------------------------------------
// Secrets generated in state (the in-IaC replacement for `openssl rand -base64 32`).
// The shared bearer authenticates Open WebUI -> Hermes; it lands on both services.
// ---------------------------------------------------------------------------
const sharedKey = new random.RandomBytes("hermes-shared-key", { length: 32 });
const webuiSecretKey = new random.RandomBytes("webui-secret-key", { length: 32 });

// ---------------------------------------------------------------------------
// Build and push both images during `pulumi up` (no docker CLI; the provider
// embeds buildx). Render then pulls them by their digest-pinned ref.
// ---------------------------------------------------------------------------
const registryAuth = [{ address: "ghcr.io", username: ghcrUser, password: ghcrToken }];

const hermesImage = new dockerbuild.Image("hermes-image", {
 context: { location: "./hermes" },
 dockerfile: { location: "./hermes/Dockerfile" },
 platforms: ["linux/amd64"],
 tags: [pulumi.interpolate`ghcr.io/${ghcrUser}/hermes-agent:latest`],
 push: true,
 registries: registryAuth,
});

const webuiImage = new dockerbuild.Image("webui-image", {
 context: { location: "./openwebui" },
 dockerfile: { location: "./openwebui/Dockerfile" },
 platforms: ["linux/amd64"],
 tags: [pulumi.interpolate`ghcr.io/${ghcrUser}/open-webui:latest`],
 push: true,
 registries: registryAuth,
});

// Credential Render uses to pull the private GHCR images.
const ghcrCredential = new render.RegistryCredential("ghcr", {
 name: "ghcr",
 registry: "GITHUB",
 username: ghcrUser,
 authToken: ghcrToken,
}, { provider: renderProvider });

// ---------------------------------------------------------------------------
// Hermes: a PRIVATE Render service. No public URL; reachable only over Render's
// private network. A disk holds its memory and workspace. Env values are
// encrypted at rest, which is how secrets ride on Render (no separate flag).
// ---------------------------------------------------------------------------
const hermes = new render.PrivateService("hermes-agent", {
 name: "hermes-agent",
 plan: "standard", // 2 GB; the gateway OOMs on starter's 512 MB
 region: region,
 runtimeSource: {
 // Render wants the bare repo in imageUrl; the tag/digest go in their own field.
 image: {
 imageUrl: pulumi.interpolate`ghcr.io/${ghcrUser}/hermes-agent`,
 digest: hermesImage.digest,
 registryCredentialId: ghcrCredential.id,
 },
 },
 disk: { name: "hermes-data", mountPath: "/opt/data", sizeGb: 5 },
 envVars: {
 PORT: { value: "8642" },
 API_SERVER_ENABLED: { value: "true" },
 API_SERVER_HOST: { value: "0.0.0.0" },
 API_SERVER_PORT: { value: "8642" },
 API_SERVER_MODEL_NAME: { value: modelDefault },
 // Auto-approve the agent's shell/terminal hooks (no TTY in a container),
 // so it can run code in the Modal sandbox without an interactive prompt.
 HERMES_ACCEPT_HOOKS: { value: "1" },
 // Secrets (encrypted at rest on Render):
 API_SERVER_KEY: { value: sharedKey.base64 },
 OPENROUTER_API_KEY: { value: openrouterApiKey },
 // Modal is a runtime credential, not a provisioned resource: Hermes creates
 // sandboxes at runtime with this token pair. No Modal provider, no Modal CLI.
 MODAL_TOKEN_ID: { value: modalTokenId },
 MODAL_TOKEN_SECRET: { value: modalTokenSecret },
 },
}, { provider: renderProvider });

// Open WebUI reaches Hermes at its Render-assigned internal hostname (the read-only
// slug), not the bare service name. Same region + same workspace share the network.
const hermesApiBaseUrl = pulumi.interpolate`http://${hermes.slug}:8642/v1`;

// ---------------------------------------------------------------------------
// Open WebUI: also a PRIVATE Render service, but the container joins your tailnet
// (tailscaled in userspace + `tailscale serve`), so you reach it at
// https://open-webui.<tailnet>.ts.net from your devices. Never public; the gate
// is tailnet membership. It still reaches Hermes over Render's private network.
// ---------------------------------------------------------------------------
const webui = new render.PrivateService("open-webui", {
 name: "open-webui",
 plan: "standard", // 2 GB; Open WebUI OOMs on starter's 512 MB
 region: region, // must match Hermes for private networking
 runtimeSource: {
 image: {
 imageUrl: pulumi.interpolate`ghcr.io/${ghcrUser}/open-webui`,
 digest: webuiImage.digest,
 registryCredentialId: ghcrCredential.id,
 },
 },
 disk: { name: "openwebui-data", mountPath: "/app/backend/data", sizeGb: 5 },
 envVars: {
 PORT: { value: "8080" }, // tailscale serve points at this; Open WebUI binds it
 OPENAI_API_BASE_URL: { value: hermesApiBaseUrl },
 ENABLE_OPENAI_API: { value: "true" },
 ENABLE_OLLAMA_API: { value: "false" },
 DEFAULT_MODELS: { value: modelDefault },
 WEBUI_AUTH: { value: "true" },
 WEBUI_URL: { value: webuiTailnetUrl },
 // Reached only over the tailnet, so the first sign-up is the admin.
 ENABLE_LOGIN_FORM: { value: "true" },
 ENABLE_SIGNUP: { value: "true" },
 DEFAULT_USER_ROLE: { value: "admin" },
 // Always apply env config; otherwise Open WebUI persists the first run's config
 // to its DB and ignores later changes.
 ENABLE_PERSISTENT_CONFIG: { value: "false" },
 // Don't load local embedding/RAG models (a big memory saver; the agent is the brain).
 RAG_EMBEDDING_ENGINE: { value: "openai" },
 BYPASS_EMBEDDING_AND_RETRIEVAL: { value: "true" },
 OFFLINE_MODE: { value: "true" },
 HF_HUB_OFFLINE: { value: "1" },
 // Secrets (encrypted at rest):
 OPENAI_API_KEY: { value: sharedKey.base64 }, // shared bearer to call Hermes
 WEBUI_SECRET_KEY: { value: webuiSecretKey.base64 },
 // Tailscale joins this container to your tailnet and serves Open WebUI on it.
 TS_AUTHKEY: { value: tailscaleAuthKey },
 TS_HOSTNAME: { value: webuiHostname },
 },
}, { dependsOn: hermes, provider: renderProvider });

export const hermesInternalUrl = hermesApiBaseUrl;
// Open this from any device on your tailnet:
export const openWebUiTailnetUrl = webuiTailnetUrl;

import pulumi
import pulumi_docker_build as docker_build
import pulumi_random as random

# Local SDK generated by: pulumi package add terraform-provider render-oss/render 1.8.0
# (no pulumi_render is published; the SDK is bridged from the official Render provider).
import pulumi_render as render

cfg = pulumi.Config()

# ---------------------------------------------------------------------------
# Configuration. Non-secret values via `pulumi config set`, secrets via
# `pulumi config set --secret` or a Pulumi ESC environment (see README).
# The Render provider reads renderApiKey/renderOwnerId (or RENDER_API_KEY/RENDER_OWNER_ID).
# ---------------------------------------------------------------------------
region = cfg.get("renderRegion") or "oregon"
model_default = cfg.get("modelDefault") or "gpt-5.5"

# Container registry the images are pushed to and Render pulls from (GHCR here).
ghcr_user = cfg.require("ghcrUser")
ghcr_token = cfg.require_secret("ghcrToken") # GHCR PAT with write:packages

# Long-lived secrets supplied by you (dashboard-minted). Hold in config or ESC.
modal_token_id = cfg.require_secret("modalTokenId")
modal_token_secret = cfg.require_secret("modalTokenSecret")
openrouter_api_key = cfg.require_secret("openrouterApiKey")

# Tailscale: the only "gateway". Open WebUI joins your tailnet and is reachable
# at https://<hostname>.<tailnet>.ts.net by your devices only -- never public.
# Enable HTTPS once in the Tailscale admin console so `tailscale serve` can get a cert.
tailscale_auth_key = cfg.require_secret("tailscaleAuthKey")
tailnet_dns_name = cfg.require("tailnetDnsName") # e.g. tailc6fb4e.ts.net
webui_hostname = "open-webui"
webui_tailnet_url = f"https://{webui_hostname}.{tailnet_dns_name}"

# Render provider auth (from ESC: pulumiConfig.renderApiKey / pulumiConfig.renderOwnerId).
render_provider = render.Provider(
 "render",
 api_key=cfg.require_secret("renderApiKey"),
 owner_id=cfg.require("renderOwnerId"),
)

# ---------------------------------------------------------------------------
# Secrets generated in state (the in-IaC replacement for `openssl rand -base64 32`).
# The shared bearer authenticates Open WebUI -> Hermes; it lands on both services.
# ---------------------------------------------------------------------------
shared_key = random.RandomBytes("hermes-shared-key", length=32)
webui_secret_key = random.RandomBytes("webui-secret-key", length=32)

# ---------------------------------------------------------------------------
# Build and push both images during `pulumi up` (no docker CLI; the provider
# embeds buildx). Render then pulls them by their digest-pinned ref.
# ---------------------------------------------------------------------------
registry_auth = [{"address": "ghcr.io", "username": ghcr_user, "password": ghcr_token}]

hermes_image = docker_build.Image(
 "hermes-image",
 context={"location": "./hermes"},
 dockerfile={"location": "./hermes/Dockerfile"},
 platforms=["linux/amd64"],
 tags=[pulumi.Output.concat("ghcr.io/", ghcr_user, "/hermes-agent:latest")],
 push=True,
 registries=registry_auth,
)

webui_image = docker_build.Image(
 "webui-image",
 context={"location": "./openwebui"},
 dockerfile={"location": "./openwebui/Dockerfile"},
 platforms=["linux/amd64"],
 tags=[pulumi.Output.concat("ghcr.io/", ghcr_user, "/open-webui:latest")],
 push=True,
 registries=registry_auth,
)

# Credential Render uses to pull the private GHCR images.
ghcr_credential = render.RegistryCredential(
 "ghcr",
 name="ghcr",
 registry="GITHUB",
 username=ghcr_user,
 auth_token=ghcr_token,
 opts=pulumi.ResourceOptions(provider=render_provider),
)

# ---------------------------------------------------------------------------
# Hermes: a PRIVATE Render service. No public URL; reachable only over Render's
# private network. A disk holds its memory and workspace. Env values are
# encrypted at rest, which is how secrets ride on Render (no separate flag).
# ---------------------------------------------------------------------------
hermes = render.PrivateService(
 "hermes-agent",
 name="hermes-agent",
 plan="standard", # 2 GB; the gateway OOMs on starter's 512 MB
 region=region,
 runtime_source={
 "image": {
 "image_url": pulumi.Output.concat("ghcr.io/", ghcr_user, "/hermes-agent"),
 "digest": hermes_image.digest,
 "registry_credential_id": ghcr_credential.id,
 },
 },
 disk={"name": "hermes-data", "mount_path": "/opt/data", "size_gb": 5},
 env_vars={
 "PORT": {"value": "8642"},
 "API_SERVER_ENABLED": {"value": "true"},
 "API_SERVER_HOST": {"value": "0.0.0.0"},
 "API_SERVER_PORT": {"value": "8642"},
 "API_SERVER_MODEL_NAME": {"value": model_default},
 # Auto-approve the agent's shell/terminal hooks (no TTY in a container).
 "HERMES_ACCEPT_HOOKS": {"value": "1"},
 # Secrets (encrypted at rest on Render):
 "API_SERVER_KEY": {"value": shared_key.base64},
 "OPENROUTER_API_KEY": {"value": openrouter_api_key},
 # Modal is a runtime credential: Hermes creates sandboxes at runtime with this pair.
 "MODAL_TOKEN_ID": {"value": modal_token_id},
 "MODAL_TOKEN_SECRET": {"value": modal_token_secret},
 },
 opts=pulumi.ResourceOptions(provider=render_provider),
)

# Open WebUI reaches Hermes at its Render-assigned internal hostname (the read-only slug).
hermes_api_base_url = pulumi.Output.concat("http://", hermes.slug, ":8642/v1")

# ---------------------------------------------------------------------------
# Open WebUI: also a PRIVATE Render service, but the container joins your tailnet
# (tailscaled in userspace + `tailscale serve`), so you reach it at
# https://open-webui.<tailnet>.ts.net from your devices. Never public.
# ---------------------------------------------------------------------------
webui = render.PrivateService(
 "open-webui",
 name="open-webui",
 plan="standard", # 2 GB; Open WebUI OOMs on starter's 512 MB
 region=region, # must match Hermes for private networking
 runtime_source={
 "image": {
 "image_url": pulumi.Output.concat("ghcr.io/", ghcr_user, "/open-webui"),
 "digest": webui_image.digest,
 "registry_credential_id": ghcr_credential.id,
 },
 },
 disk={"name": "openwebui-data", "mount_path": "/app/backend/data", "size_gb": 5},
 env_vars={
 "PORT": {"value": "8080"}, # tailscale serve points at this; Open WebUI binds it
 "OPENAI_API_BASE_URL": {"value": hermes_api_base_url},
 "ENABLE_OPENAI_API": {"value": "true"},
 "ENABLE_OLLAMA_API": {"value": "false"},
 "DEFAULT_MODELS": {"value": model_default},
 "WEBUI_AUTH": {"value": "true"},
 "WEBUI_URL": {"value": webui_tailnet_url},
 # Reached only over the tailnet, so the first sign-up is the admin.
 "ENABLE_LOGIN_FORM": {"value": "true"},
 "ENABLE_SIGNUP": {"value": "true"},
 "DEFAULT_USER_ROLE": {"value": "admin"},
 # Always apply env config; otherwise Open WebUI persists the first run's config.
 "ENABLE_PERSISTENT_CONFIG": {"value": "false"},
 # Don't load local embedding/RAG models (a big memory saver; the agent is the brain).
 "RAG_EMBEDDING_ENGINE": {"value": "openai"},
 "BYPASS_EMBEDDING_AND_RETRIEVAL": {"value": "true"},
 "OFFLINE_MODE": {"value": "true"},
 "HF_HUB_OFFLINE": {"value": "1"},
 # Secrets (encrypted at rest):
 "OPENAI_API_KEY": {"value": shared_key.base64},
 "WEBUI_SECRET_KEY": {"value": webui_secret_key.base64},
 # Tailscale joins this container to your tailnet and serves Open WebUI on it.
 "TS_AUTHKEY": {"value": tailscale_auth_key},
 "TS_HOSTNAME": {"value": webui_hostname},
 },
 opts=pulumi.ResourceOptions(depends_on=[hermes], provider=render_provider),
)

pulumi.export("hermesInternalUrl", hermes_api_base_url)
# Open this from any device on your tailnet:
pulumi.export("openWebUiTailnetUrl", webui_tailnet_url)

name: hermes-agent-pulumi
runtime: yaml
description: Deploy a Hermes agent (Render + Modal + Tailscale) as one declarative Pulumi program.

# The Render SDK is bridged from the official Terraform provider. Run once:
# pulumi package add terraform-provider render-oss/render 1.8.0
# Then `pulumi install` regenerates it on a fresh checkout.
packages:
 render:
 source: terraform-provider
 version: 1.1.4
 parameters:
 - render-oss/render
 - 1.8.0

config:
 renderRegion:
 type: string
 default: oregon
 modelDefault:
 type: string
 default: gpt-5.5
 ghcrUser:
 type: string
 ghcrToken:
 type: string
 secret: true
 modalTokenId:
 type: string
 secret: true
 modalTokenSecret:
 type: string
 secret: true
 openrouterApiKey:
 type: string
 secret: true
 tailscaleAuthKey:
 type: string
 secret: true
 tailnetDnsName:
 type: string
 renderApiKey:
 type: string
 secret: true
 renderOwnerId:
 type: string

variables:
 registryAuth:
 - address: ghcr.io
 username: ${ghcrUser}
 password: ${ghcrToken}
 # Open WebUI reaches Hermes at its private slug; you reach Open WebUI on the tailnet.
 hermesApiBaseUrl: http://${hermes-agent.slug}:8642/v1
 webuiTailnetUrl: https://open-webui.${tailnetDnsName}

resources:
 # Render provider auth (from config or ESC).
 render:
 type: pulumi:providers:render
 properties:
 apiKey: ${renderApiKey}
 ownerId: ${renderOwnerId}

 # Secrets generated in state: the replacement for `openssl rand -base64 32`.
 hermes-shared-key:
 type: random:RandomBytes
 properties:
 length: 32
 webui-secret-key:
 type: random:RandomBytes
 properties:
 length: 32

 # Build + push both images during `pulumi up` (no docker CLI).
 hermes-image:
 type: docker-build:Image
 properties:
 context:
 location: ./hermes
 dockerfile:
 location: ./hermes/Dockerfile
 platforms: [linux/amd64]
 tags:
 - ghcr.io/${ghcrUser}/hermes-agent:latest
 push: true
 registries: ${registryAuth}
 webui-image:
 type: docker-build:Image
 properties:
 context:
 location: ./openwebui
 dockerfile:
 location: ./openwebui/Dockerfile
 platforms: [linux/amd64]
 tags:
 - ghcr.io/${ghcrUser}/open-webui:latest
 push: true
 registries: ${registryAuth}

 # Credential Render uses to pull the private GHCR images.
 ghcr:
 type: render:RegistryCredential
 properties:
 name: ghcr
 registry: GITHUB
 username: ${ghcrUser}
 authToken: ${ghcrToken}
 options:
 provider: ${render}

 # Hermes: a PRIVATE Render service. No public URL; encrypted env vars are the secret mechanism.
 hermes-agent:
 type: render:PrivateService
 properties:
 name: hermes-agent
 plan: standard  # 2 GB; the gateway OOMs on starter's 512 MB
 region: ${renderRegion}
 runtimeSource:
 image:
 imageUrl: ghcr.io/${ghcrUser}/hermes-agent
 digest: ${hermes-image.digest}
 registryCredentialId: ${ghcr.id}
 disk:
 name: hermes-data
 mountPath: /opt/data
 sizeGb: 5
 envVars:
 PORT: { value: "8642" }
 API_SERVER_ENABLED: { value: "true" }
 API_SERVER_HOST: { value: "0.0.0.0" }
 API_SERVER_PORT: { value: "8642" }
 API_SERVER_MODEL_NAME: { value: "${modelDefault}" }
 HERMES_ACCEPT_HOOKS: { value: "1" }
 API_SERVER_KEY: { value: "${hermes-shared-key.base64}" }
 OPENROUTER_API_KEY: { value: "${openrouterApiKey}" }
 MODAL_TOKEN_ID: { value: "${modalTokenId}" }
 MODAL_TOKEN_SECRET: { value: "${modalTokenSecret}" }
 options:
 provider: ${render}

 # Open WebUI: also a PRIVATE Render service, joined to your tailnet by its image.
 open-webui:
 type: render:PrivateService
 properties:
 name: open-webui
 plan: standard  # 2 GB; Open WebUI OOMs on starter's 512 MB
 region: ${renderRegion}  # must match Hermes for private networking
 runtimeSource:
 image:
 imageUrl: ghcr.io/${ghcrUser}/open-webui
 digest: ${webui-image.digest}
 registryCredentialId: ${ghcr.id}
 disk:
 name: openwebui-data
 mountPath: /app/backend/data
 sizeGb: 5
 envVars:
 PORT: { value: "8080" }
 OPENAI_API_BASE_URL: { value: "${hermesApiBaseUrl}" }
 ENABLE_OPENAI_API: { value: "true" }
 ENABLE_OLLAMA_API: { value: "false" }
 DEFAULT_MODELS: { value: "${modelDefault}" }
 WEBUI_AUTH: { value: "true" }
 WEBUI_URL: { value: "${webuiTailnetUrl}" }
 ENABLE_LOGIN_FORM: { value: "true" }
 ENABLE_SIGNUP: { value: "true" }
 DEFAULT_USER_ROLE: { value: "admin" }
 ENABLE_PERSISTENT_CONFIG: { value: "false" }
 RAG_EMBEDDING_ENGINE: { value: "openai" }
 BYPASS_EMBEDDING_AND_RETRIEVAL: { value: "true" }
 OFFLINE_MODE: { value: "true" }
 HF_HUB_OFFLINE: { value: "1" }
 OPENAI_API_KEY: { value: "${hermes-shared-key.base64}" }
 WEBUI_SECRET_KEY: { value: "${webui-secret-key.base64}" }
 TS_AUTHKEY: { value: "${tailscaleAuthKey}" }
 TS_HOSTNAME: { value: "open-webui" }
 options:
 provider: ${render}
 dependsOn:
 - ${hermes-agent}

outputs:
 hermesInternalUrl: ${hermesApiBaseUrl}
 # Open this from any device on your tailnet:
 openWebUiTailnetUrl: ${webuiTailnetUrl}

Neither box is published to the internet. The same code, plus the Dockerfiles and the Tailscale entrypoint, is in the companion repo:

github.com/dirien/hermes-agent-pulumi

Cost

Two details shape the bill. The host starts on Render’s free Hobby workspace, but a private service with a persistent disk runs on paid compute, so you need a card on file the moment you add them. There is no free-tier version of this architecture. Both services also need the 2 GB Standard plan, since the gateway and Open WebUI each OOM on the 512 MB Starter.

	Hermes gateway	Open WebUI
Plan	Standard (2 GB)	Standard (2 GB)
Compute	~$25/mo	~$25/mo
Disk	5 GB	5 GB
Public URL	none	none (tailnet only)

That lands the host around $50/month, in line with what a comparable managed setup runs. Modal and OpenRouter bill by usage on top, so the total tracks how hard the agent works rather than a flat number. Tailscale is free for a personal tailnet, so the private gateway adds nothing to that.

The persistent disk pins each service to a single instance, so there is no autoscaling, and a redeploy drops a few seconds of connectivity while the old instance lets go of the disk. For a single-user agent that is a fine trade, and it is why “stand it up in a second region” is a copy-the-config move and not a live failover.

The OpenRouter Activity dashboard: 56 requests and 738K tokens of gpt-5.5 usage, all routed through one key.

Running the deployment

With your ESC environment referenced in Pulumi.dev.yaml, deploy with:

pulumi up

This generates the Render SDK’s resources, builds and pushes both images through buildx, and creates the two private services with their disks and encrypted env vars, all in one run, with no docker CLI and no secret import hiding in a script. When it finishes, both services show up in the Render dashboard with no public URL.

The hermes-agent and open-webui private services in the Render dashboard, both deployed with no public URL.

Access the agent

Give it a few minutes after pulumi up finishes. The images still have to be pulled and the containers booted, and Open WebUI has to join your tailnet before the URL resolves. Refresh until it loads.

Open WebUI is reachable only from a device on your tailnet, at https://open-webui.<your-tailnet>.ts.net, with no domain to buy, no DNS to point, and no certificate to manage. Open it, sign up so the first account becomes the admin, and ask the agent to run a Python command. It should fire up a Modal sandbox and return the result, with nothing exposed to the public internet at any point.

Open WebUI on the tailnet URL: the gpt-5.5 agent runs sum(i*i for i in range(1, 1001)) in a Modal sandbox and returns 333833500.

When the agent runs that code, Modal builds a throwaway sandbox specifically for it. There is no Modal resource to declare, no Modal CLI to call, and no Modal provider to bridge. The Hermes service carries MODAL_TOKEN_ID and MODAL_TOKEN_SECRET as two encrypted env vars, and that is the whole footprint.

The Modal dashboard: the per-task sandboxes the Hermes agent spun up at runtime to execute code.

Verify the deployment (optional)

If the chat UI does not load, give the services a few minutes to finish their first boot, then check two things:

Render dashboard logs for the hermes-agent and open-webui services, to confirm both started and pulled their images.
Tailscale admin console (Machines), to confirm the open-webui node joined your tailnet. If it did not, the TS_AUTHKEY is likely expired or not reusable.

Keep your AI agent private

Deploy your Hermes agent across Render, Modal, and Tailscale as one Pulumi program, with secrets held in Pulumi ESC and nothing exposed to the public internet.

Get started

Security considerations

Self-hosted AI servers get found fast: scanners like Shodan and Censys enumerate a freshly exposed one within hours to days, and the code-capable ones are already under active attack. Running an always-on agent on a machine you also use for everything else invites prompt injection on top of that. The answer here is to keep the agent unroutable from the internet and to isolate the code it runs. That defeats the scanner: nobody finds an open port to attack. It does less against the agent being turned against you from the inside, which is the harder problem.

One detail makes that isolation essential. In a headless container there is no one at the terminal to approve anything, and the same is true of its cron and CI runs, so the deployment runs Hermes with HERMES_ACCEPT_HOOKS=1 to auto-accept the shell hooks it would otherwise pause on. The practical result is that the agent’s code runs with no human in the loop, which is exactly why the Modal sandbox has to carry the weight: when the model writes a bad command, the throwaway container it runs in is what stands between that command and the gateway’s credentials, not a confirmation prompt.

Concern	Typical self-host	This deployment
Gateway reachability	Public port, or behind a login	Private service, no public URL
Chat UI reachability	Public hostname + auth	Tailnet only (`tailscale serve`)
Code execution	Runs next to the gateway	Isolated Modal sandbox, torn down after
Command approval	Human approves dangerous commands	Headless: isolation replaces the prompt
Secrets in transit	Exposed if accessed over HTTP	Encrypted env vars; tailnet-encrypted access

This is private, which is not the same as fully trusted. A few places the boundary still leaks:

The shared bearer never rotates. Hermes is unroutable from the internet, but it is wide open to Open WebUI, and the two share one static bearer key. Whoever owns the front end owns the gateway behind it. Rotate it periodically.
The tailnet policy is click-ops. The ACLs, the node’s tags, and the one-time HTTPS toggle live in the Tailscale admin console, not in pulumi up, and the reusable key has no tag scoping its blast radius. The Tailscale provider can manage that policy declaratively if you want it in code.
Signup defaults to admin. Fine on a single-user tailnet, the first thing to lock down (ENABLE_SIGNUP, DEFAULT_USER_ROLE) once more than one person can reach the URL.
Every messaging client is another entry point. Each platform you wire up adds another way in, which is why the tailnet is the baseline for securing this, not the full extent of it.
A compromised agent can still reach out. The private core stops external discovery, but it does not contain the agent itself. The gateway holds the Modal and OpenRouter tokens as env vars and can call out freely, so a prompt-injected agent could read its own environment and send them somewhere; the Modal sandbox isolates the code the agent runs, not the credentials it holds. Cap and rotate the OpenRouter key, keep the Modal token off the gateway, and allowlist outbound traffic, which on Render means a forward-proxy sidecar (Render gives you dedicated outbound IPs and inbound rules, but no destination egress filter).

My recommendations:

Keep both services private; never give Hermes a public URL
Run code in Modal sandboxes, since HERMES_ACCEPT_HOOKS=1 means no human approves commands
Rotate the shared bearer key and the Tailscale auth key periodically
Cap and rotate the OpenRouter key; a leaked credential is the likelier failure mode than an open port
Use Pulumi ESC for secrets instead of hardcoding
Lock down Open WebUI signup once more than one person can reach it
Manage Tailscale ACLs and tags deliberately, in the admin console or with the Tailscale provider

What’s next?

The deployment is the foundation the agent builds on. A private agent becomes useful when it remembers earlier context, runs code without a human at the terminal, and initiates contact on its own. The disk is what makes this persist: its memory and its schedule live on that disk, so they survive every restart.

It schedules itself. Hermes ships a first-class cron you drive in plain language, either by asking the agent in the chat window or through its CLI. The schedule takes intervals, five-field cron, or one-shots:

hermes cron create "every 1h" \
 "Scrape <listing-url> for the price, compare it with seen.json in the sandbox, message me on Telegram only if it dropped, and otherwise reply exactly [SILENT]." \
 --deliver telegram

The [SILENT] reply suppresses delivery when nothing changed, so the watcher only speaks up when there is something to say. Scheduled runs skip memory by default, so polling does not pollute what the agent has learned.

It connects where you work, with the same secret pattern. Messaging clients attach to the same gateway, and their tokens ride as encrypted env vars on the Hermes service, the exact shape you used for the Modal pair:

// alongside MODAL_TOKEN_ID on the hermes service:
TELEGRAM_BOT_TOKEN: { value: cfg.requireSecret("telegramBotToken") },

plus a platform block you add to config.yaml (the image ships only the model and terminal blocks; Hermes supports the messaging platforms on top, per its docs):

gateway:
 platforms:
 telegram:
 require_mention: true

With those two pieces, a few things are genuinely doable today:

A price or uptime watcher that only speaks up when something changed, using the [SILENT] and persistent-sandbox dedup pattern above.
A daily briefing that ranks and dedups your feeds and delivers the top items each morning, the documented daily-briefing bot. If you want it to adapt to your taste over time, that is the optional Honcho memory provider’s job; it reads your preferences out of the conversation itself. The built-in file memory has no version of that.
A PR-review teammate that reviews open pull requests against your repo’s AGENTS.md conventions and saves recurring patterns as a skill, so the fifth review needs less hand-holding than the first.

A few limits are worth stating plainly, since demos tend to overstate the capabilities. Hermes does not ship Gmail, Calendar, or market integrations out of the box; those are wired up as MCP servers or skills, which is more work than a launch video implies. It curates its own memory and improves its own skills, but it does not retrain the underlying model or autonomously change its high-level strategy, so “self-learning” here means whatever it writes into memory and skills each run.

Conclusion

Deploying a Hermes agent with infrastructure as code means you can reproduce the setup anytime, version control it, and tear it down with a single pulumi destroy. Keeping both services private, with no public URL and no exposed ports, means the code-executing core never sits on the internet for someone to find.

The part worth carrying to the next service you deploy: not every layer has a provider, and that is fine. The task is to tell a real resource apart from a runtime dependency wearing a costume. Render and the images are resources you declare; Modal and Tailscale are credentials you hand over. Draw that map first, and the program follows from it.

To generate that map instead of writing it by hand, Pulumi Neo can take a target architecture like this one and produce a first draft of the program. The agent side of this shift is covered in How Building AI Agents Has Changed in 2026 and the platform side in Agent Sprawl Is Here. Your IaC Platform Is the Answer..

If you run into issues or have questions, drop by the Pulumi Community Slack or GitHub Discussions. New to Pulumi? Get started here.

Stop Prompting. Design the Loop.

Engin Diri — Tue, 09 Jun 2026 00:00:00 +0000

For about two years, the unit of work with a coding agent was the prompt. You wrote a good one, you gave it enough context, you read what came back, and you wrote the next one. The agent was a tool, and you were holding it the entire time, one turn after another.

That part is ending. Addy Osmani, a director of AI at Google Cloud, has a name for what replaces it, and I have not stopped thinking about it since: loop engineering. You stop being the person who prompts the agent. You design the loop that prompts it for you.

In my phrasing: you stop being the thing that runs, and start designing the thing that runs. The leverage moves up a layer. What I want to do here is take an honest look at the pieces, and at the part nobody automates.

The leverage moved up a layer

The people building these tools have already made the jump. Peter Steinberger has been posting it as a monthly reminder.

Peter Steinberger (@steipete) on X.

Boris Cherny, who heads Claude Code at Anthropic, says the same thing about his own job. He does not prompt Claude anymore. He has loops running that prompt Claude and decide what to do next, scanning the issue tracker, the team chat, and the timeline for what to build. “My job is to write loops.”

A loop is a goal that prompts itself. You set the purpose, and the system keeps iterating until it’s met. In practice it finds the work, hands it out, checks the result, writes down what got finished, and decides the next thing, then it pokes the agent instead of you. You build that small system once and let it run.

Look closer, and a loop is really two loops nested. The inner one does the work against a spec. The outer one decides what the work should be: it watches an issue tracker, an error feed, a changelog, then writes the next spec and hands it down. Most people are still running that outer loop by hand, in their head, and calling it a backlog.

The part that surprised me is that this is barely a tooling problem anymore. A year ago a loop meant a pile of bash you wrote and maintained forever. Now the pieces ship inside the products, and the same shapes show up in Claude Code and in Codex. Osmani puts loop engineering one floor above the harness, the context and tooling you wire around a single agent. I wrote about that harness a couple of weeks ago. The loop is the thing that runs on top of it: it runs on a timer, it spawns helpers, and it feeds itself.

The five pieces, and the one that holds them together

Strip loop engineering down and you get roughly five building blocks, plus one place to remember things. Both Claude Code and Codex have all five now. The names differ here and there; the capability is the same.

Automations are the heartbeat. They are what make a loop an actual loop and not one run you did once. A prompt or command on a cadence, a scheduled task, a hook that fires at a point in the agent’s lifecycle, or a job on CI that keeps running after you close the laptop. Discovery and triage run themselves, and the findings that matter come to you.
Worktrees keep parallel from turning into chaos. The second you run more than one agent, the files start colliding. Two agents writing the same file is the same headache as two engineers committing to the same lines with nobody talking first. A git worktree is a separate working directory on its own branch, so one agent’s edits cannot touch another’s checkout.
Skills are intent, written down. An agent starts every session cold and fills any hole in your intent with a confident guess. A skill is that intent written on the outside: the conventions, the build steps, the “we don’t do it like this because of that one incident,” recorded once where the agent reads it every run. Without skills, the loop re-derives your whole project from zero every cycle.
Connectors let the loop touch your real tools. Built on MCP, they let the agent read the issue tracker, query a database, hit a staging API, or drop a message in chat. This is the difference between an agent that says “here is the fix” and a loop that opens the pull request, links the ticket, and pings the channel once CI goes green.
Sub-agents keep the maker away from the checker. The model that wrote the code is far too generous grading its own homework. A second agent with different instructions, and sometimes a different model, catches the things the first one talked itself into. Worktrees and a cold-context reviewer are two pieces I have written about before, back when the question was running agents in parallel without them trampling each other.

Then the sixth thing: memory. A markdown file, a Linear board, a state file, anything that lives outside the single conversation and holds what is done and what is next. It sounds too dumb to matter, and it’s the whole game. The model forgets everything between runs, so the memory has to live on disk, not in the context window. The agent forgets. The repo does not.

What makes a loop hold together

A loop running unattended is also a loop making mistakes unattended. The one thing that keeps it honest is verification, and verification needs an oracle, something outside the model that returns a hard yes or no. Passing tests, a clean build, a green pipeline, a real production signal. Without an oracle, the loop compounds confidently wrong work, faster than you can read it.

The cleanest version of this already ships in the tools. Claude Code’s /goal keeps working across turns until a condition you actually wrote holds, something like “every test in auth/ passes and lint is clean,” and after every turn a separate, faster model reads the transcript and decides whether you are there yet. The agent that wrote the code is not the one that grades it. That is the maker-and-checker split applied to the stop condition itself. Codex’s /goal reaches the same finish line a different way: the agent audits its own work against the evidence before it can call the goal done.

What the loop still won’t do for you

The loop changes the shape of the work. It does not take it off your desk. And a few things get sharper as the loop gets better, not softer.

Verification is still on you. The split reviewer is what makes “it’s done” mean something, but “done” is a claim, not a proof. Your job is still to ship code you confirmed works, which is harder to remember when the diff arrived while you were at lunch.

The bill comes in two currencies. Tokens and attention. A single unattended run can burn through millions of tokens, and that is only worth it when the tokens buy something worth more than they cost. The quieter trap is the second currency: memory is what lets a loop compound over time, and slop compounds right alongside it. A loop pointed at a vague goal does not get tired and stop. It gets faster.

Your understanding rots if you let it. The faster the loop ships code you did not write, the wider the gap between what exists in the repo and what you actually understand. A smooth loop grows that gap faster, not slower, unless you read what it made. The comfortable posture, where you stop having an opinion and take whatever the loop gives back, is the risky one. Two engineers can build the exact same loop and get opposite results, one moving faster on work they understand deeply, the other avoiding the work entirely. The loop cannot tell which one you are.

When the loop reaches production

Most of this thinking grew up around application code, where a bad run costs you a revert. When the loop reaches into infrastructure, the blast radius is a production outage rather than a revert, and the verification bar has to rise to meet it. The upside is that infrastructure hands the loop a better oracle than application code does. A plan diff is deterministic and machine-readable, a policy check returns a hard verdict, and drift and cost are numbers you can put a threshold on. A reviewer, whether human or agent, can read the change cold, with no memory of the prompt that produced it. That cold-context check is exactly what an unattended loop needs, and it’s the reason an infrastructure loop can be built to hold together while you sleep. Pulumi Neo reasons over the state graph directly, so the checker grounds every claim in what the change actually does, not in what the writer says it does.

Give your loop a real oracle

Infrastructure gives an agent loop a strong oracle: a deterministic plan diff to read and policy checks that return a hard verdict before anything ships. Wire that verification in through the Pulumi MCP server, Agent Skills, or Pulumi Neo.

Get started

Where to start

Pick the loop you can actually trust first. In order:

Start where “done” is unambiguous. CI triage, dependency bumps, a flaky-test hunt, a failing job you keep re-running by hand. Loops need an oracle, so begin where the oracle already exists.
Write the memory file before the loop. One markdown file, or a board. What is done, what is next, what was tried and failed. This is the spine, and everything else hangs off it.
Split the checker from the maker. Use /goal with a verifiable condition, or a second agent with its own instructions. Never let the agent that did the work be the one that decides the work is finished.
Cap it, then read everything. A max-iteration count, a token budget, a teardown step. Run it once, end to end, then read every line it shipped. The first run is the measurement, not the payoff.

Then look at what you built. You designed it once, and it ran without you steering each step. That is the real shift. But the leverage only holds if you wire the loop like an engineer, not like someone looking for permission to stop thinking. Read what it ships. Keep an opinion. Judgment is the one part that does not move up a layer.

The loop will do the typing. The thinking is the work.

See how Pulumi Neo closes the loop on your infrastructure

Five Stacks Before Lunch: The Parallel Coding Playbook for Pulumi

Engin Diri — Tue, 02 Jun 2026 00:00:00 +0000

AI coding has two shapes right now. One agent in a loop, sequential work, you babysitting the chat window. Call that 2x. Most teams live here. Five agents in worktrees, parallel work, fresh-context review on every change. Call that 10x. The trick: 2x is mostly prompting, 10x is mostly plumbing.

The parallel coding playbook is a five-pattern setup for running multiple AI coding agents at the same time without them stepping on each other: an issue used as the spec, a plan/build/validate loop, parallel git worktrees, fresh-session review, and a self-healing layer. The whole thing targets application code. The interesting question, and the one I keep ending up at, is what changes when the five agents are touching infrastructure.

2x is prompting, 10x is plumbing

2x is one human, one agent, one repo, one branch. The agent writes, you review, you tell it to try again, it tries again. The bottleneck is your attention. Whatever the agent’s raw throughput, your reading speed sets the ceiling.

10x moves you out of the per-change loop and into the issue loop. You write five issues with sharp acceptance criteria, send each one to its own agent in its own worktree, and let them plan, build, and validate end-to-end. You read five PRs at lunch instead of pair-programming on one all morning.

Concurrent isolation does the work. And isolation is mostly an infrastructure problem.

The five pillars

The five pillars, in one sentence each.

Issue is the spec. The GitHub issue carries the acceptance criteria. The pull request is the artifact that gets validated. Input and output of every implementation are versioned, scoped, and reviewable on their own.
Plan, build, validate. Three stages, three artifacts. A markdown plan you can read in thirty seconds. A build that produces a diff. A validate step that checks the diff against the spec.
Parallel worktrees. Each agent runs in its own git worktree so concurrent changes never trample each other. One repo, five working trees, five branches.
Fresh-session review. A different agent, a different conversation, no shared context, reads the output and judges it. The reviewer never sees the writer’s chat. An agent reviewing its own output in the same context is theater.
Self-healing layer. When the same issue keeps coming back, fix the system that allowed it. Update the rules, the skills, the AGENTS.md. The agent gets better; the bug class disappears.

The application-code version of this playbook leans on ports, node_modules, and databases to get isolation right. The infrastructure version has a different toolbox.

What changes when the agents are touching infrastructure

Walk the pillars again, this time with a Pulumi shop in mind.

Issue is the spec. For application code, the spec describes behavior. For infrastructure, the spec is a Pulumi component contract plus a Pulumi Policies excerpt. “The resulting bucket is private, lives in eu-west-1, has SSE-KMS, and is tagged owner=team-x.” That sentence compiles to a typed component signature and three policy assertions. The agent does not get to interpret “looks right.” The acceptance criteria are deterministic, which is the whole reason this works.

Plan, build, validate. Pulumi already ships the validate step. pulumi preview produces a deterministic, machine-readable diff a second reviewer can judge without the conversation that produced it. The plan is a markdown doc the agent writes before touching code. The build is pulumi up --target against a review stack scoped to the resources the issue covers. The validate step is the preview output plus the Pulumi Policies verdict.

Parallel worktrees. Worktrees alone are not enough. Two worktrees pointing at the same Pulumi stack will fight over state on the first concurrent up. The unit of isolation for infrastructure is the stack, not the worktree. Each worktree gets its own ephemeral review stack and its own ESC environment for credentials. State branches with the work, credentials branch with the work, and the cloud account does not see five agents elbowing each other.

Fresh-session review. The hardest part of the application-code version is keeping the reviewer cold. For infrastructure, the substrate hands you the cold context. The pulumi preview JSON has no memory of the prompt that produced it. A separate agent reading it has the same starting point a human reviewer has: a diff, a stack name, a policy report. Pulumi Neo reasons over the state graph directly, so the reviewer grounds every claim in what the change actually does, not what the writer says it does. Reviewer quality still depends on how well your policies cover the stack, but the cold-context part comes built in.

Self-healing layer. Most Pulumi Policies rule messages today read like assertions. “S3 bucket has no encryption.” A self-healing layer needs them to read like instructions. “S3 bucket has no encryption. Set serverSideEncryptionConfiguration with SSE-KMS to fix.” That single rewrite is the difference between an agent flailing and an agent fixing the violation on the first try. When the same rule keeps tripping, the fix is upstream of the next pull request: in the rules, in the skills, in the policy itself.

The five catches, infra edition

Every parallelism story has a catch list. The application-code version lists port conflicts, node_modules sprawl, database conflicts, token blowouts, and PR pile-up. The infrastructure equivalents map almost one to one.

Port conflicts become stack-name collisions. Two agents naming their stack dev and racing each other into Pulumi Cloud. The fix is the same hash-the-path trick the app-code playbook uses: derive the stack name from pulumi.getProject() plus a hash of the worktree path. Resource names follow the same pattern. Collisions go away.

node_modules sprawl becomes provider plugin sprawl, mostly already solved. Three worktrees each pulling their own copy of pulumi-aws would add up fast, except Pulumi already shares plugins through a single cache at ~/.pulumi/plugins. Identical provider versions are reused across worktrees automatically. Per-worktree language SDKs (node_modules, venv) still need the usual care, but the provider layer is free.

Database conflicts become state conflicts. Two agents racing each other into pulumi up on the same stack is the same hazard as two agents writing to the same migrated database. The app-code playbook reaches for Neon branches or per-worktree SQLite files to isolate state. The infra answer is simpler: each worktree gets its own review stack. State branches with the work, by construction.

Token blowouts become cloud spend per ephemeral stack. The cost vector flips. For app code, the worry is LLM bills. For infrastructure, the worry is what your five agents just spun up in five review stacks. The mitigations are boring and they work. Use TTL stacks to tear review stacks down on a schedule. Avoid retainOnDelete on review-stack resources so the teardown actually frees them. Cap retries per spec. Watch the bill.

PR pile-up is the same problem. Five reviewed diffs are still five things waiting on the merge queue. The infra-flavored mitigations: stack-scoped reviewers (the human who owns the stack approves the change to it), the Pulumi Cloud audit log for grouping by stack and time, and auto-merge for the narrow class of changes where the preview diff is clean and every policy passes. That last one is where most of the throughput hides.

Run any agent on your stacks

Pulumi hands every coding agent a deterministic preview, isolated stacks, and policy as code to work against, whether that agent is Claude Code, Cursor, or Pulumi Neo.

Get started

Where to start, this afternoon

Three steps, in order, on a stack with a small blast radius.

Write an AGENTS.md for the repo. Five paragraphs is enough. The component library, the stack naming convention, the policy rules, the review-stack TTL, and the one thing in this repo that bites every newcomer. Neo reads AGENTS.md natively, as do most coding agents. This file is the spec for how the agent should behave even before you write a spec for what it should build.
Cut a 24-hour review-stack TTL. Spin up a review stack on PR open, tear it down on PR close or after 24 hours, whichever comes first. This is the gate that turns “ephemeral” from a slogan into a line item that does not appear on next month’s bill.
Run three issues in parallel. Pick three open issues that touch unrelated resources. Spin up three worktrees, three review stacks, three ESC environments. Let each agent run end-to-end against its own stack. Then have a fourth agent read each preview JSON cold and produce a one-paragraph review. Read three PRs plus the reviewer’s summary at lunch.

That last step is the measurement. The first time you run it, half of the changes will fail validation. The second time, fewer. By the third time you will know whether your spec quality, your policies, and your stack hygiene are good enough to scale this to five, then ten, then to every issue tagged infra:fix.

If three issues finish cleanly, you have the substrate. If they do not, the gap is almost always in the spec or the policy rules, not the agent. Fix the spec, tighten the rule, run it again.

Five stacks before lunch

10x is five concurrent agents, working from five issues, against five stacks, behind five fresh-session reviews. The substrate is already there. Stacks isolate state. ESC isolates credentials. pulumi preview is the deterministic artifact a fresh reviewer can read cold. Pulumi Policies is the self-healing layer when you write the rule messages as instructions.

The remaining work is small and mostly wiring. Write the AGENTS.md. Cut the TTL. Pick three issues that touch unrelated resources. Read three PRs at lunch. Five stacks before the room empties out is a realistic Monday.

See how Pulumi Neo runs your stacks

Stop Tuning Prompts. Build a Harness.

Engin Diri — Tue, 26 May 2026 00:00:00 +0000

Anthropic shipped a piece earlier this month called How Claude Code Works in Large Codebases. I have not read anything more useful about coding agents this year. The core claim, in their words: “the ecosystem built around the model—the harness—determines how Claude Code performs more than the model alone.” In my phrasing: in a real codebase, the model is the smaller variable. The layer of context and tooling you wire around the agent matters more than which version of Sonnet or Opus is behind it.

The post stays high-level, which is the right move for a launch piece. What I want to do here is land it. Same seven pieces, but with the wiring you would actually put in a repo, in the order I would put it.

How Claude Code navigates without an index

Anthropic’s writeup says Claude Code works from the live codebase and does not require a codebase index to be built, maintained, or uploaded. The agent navigates the way an engineer would, with grep, find, ls, file reads, and reference-following. Anthropic calls this agentic search, and the upside is obvious: no separate index exists for you to keep fresh.

The downside is also obvious. An engineer who has never seen your repo and only has shell tools will flounder if you drop them in the root with no map. That is your agent on day one. Everything that follows is about giving it the map.

The AI layer in seven pieces

Every codebase used to have two artifacts engineers cared about: the code and the tests. A third exists now. Call it the AI layer, or the harness, or whatever you want. This layer is the set of context and tools you give your coding agent to operate in this specific repo. Anthropic breaks it into seven pieces, and each one solves a different scaling problem.

Anthropic gives each piece a role: CLAUDE.md is the foundation, hooks do self-improvement, skills are progressive disclosure, plugins handle distribution, LSP gives navigation, MCP is extension, subagents split exploration from editing. They are not equal in usage either. CLAUDE.md is read at the start of each session and stays in context for the duration. The others fire when relevant.

Lean and layered CLAUDE.md

The single biggest mistake I see is a root CLAUDE.md that has grown into a small book. Two thousand lines of conventions for parts of the repo the current task will never touch. Every session pays the tax. Anthropic’s own guidance is to keep these files focused on what applies broadly so they do not become a drag on performance, and you can feel that drag in practice: the agent gets cautious, slow, and oddly literal.

Keep the root file lean. What is this repo, broadly. The tech stack. The commands the agent will need (make test, make lint, how to run the dev server). General conventions that apply everywhere. That is most of what belongs there.

Local conventions go in subdirectory CLAUDE.md files. When the agent starts in a subdirectory, Claude Code walks upward from the working directory and loads every CLAUDE.md it finds on the way to the repo root, so root context is never lost and intermediate layers stack in the order you would expect. Claude Code can also discover files below the current working directory when it reads files in those subdirectories. That means services/api/CLAUDE.md only joins the session when the work reaches that service. Same for services/billing/, the frontend, the data layer.

If you already know the task is scoped to one service, start the agent in that subdirectory. The working directory becomes the focus, and the agent stays out of unrelated code unless you tell it otherwise. Most of the time, you know.

Two more cheap wins live in the same neighborhood. Scope the make test and make lint commands so the subdirectory version runs only the slice the agent is working in, instead of the whole repo on every change. And version-control your exclusion rules in .claude/settings.json so the agent never reads dist/, generated SDKs, or vendored code. Every file the agent skips is tokens you keep for the work that matters. If your directory layout is unconventional or has historical baggage, Anthropic also suggests adding a short codebase map to the root CLAUDE.md so the agent has somewhere to anchor.

Hooks that make the harness self-improving

Most teams use hooks as guardrails. Block edits in vendor/, refuse to delete migrations, kill the run if a secret turns up in a diff. That is fine and you should do it. But hooks have a second life that almost no one uses, and that second life is the more interesting one.

Both kinds register the same way, in .claude/settings.json, against named events Claude Code fires during a session:

{
 "hooks": {
 "SessionStart": [
 {
 "hooks": [
 {
 "type": "command",
 "command": "uv run --directory \"$CLAUDE_PROJECT_DIR\" python .claude/hooks/session_start_context.py"
 }
 ]
 }
 ],
 "Stop": [
 {
 "hooks": [
 {
 "type": "command",
 "command": "uv run --directory \"$CLAUDE_PROJECT_DIR\" python .claude/hooks/propose_claude_md.py"
 }
 ]
 }
 ]
 }
}

A SessionStart hook fires before the agent has done anything. Whatever the script prints to stdout is injected straight into the session as context, so you can preload the things the agent would otherwise have to spend a turn discovering: the current branch, the uncommitted diff, the last few commits. For a larger team you might fetch the Confluence or Notion page that owns the directory the engineer is working in. Every developer starts each session pre-oriented, with no manual setup.

"""SessionStart hook — prints orientation Claude reads as session context."""
import os, subprocess
from pathlib import Path

root = Path(os.environ.get("CLAUDE_PROJECT_DIR", "."))

def git(*args):
 out = subprocess.run(["git", *args], cwd=root, capture_output=True, text=True)
 return out.stdout.strip()

print("# Orientation\n")
print(f"## Branch\n{git('rev-parse', '--abbrev-ref', 'HEAD')}\n")
print(f"## Uncommitted changes\n{git('status', '--porcelain') or '(clean)'}\n")
print(f"## Recent commits\n{git('log', '-5', '--oneline')}\n")

The Stop hook is the more interesting one. It fires when the agent finishes its turn. At that moment the session context is still fresh, the diff is still small, and you have a free shot at a question nobody asks: did anything I changed invalidate the rules I wrote down? Spawn a separate headless Claude session, hand it the diff and the relevant CLAUDE.md files, ask it to propose updates, and write the result to a markdown review file. You read it when you are ready. The CLAUDE.md files stop going stale on their own.

The trick is to make the hook itself cheap and dispatch the LLM call in the background, so the end of every turn does not block on a reflection:

"""Stop hook — dispatch a headless Claude reflection in the background."""
import os, subprocess, sys
from pathlib import Path

# The reflector spawns its own headless Claude, whose Stop hook lands back
# here. The lock prevents infinite recursion.
if os.environ.get("REFLECT_LOCK"):
 sys.exit(0)

root = Path(os.environ.get("CLAUDE_PROJECT_DIR", "."))
diff = subprocess.run(
 ["git", "diff", "HEAD"], cwd=root, capture_output=True, text=True
).stdout
if not diff.strip():
 sys.exit(0)

env = {**os.environ, "REFLECT_LOCK": "1"}
subprocess.Popen(
 ["uv", "run", "python", ".claude/hooks/reflect_claude_md.py"],
 cwd=root, env=env,
 stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
)

reflect_claude_md.py is the part that calls a headless claude against the diff and writes .claude/claude-md-review.md. You can grow it from twenty lines to two hundred without ever blocking the agent.

The pattern that ties the two together: hooks let the harness improve itself in the background while you do the actual work.

Path-scoped skills

Skills are where the agent learns how to do a thing. CLAUDE.md is conventions (“every route is registered here”). Skills are workflows (“here is how you add a new route in this repo, end to end”). The two overlap, but the framing keeps me honest: rules in CLAUDE.md, recipes in skills.

The piece of the skills system most teams miss is the path scope. A skill can declare which directories it activates in. A create-api-endpoint skill that only loads when the agent is editing under services/api/ is invisible the rest of the time. With dozens of skills in a real repo, scoping is the difference between a useful library and a wall of irrelevant prompts.

The mental model: progressive disclosure for expertise. Most knowledge in a large codebase is local. Load it locally.

Symbol-level search through LSP and MCP

grep is fine until it isn’t. Past six-digit line counts, plain string search gets slow, returns too much, and burns tokens reading files the agent did not need to open. You also lose what every IDE has done for decades: jump-to-definition, find-references, hover-for-types.

You can give the agent the same navigation. Run a language server locally, wrap it in a small MCP server, expose two or three tools: where_is, find_references, goto_definition. The agent now searches by symbol, not by string. A request like “find every place monthly_total_cents is referenced” returns one definition and the actual references, instead of fifty grep hits that mention the substring in unrelated comments.

This is also where bigger orgs invest. Custom MCP servers that expose internal search systems, the code-ownership graph, the design-doc index. The patterns are the same; the targets are domain-specific. The point is that the agent does not have to brute-force its way through your repo when you already have better tools for finding things.

Image: Anthropic, How Claude Code Works in Large Codebases.

Subagents for exploration

The rule I follow: split exploration from editing. A subagent runs in its own context window. You ask which files implement the billing webhook flow, or what the user model looks like across services. It does the digging, and only the summary comes back to your primary session.

The win is context budget, not parallelism. Exploration is wasteful by nature. The agent reads forty files to find the three that matter, and most of those forty get thrown away. If that happens in your primary session, your editing turns start with a context window already half full of noise. If it happens in a subagent, the noise stays there. You get the answer.

Use the built-in Explore subagent liberally. Custom subagents earn their place when you have a workflow specific enough that a generic explorer is the wrong tool. The file shape is small: a single markdown file under .claude/agents/, a short frontmatter block, and a prompt body. name, description, tools, and model are enough to start:

---
name: explorer
description: Read-only repo explorer. Map a service or package without burning the main session's context, then return findings.
tools: Read, Grep, Glob
model: sonnet
---

You are a read-only explorer. The parent agent will hand you one service or
package to map. Read its `CLAUDE.md` if there is one, then trace entry points,
the public surface, and dependencies. Return findings as your final response.
No edits.

Restricting tools to read-only is the load-bearing line. The model only sees the tools you expose, so an explorer subagent without Write or Edit has nothing to call when it gets tempted, even if the prompt body forgot to say so. Treat that as a strong default. If you need a hard guarantee, layer a PreToolUse hook on top.

Don’t let it rot

The harness is not a one-time setup. Models improve, and rules written for last year’s model often constrain this year’s. A note like “always split refactors into single-file changes” might have saved you in 2024 and might block a beneficial cross-file edit in 2026. Anthropic suggests reviewing your CLAUDE.md files every three to six months, or whenever performance feels like it has plateaued after a major model release. The stop-hook reflection gives you a head start. The rest is on you.

Extend the harness to your stack

Package your infrastructure workflows as Agent Skills and expose your Pulumi state through the Pulumi MCP server, so your coding agent works your stack the way your team does.

Get started

Assign an owner

The last piece is not technical. The teams that get value out of Claude Code at scale have someone who owns the harness. A small platform-engineering team, or one DRI, or a hybrid PM/engineer doing it half-time. Their job is the same shape as owning a CI pipeline: write the conventions, build the skills, run the LSP wrapper, version the hooks, evangelize what works, retire what does not.

Plugins are the distribution vehicle. A good harness that lives in one engineer’s dotfiles stays tribal. The same harness packaged as a plugin (or a private marketplace) is how a team of five hundred ends up running the same skills, the same MCP servers, and the same hooks without anyone having to remember to copy a config.

The pattern that fails: ship Claude Code to the org on a Friday, hope adoption goes viral, watch every team grow its own slightly different version of CLAUDE.md for six months. The pattern that works: a quiet build-out period, a small set of approved skills, a working plugin or two, a documented governance story, then broad access.

Treat the harness like infrastructure.

Where to start

The order that has worked for me, in any repo:

Trim the root CLAUDE.md until it fits on one screen. Move the rest into subdirectories.
Add a Stop hook that proposes updates to those CLAUDE.md files in headless mode.
Convert your three most common repeated tasks into path-scoped skills.
Run a language server behind an MCP server. Stop searching by string.
Get comfortable dispatching exploration to subagents.

Most teams will plateau on step one for a week and find the agent is already noticeably sharper. The rest compounds. I have written more on the agent-tooling shift this is part of in How Building AI Agents Has Changed in 2026, and on the workflow side in The Claude Skills I Actually Use for DevOps and Superpowers, GSD, and GSTACK.

The model will keep getting better. The harness is the work.

How Building AI Agents Has Changed in 2026

Engin Diri — Thu, 14 May 2026 00:00:00 +0000

Twelve months ago, building an AI agent meant picking a framework, defining your tools, standing up a RAG pipeline, and writing a stack of glue code to wire it all together. That was the default playbook. The post-mortem on six months of work usually went the same way: half the time went into infrastructure that had nothing to do with the agent’s actual job.

That isn’t where the work is anymore. Most of the middle layer is gone. The SDKs ship with the tools, the skills system replaced the upfront tool registry, and longer context windows pushed vector search out of the default slot it held all of last year.

The shape is the same as a lot of infrastructure shifts before it. The hard thing got cheap, the cheap thing got expected, and the question moved up a level.

The old playbook

A 2024 to 2025 agent project looked like this. You picked a framework, usually LangChain, LlamaIndex, or an early version of Pydantic AI. You wrote tool definitions, usually a wrapper around an API the agent would call. You stood up a RAG pipeline: chunk your documents, embed them, pick a vector database, write retrievers, layer reranking on top. Then you wrote the agent loop yourself, including prompt assembly, tool dispatch, retry logic, and observability.

This was the default for good reasons. Foundation models had short context windows. They didn’t ship with file access. They couldn’t run code. If you wanted an agent to do anything useful with your data, you had to bring the data to the model in pre-digested chunks.

The cost wasn’t only setup time. It was infra bills, retries against embedding APIs, and a context strategy that fought the model as the model got better. By mid-2025 the retrieval layer was often the bottleneck on quality. The agent would ask a question, get five plausible-looking chunks, and answer from those instead of the document you actually wanted it to read. Chunking decisions made on a Tuesday in March were still hurting answer quality six months later.

Most teams I talked to in 2025 were tuning their RAG pipeline. Almost nobody enjoyed it.

The shift: three things changed at once

Three changes landed close enough together that they collapsed the middle layer.

Built-in tools. The Claude Agent SDK ships with Read, Write, Edit, Bash, Grep, Glob, WebSearch, and WebFetch out of the box. OpenAI’s Codex SDK is similar in shape, with shell and file tools available to the agent by default. These are the tools every agent project was rebuilding in 2024, often as a side quest to the work the agent was actually meant to do. A Read that handles binary files. A Bash that streams output and respects working directory. A Grep that doesn’t choke on large files. The 80% of agent tooling everyone was paying their team to reimplement is now table stakes.

The consequence is that you can give an agent the ability to do real work with about ten lines of configuration. The flip side is that the differentiator moved up a layer. The value isn’t in having Read. It’s in what the agent does with it.

Anything outside the built-in toolbox plugs in through MCP servers. The registry has grown nearly 8x since early 2025, and every major model vendor now ships first-party support. The picture in 2026 is more layered than that, though. A lot of what used to call for an MCP server is now better served by the agent invoking a CLI through Bash and wrapping the recipe in a skill. Benchmarks put CLI-based tool calls at a fraction of the context cost of equivalent MCP calls, with fewer round-trips and fewer failure modes. MCP still earns its place for protocol-heavy work like browser control, OAuth flows, and streaming services, but it stopped being the automatic answer to “how do I give my agent a new capability.”

Skills replaced tool stuffing. The old way was to register every tool the agent might need at startup, eating context every turn whether the agent used the tool or not. A hundred tools meant a heavy system prompt before the agent had thought about anything. The skills pattern flips that. A skill is a small markdown package with a name and a one-line description. The agent sees the description (around 100 tokens) and only loads the body when it decides the skill is relevant. A hundred skills no longer means a hundred tools’ worth of context tax. Anthropic frames this as progressive disclosure: because the body only loads on demand, the amount of content you can bundle into a single skill is effectively unbounded.

Progressive disclosure isn’t a new idea. What’s new is that the agent harness now treats it as the default loading strategy instead of something you have to engineer.

RAG got demoted. This is the change with the biggest blast radius and the smallest amount of commentary. A year ago, “we need to add RAG” was the reflex answer when somebody asked how an agent would handle a corpus. Today that question splits three ways. If the corpus fits in the context window, put it in. If the agent can grep the filesystem, let it grep. If the corpus is genuinely too large for either, vector search is still right, but you’ll find that’s a smaller set of cases than it used to be. You can see this in the coding agents that already ship today. Cursor, Claude Code, and Devin lean on grep, find, and direct file reads more than vector search. LlamaIndex’s own writing on agentic retrieval is one of the clearer reads on where this is going.

Vector search didn’t get worse. The context around it improved enough that it stopped being the right first move.

Taken together, what got pulled into the SDK is the middle of an agent project: the tools layer, the retrieval layer, and the loop. What’s left for the team is the system prompt, the skills, and the policies around what the agent is allowed to do.

When you still need a framework

The first reaction to a lot of this is to declare that frameworks are over. They aren’t, but the cases where you reach for one have narrowed.

Pydantic AI is still the right choice when you want strong typing, deterministic output schemas, and an evaluation loop that matches how the rest of your Python codebase already thinks. LangGraph is still the right choice when your problem is genuinely a graph of agent states with branching and human approval steps. OpenAI’s Agents SDK is built around explicit handoffs between agents and earns its place when that pattern fits how you want to decompose the work. CrewAI is the fastest path I’ve seen for prototyping a multi-agent system, as long as you can live with its opinions. Any team running production traffic across multiple model providers is going to want a routing layer that the official SDK from any single vendor isn’t going to give them. Anthropic’s own writing on building effective agents lands in the same place: start with the simplest thing, add complexity only when the problem demands it.

The mental model that works for me: start with the SDK, reach for a framework when you outgrow it. “Outgrow” usually means one of four things:

Multi-provider routing. You’re running production traffic across more than one model vendor and need a routing layer the official SDKs don’t ship.
Multi-agent orchestration. Your problem genuinely decomposes into separate agents with handoffs, branching, or human approval steps.
Deterministic typing. You need strong schemas and validation around inputs and outputs, and the rest of your codebase already thinks that way.
Production observability. You need eval loops, replay, or tracing beyond what the SDK provides out of the box.

If none of those four are biting, the SDK is probably enough, and adding a framework on top is a layer you’ll regret in six months.

Point your agent at your infrastructure

Give your coding agent grounded context from your Pulumi state graph through the Pulumi MCP server and Agent Skills, or reach for Pulumi Neo when you want an infrastructure agent built in.

Get started

Where this lands for infrastructure work

Two things from the new agent shape map cleanly onto infrastructure work. The first is that “built-in tools plus governed actions” is the model an IaC platform was already running. The SDK assumes the agent has tools that do real work. The platform assumes those tools have policies, audit logs, and short-lived credentials around them. Those assumptions stack.

The second is that a state graph is already structured context. You don’t need to chunk it. You don’t need to embed it. An agent reasoning over a Pulumi stack can grep its way through the program graph the same way it greps a codebase, and the answers are grounded in the same source of truth the rest of your platform uses. I wrote the deeper version in Grounded AI: Why Neo Knows Your Infrastructure. The dark-factory and sprawl posts (The Dark Factory Pattern for Infrastructure and Agent Sprawl Is Here. Your IaC Platform Is the Answer.) are the places to go if you want to push on this further.

Start with the SDK

A year ago, an agent project was 80% glue code and 20% the thing the agent actually did. On most projects today that ratio is flipped. If you’ve been sitting on an agent idea, build it the SDK way first and reach for a framework only when you hit something the SDK genuinely can’t do. Most teams will be surprised how often they don’t.

There’s one agent you don’t have to build at all. Pulumi Neo is the same SDK-first shape applied to the IaC slice: tools that reason directly over your state graph, governed by the controls the rest of your platform already runs on. Save your own SDK time for the agents only you can build.

See how Pulumi Neo works

The Dark Factory Pattern for Infrastructure: Running Pulumi Lights-Out

Engin Diri — Tue, 05 May 2026 00:00:00 +0000

The original dark factory was Fanuc’s robotics plant in Oshino, Japan, where the lights are off because nobody is on the floor. Robots build robots. Parts move through the line for weeks at a time without a person walking past them.

The same pattern is now showing up in software. Three engineers at StrongDM shipped roughly 32,000 lines of production code without writing or reviewing any of it. Stripe’s “Minions” agent system merges over a thousand pull requests every week. In January, Dan Shapiro of Glowforge published a five-level autonomy ladder that landed cleanly enough to become the shorthand most people now use, and BCG put out a piece calling it the dark software factory.

Almost every public writeup so far is about application code. The harder question is what this looks like for infrastructure.

What a dark factory actually is

Shapiro’s ladder is the cleanest framing I’ve seen. He borrows it from the SAE’s self-driving levels, and it fits surprisingly well:

Level	What it is	Driving analogy
0	Spicy autocomplete	Stick shift; you do everything.
1	Coding intern (boilerplate)	Cruise control.
2	Junior developer (interactive pair)	One hand on the wheel.
3	AI writes the majority; you review every PR	Eyes still on the road.
4	Spec-driven; agent runs unattended for hours; you review later	Sleeping at the wheel, you can still wake up.
5	Dark factory; no human review of code before production	No steering wheel at all.

Most teams are at level 2 or 3. A few of the more aggressive ones are at 4. Level 5 is the experiment. Most teams won’t get there safely, and probably shouldn’t try to. The interesting design question is what has to be true for level 5 to be safe at all, and that question gets sharper when the thing being shipped is infrastructure.

A dark factory is not a coding harness. A harness is the framework an agent runs inside; the dark factory is the surrounding system that makes a harness’s output mergeable without a human reading the diff. Copilot and Cursor sit at the other end: interactive, the human stays in the loop on every keystroke. The dark factory takes the human out of the per-change loop entirely and puts them at the top, writing the spec and the acceptance criteria.

The wall between generator and validator

Strip the dark factory down to its layers and there are four of them.

flowchart LR
A[Inputs<br/>Humans] --> B[Code Generation<br/>Autonomous]
B --> C[Validation<br/>Autonomous, isolated]
C -->|pass| D[Merge & Deploy<br/>Autonomous + existing CI/CD]
C -->|fail| B
A -.->|holdout scenarios<br/>generator never sees these| C

The single most important rule is that Code Generation and Validation must be completely isolated. The generator never sees the acceptance scenarios. A separate evaluator does, and it judges the generator’s output against scenarios the generator could not have memorized.

The reason is sycophancy. LLMs are too eager to agree with their own prior turns and too willing to declare victory on something they just produced. Without isolation, the same model that wrote the change is the one telling you it’s fine. The practical concern is direct: a test stored in the same codebase as the implementation will get lazily rewritten to match the code, not the other way around. It isn’t malice; it’s the agent doing exactly what it was asked, badly. The wall is what stops that.

StrongDM’s pattern for this is holdout scenarios: plain-English BDD acceptance tests stored where the generator cannot reach them. Each scenario runs three times against an ephemeral deployment, two of three must pass, and the overall pass rate has to clear 90% before the change moves forward. If the generator fails, it gets a one-line failure message (“SQL Injection Detection failed: endpoint returned 500”), not the scenario text. It cannot game the test.

Without that wall, you don’t have a quality gate. You have theater.

Why infrastructure is the harder version

Application code factories can lean on tests, linters, and type checkers. Infrastructure adds blast radius, drift, secrets, irreversible actions, and multi-region state. A code dark factory shipping a broken UI causes a bad user experience. An infrastructure dark factory shipping a broken IAM policy ends in a postmortem.

A few things make this manageable on Pulumi specifically.

The orchestrator does not need to be invented. The Pulumi Automation API is the engine as an SDK in Python, TypeScript, Go, .NET, Java, or YAML, which is the same surface a dark factory orchestrator runs on. Credentials don’t have to be long-lived: ESC and OIDC issue short-lived ones per run, so the agent never sees a static secret.

Policy doesn’t have to be probabilistic: CrossGuard enforces deterministic rules at preview time. Execution doesn’t have to happen on a laptop: Pulumi Cloud Deployments runs pulumi up inside a governed runner with audit logs and approval rules already wired. And the reasoning layer doesn’t have to start from scratch: Pulumi Neo is grounded in your state graph and ships with three modes (Auto, Balanced, Review) that line up cleanly with Shapiro’s levels 5, 4, and 3.

That doesn’t make Pulumi a dark factory by itself. It means the parts that an application-code factory has to build from scratch are pieces a Pulumi shop already has: a credential broker, a policy engine, a governed runner, a state-aware reasoning layer, an audit trail.

And one more piece nobody talks about: pulumi preview produces a clean, deterministic validation artifact, and CrossGuard evaluates that artifact without ever seeing the conversation that produced the program. That’s the same context-free judgment the holdout pattern depends on, applied at the policy layer instead of the acceptance-test layer. For infrastructure, half the wall is already built.

The interesting work is the part that nobody ships in a box.

The interesting work

What no platform ships for you is the wall: the holdout scenarios for infrastructure, the isolated evaluator that runs them, and the agreement on which stacks are even allowed to run lights-out.

The happy-path orchestrator is small. It pulls a spec, runs preview, hands the preview to an isolated evaluator (with its own credentials and its own access to the cloud, no access to the generator’s prompt or output), and branches on the verdict. Auto mode runs up immediately. Balanced mode submits a deployment that requires approval. Review mode opens a PR for a human. Every branch records a stack version traceable in the audit log. Retries, observability, secret rotation, and the rest of the production-grade plumbing add up to real code, but the shape is small.

The wall is the part that takes a week to get right. You write five plain-English scenarios for one stack (“after pulumi up, the bucket is private, has SSE-KMS, lives in eu-west-1, and is tagged owner=team-x”) and a janky evaluator that runs preview and up against an ephemeral copy, queries the cloud, and asks a separate model whether the resulting state satisfies the scenario. Triple-run, 90% pass gate. Then you watch it for a few weeks before you let anything auto-apply.

Ground your agent in real state

Pulumi Neo reasons over your actual state graph and runs in Review, Balanced, or Auto mode, so you can grant it more autonomy as your confidence grows.

Get started

A four-phase rollout

This is the same path the application-code factories walked, with the gates tightened.

Phase 1: better context, this afternoon

Write an AGENTS.md for your most active stack repo. Pulumi Neo reads it natively, as do most coding agents. While you’re there, look at your CrossGuard rules and rewrite the error messages as instructions. Not “S3 bucket has no encryption” but “S3 bucket has no encryption. Set serverSideEncryptionConfiguration with SSE-KMS to fix.” That single change is the difference between an agent flailing and an agent fixing the policy violation on the first try. Wire pulumi preview as a build-before-push gate so PRs don’t show up just to fail CI.

Phase 2: spec-driven with holdouts, this week

Pick one stack with a small blast radius. A review-stack lifecycle is ideal. Write five plain-English holdout scenarios for it and the janky evaluator above. Humans still approve every PR. Don’t auto-merge yet. You’re earning the data, not declaring trust.

Phase 3: take the human out of the merge

Only after the three measurable gates hold over twenty PRs (scenario pass rate above 90%, false positive rate below 5%, human override rate below 10%) flip auto-apply on for that one stack. Add a weekly drift sweep that goes through the same scenario gate as everything else.

Phase 4: lights out

Expand the auto-apply flag to every stack with strong scenario numbers. Wire your issue tracker so tickets tagged infra:fix flow through the pipeline. Mock the cloud APIs that are slow or flaky enough to make scenario evaluation expensive. At this point the orchestrator is configuration, not architecture.

What could go wrong

None of these have clean fixes. The mitigations below reduce risk; they don’t eliminate it. Any team running level 5 should expect to eat one or two of these in the first year.

The validator approves a bad change. This is the obvious one. The standard mitigation is layered: triple-run each scenario with a 2-of-3 threshold, a 90% gate over the run set, a human audit of the first fifty auto-applied changes, and your existing policies still run after the validator says yes.

The agent gets a destroy permission it shouldn’t have. There’s a class of operations that should not sit in the autonomous loop yet: dropping a database, deleting a hosted zone, rotating a root key, anything that crosses a regulated data boundary. Scope what each agent identity can do at the credential layer, require human approval for anything destructive, and start every stack at Review mode. Tag changes, security-group adjustments, and instance resizes can run autonomously today. Release-branch cuts and config promotions can probably run by next quarter. The destructive class earns its way in over months.

You need all three of those layers. Approvals without policy means anything a human approves in a hurry ships. Policy without approvals means a sufficiently clever spec eventually finds the gap. Both without a human kill switch means an incident at 3 a.m. has nobody to escalate to.

Costs blow up. Cap retries at three per spec, alert on token spend per run, and remember that StrongDM reported roughly $1,000 per day per engineer-equivalent. That’s still cheaper than a salary, but only if you put the cap in place before you find out.

Where to start

Most of what a dark factory needs already exists in any reasonably mature platform. Whatever you have for state, policy, credentials, audit, and a deployment runner is the substrate. The interesting work is not building the factory. It’s the wall: the holdout scenarios that make the gap between “the model says it’s fine” and “the system is actually fine” mean something.

For most teams, Phase 1 alone is the win. Full Level 5 may stay out of reach indefinitely, and that’s fine. The path itself forces useful work: clearer specs, named bottlenecks, the deterministic gates humans had been running in their heads.

Write an AGENTS.md and five holdout scenarios for one stack this week. That’s enough to get a real signal on whether the pattern fits your team. The rest of the path is the same problem the application-code factories have already worked through, with the gates set tighter.

Agent Sprawl Is Here. Your IaC Platform Is the Answer.

Engin Diri — Wed, 22 Apr 2026 00:00:00 +0000

Somewhere in your company right now, a developer is building an AI agent. Maybe it’s a release agent that cuts tags when tests pass. Maybe it’s a cost agent that shuts down idle EC2 overnight. It’s running, it’s in production, and there’s a decent chance the platform team doesn’t know it exists.

This isn’t a thought experiment. OutSystems just surveyed 1,900 IT leaders and the numbers are rough: 96% of enterprises run AI agents in production today, 94% say the sprawl is becoming a real security problem, and only 12% have any central way to manage it. Twelve percent. You can read the full report here.

The real question is where those agents run. Inside the platform you’ve already built, or somewhere off to the side where nobody on the platform team can see them.

The new platform tension

Platform teams have always had two jobs that pull in opposite directions. Let developers ship without waiting on a ticket. Keep the infrastructure coherent while they do. Golden paths, review stacks, a catalog of components that don’t fight each other.

Agents break the second half of that deal.

A developer with a sharp prompt can spin up an SRE agent that watches a queue, a release agent that cuts tags when the test suite goes green, or a cost agent that kills idle infra at 2 a.m. That’s useful. It’s also running on your production cloud account, using credentials you never provisioned, writing to systems you never approved, and the only audit trail is whatever the developer remembered to log. The Salesforce 2026 Connectivity Benchmark pegs the average enterprise at twelve agents today, projected to grow 67% over the next two years. Most teams aren’t ready for one, let alone twenty.

This is the same shape as every sprawl problem before it. I wrote about the last one in How Secrets Sprawl Is Slowing You Down, and the pattern keeps repeating. When something useful gets cheap, it proliferates. When it proliferates without structure, it becomes a liability.

The clock is also ticking on the compliance side. The EU AI Act’s high-risk obligations kick in on 2 August 2026. Colorado’s AI Act goes live on 30 June 2026 after last year’s delay. A folder of unreviewed agent scripts isn’t going to hold up against either of those.

Three ways to respond (only one of them works)

There are roughly three paths from here.

Do nothing. Accept the sprawl and hope nothing catches fire. This is the default, and it’s also how you end up explaining to an auditor why some finance agent moved data between three systems last Thursday and nobody remembers which prompt triggered it.

Mandate centralization. Tell developers every agent has to be registered and approved before it runs. This sounds responsible on a slide, and it falls apart inside a sprint. Developers route around friction. If the official path takes a week and the unofficial path takes an afternoon, the unofficial path wins, and you’ve just pushed the sprawl underground where you can’t see it anymore.

Make the platform the obvious path. Build the thing developers actually want to use. A place where an agent inherits the guardrails, credentials, policies, and audit trail by default, because that’s what’s on offer. Adoption becomes a side effect of shipping something good.

Option three is the only one that scales. It’s also the one where most platform teams look at their existing stack and assume they need to build a pile of new scaffolding. I don’t think they do, and the rest of this post is why.

The seven things an AI agent needs from your platform

An agent needs seven concrete things from the platform it runs on. Each one maps to a Pulumi primitive you already own.

1. A trustworthy context lake

Agents are only as good as the context they can reason over. Drop a generic LLM into your cloud account and you’ll get plausible-sounding nonsense, because the model has never seen your environment. What you actually need is a grounded source of truth: what resources exist, how they relate, which stack owns what, which version is running where.

Pulumi state is already that. Your program graph, your stack outputs, your resource metadata, all of it adds up to a structured record of what you’ve actually deployed. Pulumi Neo reasons directly over that graph, which is why it can tell you why a deployment drifted instead of guessing. I wrote the long version of that argument there. Short version: you already have the context lake. Point agents at it.

2. Pre-cleared integrations

An agent that needs to touch five systems shouldn’t need five separate credential dances. That’s where credential sprawl starts. Every agent gets a long-lived key, every key ends up in somebody’s .env, and every rotation turns into an incident.

The Pulumi surface here is the 200+ providers plus Pulumi ESC handling dynamic credentials through OIDC. An agent doesn’t ask for an AWS access key. It asks ESC for a short-lived, scoped token bound to the environment it’s allowed to operate in, and the token expires when the task ends. No static keys, no rotation pain, no awkward postmortem about how something got committed to GitHub. The ESC patterns I walked through in the Claude skills post work just as well for an autonomous agent as they do for a human developer, which is really the whole point.

3. Governed actions

There’s a real difference between “an agent can see your infrastructure” and “an agent can change your infrastructure.” The second one is where you actually need structure. Pulumi Deployments gives you that structure: defined workflows, controlled triggers, running inside your Pulumi Cloud boundary instead of whatever environment the developer happened to spin up. The Automation API lets you build higher-order orchestration on the same primitives your developers already use.

The framing I keep coming back to goes like this. An agent shouldn’t call pulumi up directly. It should submit an action to a governed pipeline that runs pulumi up on its behalf, inside an environment you control, with a log trail and the guardrails already in place. Same effect, very different threat model.

4. Deterministic policy

Real governance lives outside the prompt. “Please don’t delete production” is a wish written into a system prompt, not an enforced control. And when an agent overrides your intent to do what it thought you meant, it’s behaving exactly the way the technology was designed to behave.

Pulumi Policies is the answer the IaC community landed on years ago: policy as code, written in a real programming language, evaluated deterministically at preview and update time. Disallow production RDS deletions. Require encryption at rest. Block S3 buckets with public ACLs. An agent running through Pulumi hits those gates whether it “wants” to or not, because the gates live in the pipeline and not in the prompt. This is the pillar most teams underweight, and it’s the first one most auditors ask about.

5. An audit trail

When something goes wrong at 3 a.m. (and with enough agents running, something will), you need answers fast. What changed, who changed it, and why. Not just “which agent,” but which version of which agent, triggered by what event, authorized by which policy, touching which resources.

Pulumi Cloud’s activity log, the stack update history, and ESC audit logs already capture all of that. Every update is versioned. Every secret access is logged. Every policy evaluation is recorded. When an agent submits a change through your Pulumi pipeline, it inherits that audit surface for free. The alternative is reconstructing an incident from a mix of Slack messages, container logs, and developer memory, which is roughly the state most teams without a platform are in today.

6. A review process

Not every agent action should wait for a human. But agents do need a promotion path, the same way new platform components do. Experimental, then reviewed, then trusted, then autonomous. That’s exactly what pulumi preview, review stacks, and Deployments PR workflows already model for human contributors. An agent that wants to make a change should have to submit it the same way a junior engineer would. As a diff, with a plan, against a preview environment, until it earns the trust to skip steps.

This connects back to the pattern I laid out in Golden Paths: Infrastructure Components and Templates. Golden paths were never only for humans. They’re just paths, and agents can walk them too.

7. Human-in-the-loop approval

The last pillar is the one that keeps the other six honest. Some decisions shouldn’t be automated, full stop. Production rollbacks outside business hours. Destructive changes above a certain blast-radius threshold. Anything that touches a regulated data boundary. For those cases, you want a forced human checkpoint that the agent can’t route around.

Pulumi Deployments approvals already play that role for human changes. Pulumi Neo’s review steps add the AI-aware version: a structured plan, a diff, a named approver, and a record of what they decided and why. I walked through what this looks like in practice in Self-Verifying AI Agents. Short version: an agent that proposes is much safer than an agent that commits.

Govern agent changes through your platform

Route changes from any coding agent, including Claude Code, Cursor, and Pulumi Neo, through the same infrastructure as code controls, policies, and audit trail your team already runs.

Get started

Why IaC is the natural substrate for this

Step back from the seven pillars and look at what they have in common. Context, integrations, governed actions, deterministic policy, audit, review, approval. None of those are new problems that AI agents invented. They’re the problems infrastructure-as-code has been quietly solving for a decade, for human developers.

Every meaningful agent action ends up being a change, whether that’s to infrastructure, configuration, secrets, or state. IaC is the one layer in your stack that already treats change as the unit of work. Plan, preview, apply, record. If you want governance for agents and you don’t want to build it twice, the most efficient move is to route agent changes through the same substrate your humans already use.

I made the same point from a different angle in Token Efficiency vs Cognitive Efficiency: Choosing IaC for AI Agents. An IaC platform that models your world as a graph of typed resources is a much better reasoning surface for an agent than a stack of YAML or a bash script somebody wrote on a Friday. The structure is what makes it work.

What this means for the platform engineer

There’s a narrative floating around that AI is going to make platform engineers less relevant. I haven’t seen it hold up against an actual production environment. Every stat I’ve looked at points the other way. Gartner expects 70% of enterprises to deploy agentic AI as part of IT infrastructure and operations by 2029, up from less than 5% in 2025. LangChain’s State of Agent Engineering report already has 57% of teams running agents in production today. And Gartner projects that 80% of large software engineering orgs will have a platform team by end of 2026, up from 45% in 2022. More agents means more changes, more changes means more blast radius, and more blast radius means more need for the thing platform teams are uniquely equipped to provide.

Your classic responsibilities haven’t gone anywhere either. Golden paths, service catalogs, CI/CD, on-call rotations, all of that is still yours. Agents are an additional layer that needs the same discipline. The upside is that if your platform already runs on a mature IaC surface, you’re extending a muscle you’ve been building for years instead of growing a new one.

The developer-facing side matters too. A developer building an agent needs to know what’s available to them, needs templates that work on the first try, and needs to see what teammates have already built so they don’t start from a blank page. That’s the territory the Claude skills post and IDP Strategy: Self-Service Infrastructure That Balances Autonomy With Control cover. That’s the experience layer that makes developers actually choose your platform instead of routing around it. You need both sides working at once. The governance your security team cares about, and the experience your developers will actually reach for.

Close the window

The agents your developers are shipping this week are going to outlive the experiment that started them. Some of them will become critical. At least one will cause an incident. At least one will eventually show up in an audit. All of them are going to be easier to govern if they were built on your platform from day one than if you try to wrap policy around them later.

If you want the longer view on where this is going, AI Predictions for 2026: A DevOps Engineer’s Guide is the companion piece. If you want the developer-facing version of the grounding argument, Grounded AI is what to read next.

Either way, here’s where I land. The substrate for agent governance is already running in your stack. You’ve been pointing it at human changes for years. Now point it at the agents too.

See how Pulumi Neo governs agent actions

Superpowers, GSD, and GSTACK: Picking the Right Framework for Your Coding Agent

Engin Diri — Mon, 13 Apr 2026 00:00:00 +0000

Three community frameworks have emerged that fix the specific ways AI coding agents break down on real projects. Superpowers enforces test-driven development. GSD prevents context rot. GSTACK adds role-based governance. All three started with Claude Code but now work across Cursor, Codex, Windsurf, Gemini CLI, and more.

Pulumi uses general-purpose programming languages to define infrastructure. TypeScript, Python, Go, C#, Java. Every framework that makes AI agents write better TypeScript also makes your pulumi up better. After spending a few weeks with each one, I have opinions about when to use which.

The problem all three frameworks solve

AI coding agents are impressive for the first 30 minutes. Then things go sideways. The patterns are predictable enough that three separate teams independently built frameworks to fix them.

Context rot. Every LLM has a context window. As that window fills up, earlier instructions fade. You start a session asking for an S3 bucket with AES-256 encryption, proper ACLs, and access logging. Two hours and 200K tokens later, the agent creates a new bucket with none of those requirements. The context window got crowded and your original instructions lost weight.

No test discipline. Agents write code that looks plausible. Plausible code compiles. Plausible code even runs, for a while. But plausible code without tests is a liability. The agent adds a feature and quietly breaks two others because nothing verified the existing behavior was preserved.

Scope drift. You ask for a VPC with three subnets. The agent decides you also need a NAT gateway, a transit gateway, a VPN endpoint, and a custom DNS resolver. Helpful in theory. In practice, you now have infrastructure you never requested and barely understand. You will also pay for it monthly.

These problems are not specific to Claude Code or any particular agent. They happen with Cursor, Codex, Windsurf, and every other LLM-powered coding tool. The context window does not care which brand name is on the wrapper.

Superpowers: the test-driven discipline enforcer

Superpowers was created by Jesse Vincent and has accumulated over 149K GitHub stars. The core idea is simple: no production code gets written without a failing test first.

The framework enforces a 7-phase workflow. Brainstorm the approach. Write a spec. Create a plan. Write failing tests (TDD). Spin up subagents to implement. Review. Finalize. Every phase has gates. You cannot skip ahead. The iron law is that production code only exists to make a failing test pass.

This sounds rigid. It is. That is the point.

Superpowers includes a Visual Companion for design decisions, which helps when you are making architectural choices that need visual reasoning. The main orchestrator manages the entire workflow from a single context window, delegating implementation work to subagents that run in isolation.

The tradeoff is that the mega-orchestrator pattern means the orchestrator itself can hit context limits on very long sessions. One big brain coordinating everything works well until the big brain fills up. For most projects, this is not an issue. For marathon sessions with dozens of files, keep it in mind.

The workflow breaks down into skills that trigger automatically:

Skill	Phase	What it does
`brainstorming`	Design	Refines rough ideas through Socratic questions, saves design doc
`writing-plans`	Planning	Breaks work into 2-5 minute tasks with exact file paths and code
`test-driven-development`	Implementation	RED-GREEN-REFACTOR: failing test first, minimal code, commit
`subagent-driven-development`	Implementation	Dispatches fresh subagent per task with two-stage review
`requesting-code-review`	Review	Reviews against plan, blocks progress on critical issues
`finishing-a-development-branch`	Finalize	Verifies tests pass, presents merge/PR/keep/discard options

The results speak for themselves. The chardet maintainer used Superpowers to rewrite chardet v7.0.0 from scratch, achieving a 41x performance improvement. Not a 41% improvement. 41 times faster. That is what happens when every code change has to pass a test: the agent optimizes aggressively because it has a safety net.

Superpowers works with Claude Code, Cursor, Codex, OpenCode, GitHub Copilot CLI, and Gemini CLI.

GSD: preventing context rot before it ruins your project

GSD (Get Shit Done) was created by Lex Christopherson and has over 51K stars. Where Superpowers focuses on test discipline, GSD attacks the context window problem directly.

The key architectural decision: GSD does not use a single mega-orchestrator. Instead, it assigns a separate orchestrator to each phase of work. Each orchestrator stays under 50% of its context capacity. When a phase completes, the orchestrator writes its state to disk as Markdown files, then a fresh orchestrator picks up where the last one left off.

Think about why this matters. With a single orchestrator, your 200K token context window is a shared resource. Instructions from hour one compete with code from hour three. GSD sidesteps this entirely. Every phase starts with a full context budget because the previous phase’s orchestrator handed off cleanly and shut down.

The state files use XML-formatted instructions because (it turns out) LLMs parse structured XML more reliably than freeform Markdown. GSD also includes quality gates that detect schema drift and scope reduction. If the agent starts cutting corners or wandering from the plan, the gates catch it.

GSD evolved from v1 (pure Markdown configuration) to v2 (TypeScript SDK), which tells you something about the level of engineering behind it. The v2 SDK gives you programmatic control over orchestration, not just static instruction files.

The tradeoff: GSD has more ceremony than the other two frameworks. For a quick script or a single-file change, the phase-based workflow is overkill. GSD earns its keep on projects that span multiple files, multiple sessions, or multiple days.

The core commands map to a phase-based workflow:

Command	What it does
`/gsd-new-project`	Full initialization: questions, research, requirements, roadmap
`/gsd-discuss-phase`	Capture implementation decisions before planning starts
`/gsd-plan-phase`	Research, plan, and verify for a single phase
`/gsd-execute-phase`	Execute all plans in parallel waves, verify when complete
`/gsd-verify-work`	Manual user acceptance testing
`/gsd-ship`	Create PR from verified phase work with auto-generated body
`/gsd-fast`	Inline trivial tasks, skips planning entirely

GSD supports the widest range of agents: 14 and counting. Claude Code, Cursor, Windsurf, Codex, Copilot, Gemini CLI, Cline, Augment, Trae, Qwen Code, and more.

GSTACK: when you need a whole team, not just an engineer

GSTACK was created by Garry Tan (CEO of Y Combinator) and has over 71K stars. It takes a fundamentally different approach from the other two frameworks.

Instead of disciplining a single agent, GSTACK models a 23-person team. CEO, product manager, QA lead, engineer, designer, security reviewer. Each role has its own responsibilities, its own constraints, and its own slice of the problem.

The framework enforces five layers of constraint. Role focus keeps each specialist in their lane. Data flow controls what information passes between roles. Quality control gates ensure standards at handoff points. The “boil the lake” principle means each role finishes what it can do perfectly and skips what it cannot, rather than producing mediocre work across everything. And the simplicity layer pushes back against unnecessary complexity.

The role isolation is what makes GSTACK distinctive. The engineer role does not see the product roadmap. The QA role does not see the implementation details. Each role only receives the context it needs to do its job. This is not just about efficiency. It prevents the kind of scope creep where an agent that knows everything tries to do everything.

“Boil the lake” is my favorite principle across all three frameworks. It is the opposite of how most agents work. Agents default to attempting everything and producing something mediocre. GSTACK says: do fewer things, but do them right.

The tradeoff: 23 specialist roles feels heavy for pure infrastructure work. If you are writing Pulumi programs and deploying cloud resources with component resources, you probably do not need a product manager role or a designer role. GSTACK shines when you are building a product, not just provisioning infrastructure.

Each slash command activates a different specialist:

Command	Role	What it does
`/office-hours`	YC partner	Six forcing questions that reframe your product before you write code
`/plan-ceo-review`	CEO	Four modes: expand scope, selective expand, hold, reduce
`/plan-eng-review`	Engineering manager	Lock architecture, map data flow, list edge cases
`/review`	Staff engineer	Find bugs that pass CI but break in production, auto-fix the obvious ones
`/qa`	QA lead	Real Playwright browser testing, not simulated
`/ship`	Release engineer	One-command deploy with coverage audit
`/cso`	Security officer	OWASP and STRIDE security audits

GSTACK works with Claude Code, Codex CLI, OpenCode, Cursor, Factory Droid, Slate, and Kiro.

Where each framework fits

	Superpowers	GSD	GSTACK
What it locks down	The dev process itself	The execution environment	Who decides what
Orchestration	Single orchestrator	Per-phase orchestrators	23 specialist roles
Context management	One window	State-to-disk, fresh per phase	Role-scoped handoffs
Where it shines	TDD, subagent delegation, disciplined plan execution	Marathon sessions, parallel workstreams, crash recovery	Product strategy, multi-perspective review, real browser QA
Where it struggles	Anything beyond the build phase	Overkill for small tasks, no role separation	The actual writing-code part
Best for	Solo devs who need test discipline	Complex projects that span days or weeks	Founder-engineers shipping a product
GitHub stars	149K	51K	71K
Agent support	6 agents	14+ agents	7 agents

For infrastructure work, GSD’s context management matters most. Long Pulumi sessions that provision dozens of resources across multiple stacks are exactly the scenario where context rot bites hardest. GSD’s phase-based approach keeps each orchestrator fresh.

Superpowers’ TDD workflow maps well to application code where unit tests are straightforward. Infrastructure testing is different. You cannot unit test whether an IAM policy actually grants the right permissions. You can test the shape of the policy with Pulumi’s testing frameworks, but the real validation happens at pulumi preview and pulumi up. Superpowers still helps here (discipline is discipline), but the TDD cycle is less natural for infra than for app code.

GSTACK shines when the project has product dimensions. If you are building a SaaS platform where the infrastructure serves a product vision, GSTACK’s multi-role governance keeps the product thinking connected to the engineering work. For pure infra provisioning, the extra roles add overhead without much benefit.

My honest take: none of these is universally best. Knowing your failure mode is the real decision.

What keeps going wrong	Try this	The reason
Code works today, breaks tomorrow	Superpowers	Forces every change through a failing test first
Quality drops after the first hour	GSD	Fresh context per phase, nothing carries over
You ship features nobody asked for	GSTACK	Product review before engineering starts
All of the above	GSTACK for direction, bolt on Superpowers TDD	No single framework covers everything yet

Point your coding agent at Pulumi

Whichever framework and agent you run, Pulumi defines infrastructure in TypeScript, Python, and Go, so your agent generates cloud resources it already knows how to write.

Get started

Combining frameworks with Pulumi workflows

These frameworks solve the “how” of agent orchestration. Skills (like the ones from Pulumi Agent Skills) solve the “what,” teaching agents the right patterns for specific technologies. Frameworks and skills complement each other. A skill tells the agent to use OIDC instead of hardcoded credentials. A framework makes sure the agent still remembers that instruction 200K tokens later.

GSD’s state-to-disk approach pairs naturally with Pulumi stack outputs. Each phase can read the previous phase’s stack outputs from the state files, so a networking phase can provision a VPC and the compute phase can reference the subnet IDs without any context window gymnastics.

Superpowers’ TDD cycle maps to infrastructure validation. Write a failing test (the expected shape of your infrastructure). Run pulumi preview (red, the resources do not exist yet). Run pulumi up (green, the infrastructure matches the test). This is not a perfect analogy since infrastructure tests are broader than unit tests, but the discipline of “verify before moving on” translates directly.

You do not have to pick one framework and commit forever. Try GSD for a long multi-stack project. Try Superpowers for a focused library. See which failure mode bites you most and let that guide your choice.

Getting started

github.com/obra/superpowers

github.com/gsd-build/get-shit-done

github.com/garrytan/gstack

All three frameworks support multiple agents. For Claude Code, the install commands are straightforward:

# Superpowers
/plugin install superpowers@claude-plugins-official

# GSD (the installer asks which agents and whether to install globally or locally)
npx get-shit-done-cc@latest

# GSTACK
git clone --single-branch --depth 1 https://github.com/garrytan/gstack.git ~/.claude/skills/gstack && cd ~/.claude/skills/gstack && ./setup

Check each repository’s README for Cursor, Codex, Windsurf, and other agents.

If you want a managed experience that handles orchestration for you, Pulumi Neo is grounded in your actual infrastructure, not internet patterns. It understands your stacks, your dependencies, and your deployment history. The 10 things you can do with Neo post shows what that looks like in practice.

Pick one and give it a project. You will know within an hour whether it fixes your particular failure mode.

Try Pulumi for Free

KubeCon EU 2026 Recap: The Year AI Moved Into Production on Kubernetes

Engin Diri — Wed, 01 Apr 2026 00:00:00 -0700

Amsterdam in late March still has that sharp North Sea wind, but inside the RAI Convention Centre, 13,350 people generated enough energy to heat the building twice over. KubeCon + CloudNativeCon EU 2026 was the biggest European edition yet, and the shift from previous years was impossible to miss. AI dominated the conference.

I spent most of the conference at the Pulumi booth, and that turned out to be the best vantage point. Hundreds of visitors stopped by over four days, and I kept asking the same question: what are you actually running in production with AI on Kubernetes? The answers shaped this post more than any keynote did. Almost everyone had a proof of concept. Almost nobody had a production story they were happy with.

Here is the stat that framed the entire conference for me: 66% of organizations use Kubernetes to host generative AI workloads, but only 7% deploy to production daily. That gap between experimentation and actual production use matched what I was hearing at the booth. The CNCF’s own survey now counts 19.9 million cloud native developers worldwide, 7.3 million of them building AI workloads. The tooling and the infrastructure need to catch up.

My takeaway after four days on the ground: lots of working demos, very few production setups people trust. Teams are trying to scale inference, put guardrails around agents, and make GPU infrastructure behave like anything else they run.

Here is what I saw.

From training to inference: the big pivot

About 67% of AI compute now goes to inference, not training. The inference market is projected to hit $255 billion by 2030. It’s also where most of the operational complexity lives.

NVIDIA leaned into this hard. Their open-source stack around NeMo and Dynamo got significant stage time, but the bigger move was donating three projects to the CNCF: the DRA driver for fractional GPU allocation, the KAI Scheduler for GPU-aware scheduling, and Grove. Moving these to community governance signals that GPU infra is becoming part of the standard Kubernetes toolkit.

The CNCF donations that will reshape AI on Kubernetes

Every KubeCon has its crop of new CNCF projects, but this year’s batch felt different. We are starting to see the building blocks of an AI runtime for Kubernetes.

llm-d was the headline donation. Created by IBM Research, Red Hat, and Google Cloud, it splits inference workloads by separating prefill and decode phases across different pods. The collaborator list reads like an industry consortium: NVIDIA, CoreWeave, AMD, Cisco, Hugging Face, Intel, Lambda, Mistral AI, UC Berkeley, and UChicago. When that many organizations agree on a single approach to distributed inference, pay attention.

NVIDIA’s DRA driver enables fractional GPU allocation and multi-node NVLink support. GPU multi-tenancy is one of the hardest unsolved problems in Kubernetes right now. Scheduling, isolation, cost attribution — all of it breaks down when multiple workloads share a GPU. The DRA driver does not solve everything, but it gives the community a real starting point.

KAI Scheduler entered the CNCF Sandbox for GPU-aware scheduling. If llm-d handles the inference runtime and the DRA driver handles allocation, KAI Scheduler handles placement. Together, these three projects form the skeleton of a GPU-native Kubernetes stack.

Velero, donated by Broadcom, moved into CNCF Sandbox for backup and restore. AI workloads are stateful now (model weights, checkpoints, fine-tuning data), and backup is no longer optional. Good timing.

Microsoft AI Runway is an open-source Kubernetes API for inference that plugs in Hugging Face model discovery, GPU memory fit calculations, and cost estimates. Think of it as a model-aware control plane. HolmesGPT and Dalec, also from Microsoft, entered CNCF Sandbox for AI-powered troubleshooting and dependency analysis.

The Kubernetes AI Conformance Program is growing fast, with certifications nearly doubled and three new requirements proposed for Kubernetes 1.36. Conformance programs are boring until they are not. This one will determine which distributions can credibly claim AI readiness.

Agentic AI gets an identity layer

If inference was this year’s production story, agentic AI was the architecture story. Agents are proliferating, and nobody has quite figured out how to manage and secure them inside Kubernetes yet.

kagent, donated to CNCF Sandbox by Solo.io, defines agents as Kubernetes CRDs. It ships with pre-built MCP (Model Context Protocol) servers for Kubernetes, Istio, Helm, Argo, Prometheus, Grafana, and Cilium. An agent becomes a first-class Kubernetes resource, schedulable and observable and subject to RBAC, instead of a rogue process running in someone’s notebook.

kagenti from IBM goes after the identity problem directly. Using SPIFFE/SPIRE, it gives agents cryptographic identities. When an agent calls an API, you can verify exactly which agent made the call, what trust domain it belongs to, and whether it is authorized. This kind of security work needs to happen before agents proliferate across production clusters. Retrofitting identity later is ugly.

Dapr Agents took a different angle with the actor model and durable execution. Each agent gets reliable state management and exactly-once messaging semantics. If your workflows cannot tolerate lost messages or duplicate actions, this matters.

agentregistry showed up as a centralized discovery service for MCP servers and agents. As agents and tool servers multiply, you need a registry to find and manage them, the same way container registries became necessary for images.

David Soria Parra from Anthropic gave a talk on MCP evolving beyond simple tool-calling into richer interaction patterns (sched). Google announced the Kubernetes Agent Sandbox for running agentic AI workloads in secure, isolated environments — we’ve since published a full walkthrough of what it is and how to deploy it.

AI gateways and inference routing

Gateway infrastructure had its own mini-conference within KubeCon. The Gateway API Inference Extension from the Kubernetes SIG introduces model-aware routing and load balancing at the gateway level. Instead of routing by URL path, your gateway routes by model name, version, and capacity. That changes how inference traffic flows through a cluster in a fundamental way.

Envoy AI Gateway builds on Envoy’s existing proxy capabilities with token-aware rate limiting and provider failover. If your primary inference provider is saturated, traffic shifts to a secondary automatically. Rate limiting by token count rather than request count makes much more sense for LLM workloads, where a single request can consume vastly different amounts of compute.

I want to call out Agentgateway specifically. Written in Rust, it proxies LLM traffic, MCP connections, and agent-to-agent communication, with Cedar and CEL policy engines for fine-grained access control. Rust’s performance characteristics matter here because inference gateway latency adds directly to user-perceived response time.

Kuadrant, now in CNCF Sandbox, layers policy on top of gateway infrastructure and includes MCP server aggregation. Gateways are evolving from dumb traffic proxies into intelligent control planes for AI workloads, and these four projects are driving that shift.

Platform engineering absorbs LLMOps

The observability and platform engineering vendors showed up in force. The message was consistent: LLMOps is just platform engineering with new requirements.

Chronosphere demonstrated parallel AI investigation, with multiple agents analyzing different aspects of an incident simultaneously and combining their findings. SUSE Liz takes a domain-specialized approach, deploying different AI agents for different operational domains rather than one general-purpose assistant. groundcover combines eBPF with OpenTelemetry to give coding agents rich runtime context about the systems they are modifying. That last one is subtle but important: if an AI agent is writing code that touches a service, it should understand that service’s actual runtime behavior, not just its source code.

Dynatrace and DevCycle partnered to make feature flags observable primitives via OpenFeature. Rolling out AI features behind feature flags is table stakes, but having those flags show up in your observability pipeline as first-class signals closes a real gap.

Shadow AI governance emerged as its own theme. CAST AI’s Kimchi can route requests across 50+ models while providing centralized visibility into what models are being used, by whom, and at what cost. Every large organization I talked to had some version of the same problem: teams spinning up model endpoints without central oversight, burning through GPU budgets, creating compliance blind spots they did not even know about.

GPU multi-tenancy remains genuinely unsolved. Scheduling, workload isolation, cost attribution across shared GPUs — all of it breaks down at scale. Multiple talks addressed pieces of this, but nobody had a complete answer.

Sovereignty shapes infrastructure architecture

Regulation came up in almost every conversation. The EU Cyber Resilience Act is driving compliance requirements deep into software supply chains, and every European organization I spoke with is feeling the pressure. Teams are already changing how they build and deploy software.

Sovereign Kubernetes is a platform architecture requirement now, not something you can defer to next quarter. Organizations need Kubernetes distributions and cloud regions that guarantee data residency, and they need the tooling to enforce those guarantees programmatically. Self-hosted models are proliferating partly because of capability and cost, but data sovereignty is the accelerant. If your data cannot leave a jurisdiction, neither can your model.

Runtime isolation is expanding beyond containers. Several talks covered KVM-based isolation for AI workloads, which is heavier than containers but necessary when the threat model includes side-channel attacks on shared GPU memory. The sandboxing conversation has gotten more sophisticated since last year.

These constraints are not uniquely European. Any organization operating across jurisdictions faces similar pressures, and the regulatory direction globally is toward more data sovereignty requirements, not fewer.

What this means for your team

Four days in Amsterdam distilled into five things I would act on now:

Treat inference workloads like production services. If you are still deploying models with scripts and hope, stop. Inference infrastructure needs the same IaC discipline as any other production system: version-controlled, tested, policy-enforced.
Evaluate the Gateway API Inference Extension and llm-d. These are not speculative projects. They have broad industry backing and solve real problems around inference routing and distributed serving. Get them into your test environments.
Plan agent identity before agents proliferate. SPIFFE/SPIRE for agent identity is not optional if you are running agents in production. Retrofitting identity onto an existing agent fleet is painful. Start with kagenti now.
Platform teams should own AI infrastructure. Shadow AI is already happening in your organization. The platform engineering team needs to provide self-service AI infrastructure with guardrails before ungoverned model endpoints become a security and cost problem.
Sovereignty and GPU multi-tenancy are universal. Even if you are not subject to the EU Cyber Resilience Act today, data residency requirements are spreading globally. GPU multi-tenancy will affect every organization running inference at scale.

Kubernetes spent the past decade proving it could orchestrate containers. The next decade will test whether it can orchestrate intelligence. Based on what I saw in Amsterdam, the community is building the right pieces, but the gap between what exists and what production demands is still wide. That gap is where the interesting work happens.

Token Efficiency vs Cognitive Efficiency: Choosing IaC for AI Agents

Engin Diri — Tue, 03 Mar 2026 00:00:00 +0000

When an AI agent writes infrastructure code, two things matter: how compact the output is (token efficiency) and how well the model actually reasons about what it’s writing (cognitive efficiency). HCL produces fewer tokens for the same resource. But does that make it the better choice when agents need to refactor, debug, and iterate? We ran a benchmark across Claude Opus 4.6 and GPT-5.2-Codex to find out.

These two goals pull in opposite directions

You might assume that the language producing fewer tokens is also the one models reason about best. Research into LLM-driven infrastructure generation suggests otherwise.

Where HCL wins on tokens

HCL is declarative and minimal. It requires no imports, no runtime constructs, and no language scaffolding. For simple infrastructure generation, HCL leads to fewer tokens and lower generation cost.

For a straightforward resource definition, HCL gets straight to the point:

resource "aws_s3_bucket" "example" {
 bucket = "my-bucket"
}

Compare that with the Pulumi TypeScript equivalent:

import * as aws from "@pulumi/aws";

const bucket = new aws.s3.Bucket("example", {
 bucket: "my-bucket",
});

The HCL version requires fewer tokens. No import statement, no variable declaration, no constructor syntax. For single-shot generation of simple resources, that compactness matters. But the picture changes once you account for deployability and refactoring.

What the data shows

HCL’s token advantage is real for simple generation. But agents don’t just generate once. They validate, repair failures, and refactor. We built a benchmark that measures the full cycle: an open-source tool that sends identical prompts to Claude Opus 4.6 (claude-opus-4-6) and OpenAI GPT-5.2-Codex (gpt-5.2-codex), requesting both Terraform HCL and Pulumi TypeScript for the same AWS infrastructure (VPC with public and private subnets across 2 AZs, an EC2 instance with security groups for SSH and HTTP, and an RDS PostgreSQL instance with a security group allowing port 5432 only from the EC2 security group, plus all networking: internet gateway, NAT gateway, route tables, and associations). We measured token consumption, cost, and deployability across two scenarios: initial generation and refactoring into reusable components.

Methodology: Temperature 0, 5 runs per combination, randomized execution order. Each generated output goes through a three-stage validation pipeline: formatting (terraform fmt for HCL, prettier for TypeScript), static analysis (terraform validate for HCL, tsc --noEmit for TypeScript), and provider-level validation (terraform plan for HCL, pulumi preview for TypeScript). Both plan and preview check against real AWS provider schemas without creating resources, making their pass rates comparable across formats. If plan/preview fails, the benchmark feeds the error back to the model for one self-repair attempt. At temperature 0, Claude Opus 4.6 produced near-identical outputs across runs (sd=0-4 tokens). GPT-5.2-Codex showed more natural variation (sd=130-165 tokens). With 5 runs per combination the results are directional, not statistically conclusive. Costs are estimates based on published pricing as of 2026-02-22. Full methodology and reproducible code:

github.com/dirien/iac-token-benchmark

Scenario 1: Generation

HCL uses fewer tokens for generation:

Provider	Format	Output tokens (mean)	LOC (mean)	Cost (mean)	Plan/Preview pass	Repairs needed
Claude Opus 4.6	Terraform	2,007	212	$0.051	5/5	0/5
Claude Opus 4.6	Pulumi TS	2,555	200	$0.065	5/5	0/5
GPT-5.2-Codex	Terraform	1,565	110	$0.022	2/5	2/5
GPT-5.2-Codex	Pulumi TS	2,322	147	$0.033	0/5	5/5

HCL produces 21-33% fewer output tokens across both models. For simple resource generation, this translates directly to lower cost. Pulumi TypeScript uses more tokens for fewer lines of code because imports, type annotations, and constructor syntax add tokens without adding functional lines.

The Plan/Preview column tells a more complete story. Claude Opus 4.6 produced deployable code on the first pass for both formats: 5/5 for Terraform and 5/5 for Pulumi. Neither needed repairs. GPT-5.2-Codex struggled with both formats, but Terraform fared slightly better (2/5 vs 0/5).

Scenario 2: Refactoring into reusable components

We took each model’s generation output and asked it to refactor the code into a reusable module or component with parameterized environment name, instance sizes, and availability zone count.

Provider	Format	Output tokens (mean)	LOC (mean)	Cost (mean)	Plan/Preview pass
Claude Opus 4.6	Pulumi TS	2,720	218	$0.082	5/5
Claude Opus 4.6	Terraform	3,379	345	$0.095	5/5
GPT-5.2-Codex	Pulumi TS	2,477	248	$0.038	4/5
GPT-5.2-Codex	Terraform	1,356	119	$0.021	0/5

This is where the results get interesting. Opus + Pulumi refactoring used 20% fewer tokens, cost 14% less, and passed pulumi preview on every run (5/5) with zero repairs. Opus + Terraform also ended up at 5/5 for terraform plan, but it needed repair cycles to get there. The benchmark run log tells the story:

# Pulumi refactoring: runs 29-33, sequential, no gaps = zero repairs
✓ [29/40] anthropic/pulumi-ts/refactor run 1 — 2721 tokens, $0.0817
✓ [30/40] anthropic/pulumi-ts/refactor run 2 — 2721 tokens, $0.0817
✓ [31/40] anthropic/pulumi-ts/refactor run 3 — 2721 tokens, $0.0817
✓ [32/40] anthropic/pulumi-ts/refactor run 4 — 2721 tokens, $0.0817
✓ [33/40] anthropic/pulumi-ts/refactor run 5 — 2714 tokens, $0.0816

# Terraform refactoring: 34→36→38→40→42, every run skips a number = every run triggered self-repair
✓ [34/40] anthropic/terraform/refactor run 1 — 3388 tokens, $0.0956
✓ [36/40] anthropic/terraform/refactor run 2 — 3339 tokens, $0.0944
✓ [38/40] anthropic/terraform/refactor run 3 — 3390 tokens, $0.0957
✓ [40/40] anthropic/terraform/refactor run 4 — 3388 tokens, $0.0956
✓ [42/40] anthropic/terraform/refactor run 5 — 3388 tokens, $0.0956

The skipped numbers (35, 37, 39, 41) are self-repair turns where the model received terraform plan errors and regenerated the code. Each repair consumed additional tokens that do not show up in the first-generation cost but do show up in the total pipeline cost.

GPT-5.2-Codex tells a different story. Both formats needed repair on every run, but what happened after repair is what matters:

# Codex + Terraform refactoring: repaired, but still failed plan (0/5 deployable)
terraform run 1 turn 2: plan_valid=False tokens=1559
terraform run 2 turn 2: plan_valid=False tokens=1165
terraform run 3 turn 2: plan_valid=False tokens=1068
terraform run 4 turn 2: plan_valid=False tokens=1134
terraform run 5 turn 2: plan_valid=False tokens=1101

# Codex + Pulumi refactoring: repaired, and preview passed (4/5 deployable)
pulumi-ts run 1 turn 2: plan_valid=True tokens=2246
pulumi-ts run 2 turn 2: plan_valid=True tokens=2567
pulumi-ts run 3 turn 2: plan_valid=True tokens=2187
pulumi-ts run 4 turn 2: plan_valid=True tokens=2531
pulumi-ts run 5 turn 2: plan_valid=False tokens=2647

Codex + Terraform used fewer tokens but produced zero deployable refactored code after repair. Codex + Pulumi used more tokens but recovered to deployable code 4 out of 5 times. TypeScript’s type errors gave the model enough information to fix the problems. HCL’s plan errors did not.

Total pipeline cost

This is the number that matters for production agent workflows. It includes generation, any self-repair cycles, and refactoring:

Provider	Format	Total tokens (mean)	Total cost (mean)
Claude Opus 4.6	Pulumi TS	8,183	$0.146
Claude Opus 4.6	Terraform	14,669	$0.249
GPT-5.2-Codex	Terraform	8,723	$0.084
GPT-5.2-Codex	Pulumi TS	15,211	$0.138

Opus + Pulumi had the lowest total pipeline cost at $0.146, 41% cheaper than Opus + Terraform at $0.249. The difference comes entirely from repair cycles: Pulumi needed zero repairs across both scenarios, while Terraform refactoring triggered self-repair on every run.

With Codex, Terraform had the lower pipeline cost ($0.084 vs $0.138), driven by its smaller token output. But Codex + Terraform produced zero deployable refactored code (0/5 plan pass), while Codex + Pulumi produced deployable code 4 out of 5 times.

What this means

HCL uses fewer tokens per generation. For single-shot resource creation, HCL’s compactness saves 21-33% on output tokens. That advantage is consistent across both models.
Pulumi produces deployable refactored code more reliably. With Opus, Pulumi refactoring passed preview 5/5 with zero repairs. Codex + Pulumi passed 4/5. Codex + Terraform passed 0/5.
Total pipeline cost favors Pulumi with Opus. Opus + Pulumi cost 41% less than Opus + Terraform across the full pipeline ($0.146 vs $0.249), because Terraform refactoring needed repair cycles that Pulumi did not.
The tradeoff depends on your model and workflow. Codex + Terraform is cheapest on raw tokens but produces no deployable refactored code. Codex + Pulumi costs more per token but actually deploys. Opus + Pulumi is the best of both: fewer refactoring tokens and zero repairs.

Why Pulumi refactoring deploys cleaner

The TerraFormer project identified what they call the correctness-congruence gap: LLMs generate configurations that look valid but fail to match the user’s architectural intent. An error taxonomy study cataloged the same pattern across models. A survey of LLMs for IaC found the gap between syntax validity and architectural correctness widens with infrastructure complexity.

Refactoring is where this gap bites hardest. Turning a flat resource list into a parameterized, reusable module requires the model to restructure dependencies, introduce variables, and compose abstractions. With Pulumi, the model can use TypeScript’s standard refactoring patterns: extract a class, add typed constructor parameters, compose functions. These are patterns it has practiced across millions of repositories during training. With HCL, the same refactoring requires count, for_each, dynamic blocks, and module variable plumbing, domain-specific constructs that have far less representation in training data.

Our benchmark confirms this directly. Opus produced 2,720 tokens for Pulumi refactoring versus 3,379 for Terraform, a 20% reduction, and every Pulumi run passed pulumi preview without repair. The Terraform refactoring runs all triggered self-repair because the restructured HCL modules had issues that terraform plan caught.

Training data distribution makes this structural. LLMs have far more TypeScript than HCL in their corpora. A model refactoring TypeScript draws on patterns from the entire open-source ecosystem. A model refactoring HCL modules has a much smaller pool. Since general-purpose languages dominate new code production, this gap will widen over time.

Tooling can close the gap further. The Pulumi MCP server gives AI agents direct access to resource schemas at generation time. A tool like get-resource returns every property, type, and required field for a given cloud resource. The agent does not have to guess from what it memorized during training. It can look up the correct schema before writing a single line of code.

This changes the workflow from “generate, fail, read error, retry” to “look up schema, generate correctly.” Agent skills push this further by encoding working Pulumi idioms as structured prompts, so the model starts from a known-good baseline. Terraform has no equivalent to this MCP-based schema lookup. That difference matters more with every iteration.

Let agents generate deployable infrastructure

Point Claude Code, Codex, or any agent at the Pulumi MCP server so it looks up real resource schemas and generates infrastructure as code that deploys.

Get started

Where the industry is heading

One way to think about IaC language choice is through the lens of AI engineering maturity levels. At Level 3 (agentic coding), agents generate infrastructure from prompts. HCL’s 21-33% token savings on generation matters here. At Levels 4-5, agents iterate on specifications, refactor code, and maintain systems over time. Our benchmark shows this is where Pulumi pulls ahead: 41% lower total pipeline cost with Opus, and more deployable refactored output with both models.

The industry is moving toward Levels 4-5. Agents are taking on refactoring, feature flags, environment parameterization. 43% of DevOps teams need four or more deployment iterations before infrastructure is production-ready. The first-generation token advantage HCL holds applies to the task that is shrinking as a share of agent workloads. The refactoring and deployability advantages that Pulumi offers apply to the tasks that are growing.

Choosing based on your workflow

If your agents primarily generate well-scoped resource definitions, HCL saves 21-33% on output tokens. That advantage is real and consistent.

If your agents need deployable output on the first pass (which avoids repair costs entirely), our data shows Opus + Pulumi is the strongest combination: 5/5 plan/preview pass for both generation and refactoring, zero repairs, lowest total pipeline cost.

If your agents evolve infrastructure over time through refactoring and modularizing, Pulumi produced deployable refactored code more reliably across both models we tested (5/5 and 4/5 vs 5/5 and 0/5 for Terraform).

Pulumi covers the full iteration loop, from generation through repair to refactoring, using the same language patterns and tooling that models already know.

Get started

Ready to explore how AI agents work with Pulumi? Check out Pulumi AI to see LLM-powered infrastructure generation in action, or get started with Pulumi’s documentation.

Join the conversation in the Pulumi Community Slack or Pulumi Community Discussions.

GitOps Best Practices I Wish I Had Known Before

Engin Diri — Thu, 19 Feb 2026 00:00:00 +0000

Getting started with GitOps can feel like trying to herd cats through a YAML factory while the factory is on fire. It’s one of those things that seems like it ought to be simple (just use Git!), but in practice is much more complex — and you may not realize how much more complex until you’re weeks or more into a project. After years of running GitOps workflows in production across dozens of clusters, I’ve collected a list of best practices that I’m hoping can save you from having to make many of the mistakes I’ve made. Think of it as the GitOps cheat sheet I wish I’d had from Day 1.

If you’re not familiar with the formal definition, the OpenGitOps project distills it into four principles:

Declarative desired state
Versioned and immutable storage
Automatic pulling
Continuous reconciliation

But those principles only define the what. This post is also about the how — the practical lessons that can make or break a GitOps implementation.

In this post, I’ll walk you through the GitOps best practices I’ve picked up from production experience, community talks, and more than a few late-night incident calls. Whether you’re just getting started with GitOps or looking to level up, these tips should help you avoid the potholes.

The GitOps mindset in a nutshell.

1. Git is your single source of truth (no, really)

This is the bedrock that the principles of GitOps are built on: every piece of your environment’s desired state lives in a Git repository. No exceptions. No “I’ll just fix it real quick with kubectl edit.” No “let me patch this configmap by hand because the PR process takes too long.”

The moment you make a manual change to your cluster, you’ve created drift between what Git says the world should look like and what it actually looks like. Drift will quietly wreck your GitOps workflows if you let it.

Put everything in Git: Kubernetes manifests, Helm values, Kustomize overlays, policy rules, even your GitOps tool configuration itself.
No manual kubectl edits. If it’s not in Git, it doesn’t exist. Period. Train your team to treat direct cluster changes like touching a hot stove.
You get an audit trail for free. Git gives you a complete history of who changed what, when, and why. That’s your compliance audit trail baked right in.

Pro Tip: Enable branch protection rules on your GitOps repos from Day 1. This prevents anyone from pushing directly to main and bypassing the review process. Future you will thank present you during the next audit.

2. Declarative over imperative, always

If you’re still running sequences of kubectl create, kubectl patch, and kubectl delete commands to manage your cluster, you’re not really doing GitOps yet. Declarative means you define what you want the end state to look like, not the step-by-step instructions to get there.

Think of it like ordering at a restaurant. Imperative: “Go to the kitchen, grab flour, knead dough, preheat the oven to 200 degrees, shape the pizza, add sauce…” Declarative: “One margherita pizza, please.” Let the system figure out how to make it happen.

Your manifests describe the end result. The GitOps operator reconciles reality to match.
It’s idempotent by design. Apply the same manifest ten times, get the same result. No side effects, no surprises.
Rollbacks are easier: just revert a commit. The operator sees the previous desired state and reconciles. A caveat though: this only works for stateless resources; database schema migrations, CRD version changes, persistent volume modifications, and rotated secrets don’t always revert cleanly by rolling back to a previous commit. Be sure to plan rollbacks involving stateful resources carefully.

Pro Tip: If you find yourself writing shell scripts that run a sequence of kubectl commands, stop and ask yourself: “Can I express this as a declarative manifest instead?” Nine times out of ten, the answer is yes.

3. Pull-based deployments are the way

Traditional CI/CD is push-based: your pipeline builds an artifact and then pushes it to the cluster. GitOps flips this. An agent running inside your cluster continuously polls a Git repository and pulls changes when it detects them.

Why does this matter? With push-based deployments, your CI system needs credentials to access your cluster. That’s a wide attack surface. With pull-based, the agent already lives inside the cluster and only needs read access to your Git repo.

Tools like ArgoCD and FluxCD run controllers inside your cluster that watch your Git repos.
Your CI pipeline never needs kubeconfig access. The agent handles deployment.
The agent doesn’t just deploy once; it continuously ensures the cluster matches the desired state in Git.

Pro Tip: You may be tempted to set your reconciliation interval to something aggressive like 1 minute so you always know exactly which version is deployed. That works for a while, but at scale (200+ applications polling every minute) you can blow through GitHub’s API rate limits (5,000 requests/hour for authenticated users) and put real pressure on the Kubernetes API server.

A better approach: set up webhook receivers. Both ArgoCD and FluxCD support incoming webhooks from GitHub, GitLab, and Bitbucket. Your Git provider pings the GitOps controller on every push, so reconciliation kicks off in seconds instead of waiting for the next poll. That alone kills most of the API rate limit pain at scale. You can then relax polling to a 5 or 10 minute fallback for the rare case where a webhook doesn’t fire.

4. Separate app code from deployment config (when it hurts not to)

Let me be upfront: if you’re a small team with one or two services, keeping your Kubernetes manifests next to your application source code in the same repo is fine. A monorepo is simpler to manage and one less thing to automate. Don’t split repos just because a blog post told you to.

That said, ArgoCD’s official best practices call separate repos “highly recommended,” and there are real reasons for that once you grow. Application code and deployment configuration have different lifecycles. You might bump resource limits in a Helm values file without touching a single line of app code. Or you might refactor your entire codebase without changing any deployment parameters. In a shared repo, every config tweak triggers your full CI pipeline, and ArgoCD invalidates the manifest cache for all applications in the repo on every commit, not just the ones that changed. That cache invalidation alone can become a performance problem with dozens of apps in one repo.

When config tweaks start blocking app code from getting through CI, it may be time to split.
When different teams need different access levels (not everyone who pushes app code should modify production deployment config), it’s time to split.
When you’re running microservices with independent release cadences that step on each other, it’s time to split.

If none of those apply yet, don’t split. You’ll know when the pain arrives.

Pro Tip: When you do split, a common pattern is two repos: one for application source code, one for deployment configuration (Helm charts, Kustomize overlays, environment-specific values). Some larger orgs go further with a third repo for environment overrides, following the fleet repo model from the FluxCD maintainers. Either way, pair it with image update automation (Flux Image Automation Controller or ArgoCD Image Updater) so image tag bumps across repos don’t turn into manual toil.

5. Use directories, not branches, for environments

This is the number one anti-pattern I encounter in the wild. Teams create a dev branch, a staging branch, and a prod branch, then cherry-pick or merge between them for promotions. It sounds logical, but it’s a trap. Branches diverge, cherry-picks get missed, configs meant for dev sneak into staging or prod, and before you know it, your environments have drifted, and you can’t tell what’s actually different between them.

A folder structure like environments/dev/, environments/staging/, environments/prod/ makes differences visible in a single diff command.
Moving a change from dev to prod is just copying or updating files across directories, reviewed via a standard pull request.
You’ll never accidentally skip a commit or introduce an environment-specific change where it doesn’t belong. No more cherry-pick roulette.

One thing to note: the commonly used rendered manifests pattern actually does use per-environment branches, but not in a human-managed way. In that pattern, config changes are still committed to main (using named directories as above), and an automated CI process pushes environment YAML to read-only environment branches, which ArgoCD reads from. If you want to try it, ArgoCD’s built-in source hydrator and Kargo Render can automate the rendering for you.

To keep things DRY, you can look to a tool like Kustomize, which lets you use a shared base environment and then apply per-environment patches. Your base directory holds the common manifests, and each environment directory contains only the differences. For example:

gitops-repo/
├── base/
│ ├── deployment.yaml
│ ├── service.yaml
│ └── kustomization.yaml
├── environments/
│ ├── dev/
│ │ ├── kustomization.yaml # replicas: 1, limits: 256Mi
│ │ └── patches/
│ ├── staging/
│ │ ├── kustomization.yaml # replicas: 2, limits: 512Mi
│ │ └── patches/
│ └── prod/
│ ├── kustomization.yaml # replicas: 3, limits: 1Gi
│ └── patches/

This minimizes duplication while keeping environment boundaries clear.

Pro Tip: If you’re working at the IaC layer, Pulumi stack configuration solves a similar problem: each stack (dev, staging, prod) has its own config file with environment-specific values, while the program logic stays shared. It’s the same principle of “one source, per-environment overrides” applied above the Kubernetes manifest level.

6. Validate before you merge

Catching errors after they’ve been applied to your cluster is expensive. Catching them in CI before the merge is cheap. Shift left as hard as you can.

Your GitOps CI pipeline should validate everything it can before the change reaches your cluster: YAML syntax, Kubernetes schema validation, policy compliance, dry-run rendering.

Tools like yamllint and kubeconform catch syntax errors and schema violations before they become runtime failures. (Note: kubeval, which you’ll see in older guides, is no longer maintained — its own README points to kubeconform as the replacement.)
Run helm template or kustomize build in CI to verify that your templates render without errors.
Use OPA/Conftest or Kyverno to enforce organizational policies (no privileged containers, required labels, resource limits set) before merge.

Here’s a minimal GitHub Actions workflow that validates your GitOps manifests on every PR:

name: Validate GitOps Manifests
on:
 pull_request:
 paths: ["environments/**"]

jobs:
 validate:
 runs-on: ubuntu-latest
 steps:
 - uses: actions/checkout@v4
 - name: Validate YAML schemas
 run: kubeconform --strict environments/**/*.yaml
 - name: Build and verify Kustomize output
 run: |
 for env in environments/*/; do
 kustomize build "$env" > /dev/null
 done
 - name: Diff against live cluster
 run: |
 kustomize build environments/staging/ | \
 kubectl diff -f - || true

Pro Tip: Add a kubectl diff step in your CI pipeline that shows exactly what would change in the cluster if the PR were merged. This gives reviewers a concrete view of the impact, not just the YAML diff.

7. Tag with commit SHAs, not “latest”

Using latest as your image tag in a GitOps workflow is playing fast and loose with your deployments. You have no idea what version is actually running, you can’t reliably roll back, and your Git history becomes meaningless because the same manifest could deploy completely different code depending on when it was synced.

Tag your container images with the Git commit SHA that produced them. This creates a direct link between your source code, your build, and your deployment.
Once a tag is pushed, it should never be overwritten. No re-tagging, no “oops let me push a fix to the same tag.”
Need to roll back? Just revert the commit that updated the image tag. The previous SHA still points to the exact same image it always did.

Pro Tip: Set up an admission controller or CI policy that rejects any manifest using latest or any other mutable tag. Make it impossible to deploy without a pinned version.

8. Automate drift detection and reconciliation

Drift is when your cluster’s actual state doesn’t match the desired state in Git. It happens when someone runs a manual kubectl command, when an autoscaler changes a replica count, or when a CRD controller modifies a resource behind your back.

The whole point of GitOps is continuous reconciliation. Your operator should constantly compare the live state against Git and bring things back in line.

Configure your GitOps tool to automatically revert manual changes, but build your exclusion lists (the set of fields and resources you intentionally skip during reconciliation) before you turn on auto-sync, not after. Controllers like Istio, cert-manager, and Crossplane legitimately modify resources, and auto-remediating those changes creates reconciliation loops that can destabilize a cluster. Start with a conservative exclusion list and tighten it over time.
Even with auto-remediation, you want to know when drift happens. Set up alerts. It might indicate a process problem or a team member who needs coaching.

Pro Tip: Be precise about what you reconcile. HPAs, VPAs, cert-manager annotations, and external-dns records all legitimately modify resources. Use ignore rules to exclude fields that are intentionally dynamic. In ArgoCD, that looks like this:

spec:
 ignoreDifferences:
 - group: apps
 kind: Deployment
 jsonPointers:
 - /spec/replicas
 - group: admissionregistration.k8s.io
 kind: MutatingWebhookConfiguration
 jqPathExpressions:
 - .webhooks[]?.clientConfig.caBundle

Also, don’t overlook resource ordering. ArgoCD sync waves and Flux’s dependsOn let you enforce that CRDs are applied before custom resources and namespaces before workloads. Getting ordering wrong is one of the most common causes of failed syncs in production.

9. Practice progressive delivery

Progressive delivery lets you gradually roll out changes, verify they’re working, and automatically roll back if something goes wrong.

GitOps and progressive delivery fit well together. Your Git repo describes the desired rollout strategy, and controllers in the cluster execute it.

Canary deployments: send a small percentage of traffic to the new version. If error rates spike, automatically roll back.
Blue-green deployments: run two identical environments, switch traffic once the new one is verified healthy.
Argo Rollouts extends Kubernetes with canary, blue-green, and analysis-driven rollout strategies that integrate with ArgoCD.
Flagger is the Flux ecosystem equivalent, with automated canary analysis, A/B testing, and blue-green deployments using service mesh or ingress controllers.
Kargo takes it a step further by orchestrating promotions across multiple stages and environments. Instead of manually wiring up pipelines to move changes from dev to staging to prod, Kargo automates the entire promotion workflow on top of ArgoCD. I did a deep dive on GitOps promotion tools and why they belong in your toolkit:

Pro Tip: Combine progressive delivery with automated analysis. Tools like Argo Rollouts and Kargo can query Prometheus metrics during a canary and automatically promote or roll back based on error rates, latency percentiles, or custom metrics.

10. Policy-as-code: your automated guardrails

Reviews are great, but humans make mistakes. Policy-as-code adds an automated layer that enforces organizational rules before a change can land in your cluster. Think of it as guardrails on a mountain road: they don’t slow you down, they just keep you from going over the edge.

OPA/Gatekeeper lets you define policies in Rego that the admission controller enforces at deploy time (e.g., no containers running as root, all deployments must have resource limits).
Kyverno is a Kubernetes-native policy engine that uses familiar YAML to define and enforce policies, including mutation and generation.
Starting with Kubernetes 1.26+, you can write CEL-based ValidatingAdmissionPolicies natively without any external tooling. A lightweight option if you don’t need the full feature set of OPA or Kyverno.
Pulumi CrossGuard lets you write policy packs in TypeScript or Python that validate infrastructure before it’s deployed. If your IaC and GitOps layers are bridged (see the next section), CrossGuard can enforce rules across both.
Run policy checks in your CI pipeline so violations are caught during the PR, not after deployment.

Pro Tip: Start with a small set of high-impact policies (required labels, no privileged containers, resource limits) and expand over time. Trying to enforce 50 policies on Day 1 will create so much friction that teams will revolt.

Try infrastructure as code with Pulumi

Define your cloud resources in TypeScript, Python, or Go, then pass stack outputs straight into your GitOps manifests so both layers stay in sync.

Get started

11. Bridge your IaC and GitOps (don’t choose one)

I hear this all the time: “We use Terraform for infrastructure and ArgoCD for deployments, and they live in completely separate worlds.” Sound familiar? Most teams treat IaC and GitOps as independent workflows, but they’re really two halves of the same story.

IaC handles “Day 0,” creating the cloud resources your cluster depends on: VPCs, the Kubernetes cluster itself, IAM roles, databases, DNS zones. GitOps handles “Day 2,” managing what runs inside the cluster: applications, addons, configurations. The gap between them is where things get messy. Your Helm charts need IAM role ARNs. Your GitOps-deployed addons need to know the cluster’s OIDC provider endpoint. That metadata has to flow from the IaC layer to the GitOps layer somehow.

The gitops-bridge pattern solves this cleanly: IaC provisions cloud resources and writes metadata (role ARNs, account IDs, endpoints) into Kubernetes resources (like ConfigMaps or Secrets) that your GitOps tool can consume directly.

Keep each tool in its lane. Don’t force Terraform’s Helm provider to fight with ArgoCD over resource ownership.
Pulumi lets you define your cloud infrastructure in real programming languages (TypeScript, Python, Go), which makes it natural to compute and pass metadata to your GitOps layer. You can use stack outputs to expose values like role ARNs and cluster endpoints, then feed them into your GitOps manifests. The Pulumi Kubernetes Operator even lets ArgoCD reconcile Pulumi stacks via GitOps.
Statsig went from “1-2 devs clicking around cloud consoles with SEVs left and right” to fully self-service by using Pulumi to generate manifests and ArgoCD to deploy them.
As you grow beyond a handful of clusters, think multi-cluster from the start. Patterns like ArgoCD ApplicationSets with cluster generators and Flux’s Kustomization targeting become important for fleet management. The gitops-bridge pattern works well here because your IaC layer can register new clusters and write their metadata into Kubernetes resources, which ApplicationSets automatically pick up.

Here’s how the gitops-bridge looks in practice. Pulumi provisions cloud resources and writes metadata into a ConfigMap that ArgoCD consumes:

import * as k8s from "@pulumi/kubernetes";

const clusterMetadata = new k8s.core.v1.ConfigMap("cluster-metadata", {
 metadata: {
 name: "cluster-metadata",
 namespace: "argocd",
 labels: {
 "gitops-bridge": "true",
 },
 },
 data: {
 aws_account_id: "123456789012",
 cluster_name: "prod-us-east-1",
 iam_role_arn: "arn:aws:iam::123456789012:role/app-role",
 oidc_provider: "oidc.eks.us-east-1.amazonaws.com/id/EXAMPLE",
 },
});

import pulumi_kubernetes as k8s

cluster_metadata = k8s.core.v1.ConfigMap(
 "cluster-metadata",
 metadata=k8s.meta.v1.ObjectMetaArgs(
 name="cluster-metadata",
 namespace="argocd",
 labels={
 "gitops-bridge": "true",
 },
 ),
 data={
 "aws_account_id": "123456789012",
 "cluster_name": "prod-us-east-1",
 "iam_role_arn": "arn:aws:iam::123456789012:role/app-role",
 "oidc_provider": "oidc.eks.us-east-1.amazonaws.com/id/EXAMPLE",
 },
)

Your ArgoCD Applications or Helm values can then reference this ConfigMap, closing the loop between IaC and GitOps without hardcoding cloud-specific values.

Pro Tip: If you’re using Terraform today, the gitops-bridge pattern works there too. But if you’re tired of wrangling HCL and want type-safe infrastructure code with real programming constructs, give Pulumi a look. The bridge from IaC to GitOps becomes much more natural when both sides speak a real programming language.

12. Be pragmatic, not dogmatic

Michael Crenshaw from Intuit gave a talk titled “How GitOps Should I Be?” at a CNCF conference. Intuit runs 45 ArgoCD instances managing 20,000 applications across 200 clusters. The approach they took: “Be strategic, be pragmatic. GitOps most of the time.”

One of the largest GitOps adopters in the world explicitly deviates from pure GitOps when the situation calls for it. Dogmatic adherence to principles that don’t serve you is just another form of technical debt.

Intuit found that GitHub’s SLA of 20 minutes meant they couldn’t rely on GitOps for region evacuation, so they wrote a kubectl cron job that bypasses Git for failover. Pragmatic.
Massive Jenkins ecosystems don’t get rewritten overnight. Intuit’s compromise: the last step in the Jenkins pipeline calls argocd app sync. It’s push-based, yes, but the source of truth stays in Git. The Jenkins pipeline doesn’t own the desired state; it just tells ArgoCD “go reconcile now” instead of waiting for the next poll. It’s a stepping stone while teams migrate off Jenkins, not a permanent architecture.
Continuous reconciliation with self-heal (auto-reverting manual changes) is where most teams deviate. In practice, many teams start with automated drift detection and alerting (so you know when someone runs a manual kubectl command) but leave auto-remediation off until they’ve built confidence in their exclusion lists. That’s a reasonable middle ground.

Pro Tip: Document your intentional deviations. Create an ADR (Architecture Decision Record) for each case where you’ve chosen not to follow pure GitOps, explaining why and what the plan is to close the gap (if ever). This turns exceptions into conscious decisions rather than accidental shortcuts.

Final thoughts

GitOps isn’t a silver bullet, but it’s one of the best approaches we have for managing Kubernetes at scale. The practices in this list aren’t theoretical. They’re what I’ve learned from running GitOps in production, watching talks from teams at Intuit and Statsig, and making every mistake in the book at least once.

If I had to boil it all down to one sentence: treat your GitOps repo like production code, because it is.

Start with the basics (Git as source of truth, declarative config, pull-based delivery) and layer on the advanced practices as your team matures. Don’t try to implement all 12 on Day 1. Pick the three that address your biggest pain points and iterate from there.

And remember, even the largest GitOps adopters in the world aren’t doing it perfectly. They’re being pragmatic. You should be too.

How Pulumi fits into your GitOps workflow

Several of the practices in this post touch on problems Pulumi is built to solve. If you’re bridging IaC and GitOps, managing environment-specific configuration, or enforcing policy across both layers, here’s where to start:

Pulumi IaC: Define cloud infrastructure in TypeScript, Python, or Go and use stack outputs to feed metadata into your GitOps manifests.
Pulumi ESC: Manage secrets and configuration across environments with fine-grained access controls, integrated with Kubernetes via the External Secrets Operator or the Secret Store CSI Driver.
Pulumi Kubernetes Operator: Let ArgoCD reconcile Pulumi stacks via GitOps, closing the loop between your IaC and Day 2 workflows.

The Claude Skills I Actually Use for DevOps

Engin Diri — Mon, 09 Feb 2026 00:00:00 +0000

When Claude Code first released skills, I ignored them. They looked like fancy prompts, another feature to add to the pile of things I would get around to learning eventually. Then I watched a few engineers demonstrate what skills actually do, and something clicked. By default, language models do not write good code. They write plausible code based on what they have read. Plausible code turns into bugs, horrible UX, and infrastructure that breaks at 3am.

Skills fill that gap. They package engineering expertise into something Claude can use. The workflows and judgment matter more than the raw information. Without skills, every conversation starts from zero. You explain the same conventions and correct the same mistakes. Every morning, back to zero.

Think about what separates a junior engineer from a senior one. Both can write code that compiles. Both can deploy infrastructure that runs. The difference is that the senior engineer knows the patterns that prevent problems before they happen. They know when to use component resources instead of plain resources. They know that creating infrastructure inside an apply() callback (Pulumi’s way of transforming outputs that are not known until deployment) breaks preview. They know that hardcoded credentials will eventually end up in a git log somewhere embarrassing.

This knowledge takes years to accumulate through painful experience. Skills let you transfer that knowledge to Claude in minutes. And here is the thing that makes them practical: if you find yourself doing the same type of task with different content each time, that is a skill waiting to be built. You encode the process once, then feed it new inputs forever.

The mechanic, the tools, and the manual

I heard an analogy recently that made skills click for me. Imagine Claude as a mechanic. A capable mechanic who knows engines, can diagnose problems, and fix most cars that come through the shop.

MCP servers are like giving that mechanic a set of tools. Wrenches, diagnostic equipment, lift systems. Without tools, the mechanic cannot do much. With tools, the mechanic can work on whatever comes through the door.

But what happens when someone brings in a Formula 1 race car? Or a 1967 Ford Mustang that has been modified beyond recognition? The mechanic knows engines in general, but these specific vehicles require specific knowledge. The F1 car has procedures that must be followed in exact order. The vintage Mustang has quirks that only someone who has worked on that model would know.

Skills are the user manuals and standard operating procedures for these specific vehicles. They tell the mechanic what needs to happen and when. They encode the expertise of someone who has done this work a thousand times.

Or think about it from a carpenter’s perspective. Skills are the process to make the table: the measurements, the design, the exact steps. MCPs are the tools: the saw, the hammer, the drill. You need both. The process alone is theoretical, and tools without a process just sit in the garage.

For DevOps engineers working with Pulumi, this matters because infrastructure as code has its own quirks and patterns. Generic AI assistance produces code that looks reasonable but breaks conventions the community learned the hard way. Skills teach Claude those conventions.

Why skills instead of MCPs

Before skills clicked for me, I tried solving the expertise problem with MCPs. I kept adding servers until I noticed Claude getting slower and making worse decisions. Turns out the GitHub MCP alone eats 46,000 tokens across 91 tools before you type anything. Cursor eventually capped MCPs at 40 tools because too many options made everything worse.

Slash commands were another option, but you had to remember to invoke them. Anthropic apparently agreed, because in January 2026 they merged slash commands into skills. One unified system instead of two.

Skills avoid this through progressive disclosure. Claude reads just the description at startup, maybe a hundred tokens. The full procedures only load when Claude decides they are relevant. Unlike those massive system prompts that used to eat through your context window, skills stay out of the way until they are needed. For DevOps engineers running long infrastructure sessions with dozens of resources, this matters. You keep your context budget for the actual work instead of burning it on instructions. Skills can also fork context, spinning up isolated subagents that do work without polluting your main conversation. Think of it like handing a colleague a written brief. They go work on it, hand back a summary, and never sit in on your conversation.

I still use MCPs for connecting Claude to external systems. The Pulumi MCP server lets Claude query the registry and validate code. But MCPs give Claude access to things. Skills teach Claude how to think about things. Different jobs. They get more useful when you combine them. One engineer built a financial reporting skill that connects to his Mercury bank account via MCP, pulls every transaction for a given month, classifies the expenses into categories, and generates a styled HTML report with totals and breakdowns. A skill that knows your deployment process connecting to MCPs that talk to your actual infrastructure is the same idea, just pointed at ops instead of accounting.

Skills are portable. They follow an open standard, so a skill you write for Claude Code works in Cursor, GitHub Copilot, or anywhere else that supports agent skills. You can even copy the skill content into ChatGPT as a starting prompt. No vendor lock-in.

Teaching Claude to write Pulumi like an expert

The first time you ask Claude to help with a Pulumi project, the process is painful. You have to explain the patterns you want. You correct mistakes. You explain why creating resources inside apply() breaks things. By the third or fourth project, you start copying your corrections from previous conversations.

I built the dirien/claude-skills pulumi-typescript skill after going through this painful process too many times. It knows the patterns that prevent common mistakes: Pulumi ESC (Environments, Secrets, and Configuration) integration, OIDC (OpenID Connect) instead of hardcoded access keys, ComponentResource abstractions (reusable groups of related resources), and proper output structuring so dependent stacks can consume them cleanly.

Skill	What it teaches Claude
`pulumi-typescript`	Pulumi with TypeScript, ESC secrets management, component patterns, and multi-cloud deployment

npx skills add https://github.com/dirien/claude-skills --skill pulumi-typescript

Skills install as markdown files in your project’s .claude/skills/ directory, so they travel with your repo and are easy to review.

The next time you ask Claude to create infrastructure, it applies these patterns automatically. You do not have to remember to invoke the skill or correct the same mistakes repeatedly.

The official Pulumi skills

Pulumi maintains its own skills repository at pulumi/agent-skills, which they announced recently. The repo includes skills for ComponentResource patterns, Automation API, and migration from Terraform, CDK, CloudFormation, and ARM. The two I use daily are pulumi-esc and pulumi-best-practices.

The pulumi-esc skill teaches Claude how to work with Pulumi ESC (Environments, Secrets, and Configuration). It knows the difference between pulumi env get, pulumi env open, and pulumi env run. It sets up OIDC for dynamic credentials, integrates with external secret stores like AWS Secrets Manager and Vault, and structures layered environment composition so your dev, staging, and production configs inherit from a shared base.

The pulumi-best-practices skill catches the mistakes that burn you in production. It stops Claude from creating resources inside apply() callbacks, enforces proper parent relationships in ComponentResources, encrypts secrets from day one, and makes sure pulumi preview runs before any deployment. These are the patterns that took me years to internalize, and now Claude follows them by default.

Skill	What it teaches Claude
`pulumi-esc`	Environment, secrets, and configuration management with OIDC, dynamic credentials, and secret store integration
`pulumi-best-practices`	Resource dependencies, ComponentResource patterns, secret encryption, and safe refactoring

npx skills add https://github.com/pulumi/agent-skills --skill pulumi-esc
npx skills add https://github.com/pulumi/agent-skills --skill pulumi-best-practices

Making Claude a monitoring expert by default

You deploy something, it works, and six months later something breaks and you realize you never added monitoring. We have all been there. The monitoring skills from the community teach Claude to add observability from the start.

The jeffallan/claude-skills repository contains a monitoring-expert skill that knows Prometheus, Grafana, and DataDog.

Skill	What it teaches Claude
`monitoring-expert`	Structured logging, metrics, distributed tracing, alerting, and performance testing for production systems

npx skills add https://github.com/jeffallan/claude-skills --skill monitoring-expert

In my testing, deploying a static website with these skills installed looks different from vanilla Claude. Instead of just creating the S3 bucket and CloudFront distribution, Claude asked about error rate thresholds before writing any code. It suggested CloudWatch alarms and created an SNS topic for alerts. The results are not always this clean. Sometimes the monitoring suggestions are generic or miss your specific SLO requirements. But the baseline shifted from “no monitoring at all” to “monitoring that needs tuning,” and that is a better starting point.

Kubernetes configuration that actually passes security review

Kubernetes has hundreds of configuration options. Most deployments use a handful of them. The problem is that the important options like security contexts, resource limits, and pod disruption budgets are easy to forget when you are focused on getting something to run.

The jeffallan/claude-skills kubernetes-specialist skill focuses on configurations that production deployments actually need. Without it, ask Claude for a deployment and you get something that runs: the right image, the right ports, maybe a service. With the skill, the same request comes back with runAsNonRoot: true in the security context, resource requests and limits that reflect actual usage patterns, liveness and readiness probes with sensible intervals, and a pod disruption budget. These are the things that make the difference between “it works in staging” and “it survives a node failure in production.” The skill also understands when RollingUpdate makes sense versus Recreate, which is the kind of judgment call that usually requires context a generic model does not have.

The wshobson/agents repository fills in the gaps around Kubernetes with CI/CD, cost management, and deployment workflows:

Skill	What it teaches Claude
`kubernetes-specialist`	Production cluster management, security hardening, and cloud-native architectures
`cost-optimization`	Cloud cost reduction across AWS, Azure, and GCP with right-sizing and reserved instances
`github-actions-templates`	CI/CD workflows, Docker builds, Kubernetes deployments, security scanning, and matrix builds
`gitops-workflow`	ArgoCD and Flux CD for automated Kubernetes deployments

npx skills add https://github.com/jeffallan/claude-skills --skill kubernetes-specialist
npx skills add https://github.com/wshobson/agents --skill gitops-workflow
npx skills add https://github.com/wshobson/agents --skill github-actions-templates
npx skills add https://github.com/wshobson/agents --skill cost-optimization

Debugging like a senior engineer

The obra/superpowers repository contains a skill that changed how I debug with Claude. The systematic-debugging skill implements a four-phase framework: root cause investigation, pattern analysis, hypothesis testing, and implementation.

Without this skill, Claude tends to suggest solutions immediately. Something is broken, here are five things that might fix it. This feels helpful but often wastes time because none of the suggestions address the actual problem.

With the systematic debugging skill, Claude approaches problems differently. It asks clarifying questions. It wants to see logs. It builds a model of what is happening before suggesting changes. When it proposes a fix, it explains why that fix addresses the root cause. Sometimes skills find problems you did not know about. One engineer pointed a skills-equipped Claude at a set of SEO pages and discovered they had been decaying for months with nobody watching. The infrastructure parallel is obvious: configuration drift, unused resources, permissions that expanded over time. A debugging skill that investigates before prescribing will find these things.

Skill	What it teaches Claude
`systematic-debugging`	Root cause investigation, pattern analysis, hypothesis testing, and verified implementation

npx skills add https://github.com/obra/superpowers --skill systematic-debugging

Catching security issues before they ship

Two skills cover different sides of security review. The wshobson/agents k8s-security-policies skill handles Kubernetes-specific hardening: NetworkPolicies, Pod Security Standards, RBAC, OPA Gatekeeper constraints, and service mesh mTLS configuration. The sickn33/antigravity-awesome-skills security-review skill covers application-level concerns like secrets management, SQL injection, XSS prevention, and input validation.

I asked Claude to check a Pulumi program that created an S3 bucket. Without the security skills, Claude confirmed the code was correct and moved on. With the skills loaded, it flagged that the bucket had no server-side encryption configured, the bucket policy allowed s3:* from an overly broad principal, and there was no access logging enabled. On the Kubernetes side, the k8s-security-policies skill catches things like missing default-deny NetworkPolicies and containers running as root. These skills are not a replacement for deterministic tools like tfsec, checkov, or trivy. Those catch known issues every time. Skills are probabilistic and work best as an extra layer during development, not as your only security gate.

Skill	What it teaches Claude
`k8s-security-policies`	Network policies, pod security standards, RBAC, and admission control for defense-in-depth
`security-review`	Secrets management, input validation, SQL injection, XSS/CSRF prevention, and dependency auditing

npx skills add https://github.com/wshobson/agents --skill k8s-security-policies
npx skills add https://github.com/sickn33/antigravity-awesome-skills --skill security-review

Incident response before you need it

At 3am when something breaks, you want runbooks. The incident-runbook-templates skill from wshobson/agents helps Claude create these before you need them. It includes a four-level severity model (SEV1 through SEV4) with response time expectations, escalation decision trees, and communication templates for status updates.

When you ask Claude to document your deployment process, it produces runbooks with diagnostic steps, rollback protocols, and verification checks. It knows kubectl commands for Kubernetes recovery and SQL procedures for PostgreSQL troubleshooting. The output needs editing. Generated runbooks tend to be thorough on the happy path but thin on the failure modes that matter most at 3am. I treat them as a first draft that gets me to 60% in minutes instead of hours, then fill in the gaps from experience.

Skill	What it teaches Claude
`incident-runbook-templates`	Detection, triage, mitigation, resolution, and communication procedures for production incidents

npx skills add https://github.com/wshobson/agents --skill incident-runbook-templates

General-purpose DevOps and SRE skills

The skills above target specific problems. The jeffallan/claude-skills repository also includes two broader skills that cover the day-to-day work that does not fit neatly into one category.

The devops-engineer skill gives Claude a senior DevOps engineer persona covering CI/CD pipelines, container management, deployment strategies like blue-green and canary, and infrastructure as code across AWS, GCP, and Azure. It enforces constraints I care about: no deploying to production without approval, no secrets in code, no unversioned container images.

The sre-engineer skill focuses on reliability: SLO/SLI definitions, error budget calculations, golden signal dashboards, and toil reduction through automation. It produces Prometheus/Grafana configs, remediation runbooks, and reliability assessments. If you run production systems and want Claude to think about error budgets instead of just uptime, this is the skill.

Skill	What it teaches Claude
`devops-engineer`	CI/CD pipelines, container management, deployment strategies, and infrastructure as code across clouds
`sre-engineer`	SLI/SLO management, error budgets, monitoring, automation, and incident response

npx skills add https://github.com/jeffallan/claude-skills --skill devops-engineer
npx skills add https://github.com/jeffallan/claude-skills --skill sre-engineer

Vetting the skills you install

Before you install everything in sight, a warning. Skills run with the same permissions as your AI agent. A malicious skill can exfiltrate credentials, download backdoors, or disable safety mechanisms, and it will look like your agent doing it.

Snyk researchers published ToxicSkills in February 2026 after scanning 3,984 skills from public registries. 13.4% had critical-level vulnerabilities, and they found 76 confirmed malicious payloads. The attack techniques included base64-encoded commands that steal AWS credentials, skills that direct you to download password-protected executables from attacker infrastructure, and jailbreak attempts that try to disable safety mechanisms. 91% of malicious skills combine code-level malware with prompt injection, so they attack on two fronts simultaneously.

Treat skills like you treat any third-party dependency:

Read the source before installing. Skills are markdown and YAML files. If you cannot read the full skill in a few minutes, that is a red flag.
Check the repository. Look at stars, contributors, and commit history. A single-commit repository from an unknown account deserves scrutiny.
Run uvx mcp-scan@latest --skills to scan installed skills for known malicious patterns, prompt injection, and credential exposure.
Be cautious with skills that fetch external content at runtime. The Snyk research found 17.7% of skills on ClawHub pull from third-party URLs, which means the skill’s behavior can change after you install it.
Stick to known repositories. Every skill recommended in this post comes from a repository with visible maintainers and community activity.

Eight malicious skills were still publicly available on ClawHub when Snyk published their findings. The skills ecosystem is young, and the vetting infrastructure is still catching up.

Build infrastructure with agent skills

Pulumi’s Agent Skills teach Claude Code, Cursor, or any coding agent the infrastructure patterns covered here. Install them, then build and deploy your stacks with fewer corrections.

Get started

Putting it together

Stacking skills is where this pays off. Install the Pulumi skills and Claude writes better infrastructure code. Add monitoring and security on top and you start catching problems that used to slip through to production.

A note on stacking: I have not hit conflicts running all of these simultaneously, but more skills means more descriptions for Claude to evaluate at startup. If you notice Claude getting slower or making odd choices, pare back to the skills you actually use for that project. Start with the Pulumi and monitoring skills, add others as you need them.

Here is how to set up a new project with all of them:

mkdir -p pulumi-skills-demo && cd pulumi-skills-demo
pulumi new aws-typescript --name skills-demo --yes

npx skills add https://github.com/dirien/claude-skills --skill pulumi-typescript
npx skills add https://github.com/pulumi/agent-skills --skill pulumi-esc
npx skills add https://github.com/pulumi/agent-skills --skill pulumi-best-practices
npx skills add https://github.com/obra/superpowers --skill systematic-debugging
npx skills add https://github.com/jeffallan/claude-skills --skill monitoring-expert
npx skills add https://github.com/jeffallan/claude-skills --skill kubernetes-specialist
npx skills add https://github.com/wshobson/agents --skill gitops-workflow
npx skills add https://github.com/wshobson/agents --skill github-actions-templates
npx skills add https://github.com/wshobson/agents --skill cost-optimization
npx skills add https://github.com/wshobson/agents --skill incident-runbook-templates
npx skills add https://github.com/jeffallan/claude-skills --skill devops-engineer
npx skills add https://github.com/jeffallan/claude-skills --skill sre-engineer
npx skills add https://github.com/wshobson/agents --skill k8s-security-policies
npx skills add https://github.com/sickn33/antigravity-awesome-skills --skill security-review

Then try these two prompts to see how many skills activate at once:

# Static website — triggers Pulumi TypeScript, monitoring, and security skills
Create a Pulumi TypeScript program for a static website on AWS with S3, CloudFront, OIDC credentials via Pulumi ESC, CloudWatch monitoring, and /security-review the infrastructure before deploying

# EKS cluster — stacks Kubernetes, GitOps, incident response, cost, and SRE skills
Create a Pulumi TypeScript program for an EKS cluster with /kubernetes-specialist security hardening, /gitops-workflow for ArgoCD deployment, /incident-runbook-templates for the cluster, /cost-optimization recommendations, and /sre-engineer SLO definitions for the services

The first prompt triggers the Pulumi TypeScript, monitoring, and security review skills in a single conversation. The second stacks Kubernetes, GitOps, incident response, cost, and SRE skills on one cluster build. You get infrastructure code, operational runbooks, and security policies from a single request.

What changes

Fair warning: not every skill works perfectly on the first try. Some need iteration. Some produce output that you have to review and tweak before it matches your standards. Skills do not replace your judgment.

That said, after a few weeks with these skills installed, I stopped correcting the same mistakes. The code Claude writes now looks like code I would write, not code I would have to fix. That is the whole point. Skills just stop you from repeating the same corrections across every conversation.

Get started

Every skill in this post includes its install command. Pick the section that matches your biggest pain point, run the npx skills add command, and try it on your next task. Skills work in Claude Code, Cursor, GitHub Copilot, and anything else that supports the Agent Skills standard.

The Pulumi Agent Skills announcement has more details, and the GitHub repository has the source. If you want something that goes further, with organizational context and deployment governance, look at Pulumi Neo. Neo is grounded in your actual infrastructure, not internet patterns. The 10 things you can do with Neo post shows what that looks like in practice.

Give it one project. That is all it took for me.

Try Pulumi for Free

Deploy OpenClaw on AWS or Hetzner Securely with Pulumi and Tailscale

Engin Diri — Mon, 26 Jan 2026 00:00:00 +0000

Update (January 2026): The lobster has molted into its final form! From Clawdbot to Moltbot to OpenClaw. With 100k+ GitHub stars and 2M visitors in a week, the project finally has a name that’ll stick. The CLI command is now openclaw and the new handle is @openclaw. Same mission: AI that actually does things. Your assistant. Your machine. Your rules. See the official getting started guide for updated installation instructions.

Update (April 2026): Refreshed for OpenClaw 2026.4.27. Upstream now recommends Node 24, but the cloud-init script in this post still installs Node 22 — both work. If you’d like Node 24, change the nvm install 22 lines to nvm install 24.

The short version: Deploy OpenClaw to AWS or Hetzner with a Pulumi TypeScript program that provisions the VM, installs Docker, Node, and OpenClaw, then joins the instance to your Tailscale network so the gateway and browser ports stay private. One pulumi up to deploy, one pulumi destroy to tear down. Total cost: about $33/month on AWS or $7/month on Hetzner.

OpenClaw is everywhere right now. The open-source AI assistant gained 9,000 GitHub stars in a single day, received public praise from former Tesla AI head Andrej Karpathy, and has sparked a global run on Mac Minis as developers scramble to give this “lobster assistant” a home. Users are calling it “Jarvis living in a hard drive” and “Claude with hands”—the personal AI assistant that Siri promised but never delivered.

The Mac Mini craze is real: people are buying dedicated hardware just to run OpenClaw, with some enthusiasts purchasing 40 Mac Minis at once. Even Logan Kilpatrick from Google DeepMind couldn’t resist ordering one. But here’s the thing: you don’t actually need a Mac Mini. OpenClaw runs anywhere: on a VPS, in the cloud, or on that old laptop gathering dust.

With all this hype, I had to try it myself. But instead of clicking through the AWS console or running manual commands on a VPS, I wanted to do it right from the start: infrastructure as code with Pulumi. Why? Because when I inevitably want to tear it down, spin up a new instance, or deploy to a different region, I don’t want to remember which buttons I clicked three weeks ago. I want a single pulumi up command.

Dan got the assignment right:

In this post, I’ll show you how to deploy OpenClaw to AWS or Hetzner Cloud (if you want European data residency or just want to spend less). We’ll use Pulumi to define the infrastructure and Tailscale to keep your AI assistant off the public internet.

What is OpenClaw?

OpenClaw is an open-source AI assistant created by Peter Steinberger that runs on your own infrastructure. It connects to WhatsApp, Slack, Discord, Google Chat, Signal, and iMessage. It can control browsers, generate videos and images, clone your voice for voice notes, and run scheduled tasks via cron. There’s a skills system for extending functionality, and you can run it on pretty much anything: Mac Mini, Raspberry Pi, VPS, laptop, or gaming PC.

The difference from cloud-hosted AI? OpenClaw runs on your server, not Anthropic’s. It’s available 24/7 across all your devices, can schedule automated tasks, and keeps your entire conversation history locally. Check the official OpenClaw documentation for the full feature list.

Prerequisites

Before getting started, ensure you have:

Pulumi CLI installed and configured
A Pulumi Cloud account
AWS account (for AWS deployment)
Hetzner Cloud account (for European deployment)
Anthropic API key
Node.js 22+ installed (Node 24 recommended; the cloud-init script in this post installs Node 22 inside the VM)
Tailscale account with HTTPS enabled (one-time setup in admin console)

This guide uses Anthropic’s API, but OpenClaw works with other providers too. Check the providers documentation if you’d rather use OpenAI, Google Gemini, or a local model via Ollama.

Understanding OpenClaw architecture

OpenClaw uses a gateway-centric architecture where a single daemon acts as the control plane for all messaging, tool execution, and client connections:

Component	Port	Description
Gateway	18789	WebSocket server handling channels, nodes, sessions, and hooks
Browser control	18791	Headless Chrome instance for web automation
Docker sandbox	-	Isolated container environment for running tools safely

The Gateway connects to messaging platforms (WhatsApp, Slack, Discord, etc.), the CLI, the web UI, and mobile apps. The Browser component lets OpenClaw open web pages, fill forms, scrape data, and download files. Docker sandboxing runs bash commands in isolated containers so your bot can execute code without risking your host system.

Setting up ESC for secrets management

Deploying OpenClaw means handling sensitive credentials: API keys, auth tokens, cloud provider secrets. You don’t want these hardcoded or scattered across environment variables. Pulumi ESC (Environments, Secrets, and Configuration) stores them securely and passes them directly to your Pulumi program — the same pattern Pulumi uses to eliminate long-lived CI secrets across 70+ repos.

Create a new ESC environment:

pulumi env init <your-org>/openclaw-secrets

Add your secrets to the environment:

values:
 anthropicApiKey:
 fn::secret: "sk-ant-xxxxx"
 tailscaleAuthKey:
 fn::secret: "tskey-auth-xxxxx"
 tailnetDnsName: "tailxxxxx.ts.net"
 hcloudToken:
 fn::secret: "your-hetzner-api-token"
 pulumiConfig:
 anthropicApiKey: ${anthropicApiKey}
 tailscaleAuthKey: ${tailscaleAuthKey}
 tailnetDnsName: ${tailnetDnsName}
 hcloud:token: ${hcloudToken}

Then create a Pulumi.dev.yaml file in your project to reference the environment:

environment:
 - <your-org>/openclaw-secrets

This approach keeps your secrets out of your codebase and passes them directly to OpenClaw during automated onboarding. To prevent stack-level overrides of these values, see locking down ESC values with fn::final.

Securing with Tailscale

By default, deploying OpenClaw exposes SSH (port 22), the gateway (port 18789), and browser control (port 18791) to the public internet. This is convenient for testing but not ideal for production use.

Tailscale creates a secure mesh VPN that lets you access your OpenClaw instance without exposing unnecessary ports publicly. When you provide a Tailscale auth key, the Pulumi program:

Removes gateway and browser ports from public access
Keeps SSH as fallback for debugging if Tailscale setup fails
Installs Tailscale on the instance during provisioning (after other dependencies)
Enables Tailscale SSH so you can SSH via Tailscale without managing keys
Joins your tailnet automatically using the auth key

The Pulumi program installs Docker, Node.js, and OpenClaw first, then configures Tailscale last. This ensures that even if the Tailscale auth key is invalid or expired, you can still SSH in via the public IP to troubleshoot.

To generate a Tailscale auth key:

Go to Tailscale Admin Console
Click “Generate auth key”
Enable “Reusable” if you plan to redeploy
Copy the key and add it to your ESC environment

Deploying to AWS

Let’s walk through the complete AWS deployment. Create a new Pulumi project:

mkdir openclaw-aws && cd openclaw-aws
pulumi new typescript

Install the required dependencies:

npm install @pulumi/aws @pulumi/tls

Do not use t3.micro instances for OpenClaw. The 1 GB memory is insufficient for installation. Use t3.medium (4 GB) or t3.large (8 GB) instead.

The Pulumi program

Running OpenClaw on AWS means setting up a VPC, subnets, security groups, an EC2 instance, SSH keys, and a cloud-init script that installs everything. That’s a lot of clicking in the AWS console. The Pulumi program below defines all of it in code.

Replace the contents of index.ts with the following:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as tls from "@pulumi/tls";

const config = new pulumi.Config();

const instanceType = config.get("instanceType") ?? "t3.medium";
const anthropicApiKey = config.requireSecret("anthropicApiKey");
const model = config.get("model") ?? "anthropic/claude-sonnet-4";
const enableSandbox = config.getBoolean("enableSandbox") ?? true;
const gatewayPort = config.getNumber("gatewayPort") ?? 18789;
const browserPort = config.getNumber("browserPort") ?? 18791;

const tailscaleAuthKey = config.requireSecret("tailscaleAuthKey");
const tailnetDnsName = config.require("tailnetDnsName");

// Generate a random token for gateway authentication
const gatewayToken = new tls.PrivateKey("openclaw-gateway-token", {
 algorithm: "ED25519",
}).publicKeyOpenssh.apply(key => {
 // Create a deterministic token from the public key (take first 48 hex chars)
 const hash = require("crypto").createHash("sha256").update(key).digest("hex");
 return hash.substring(0, 48);
});

const sshKey = new tls.PrivateKey("openclaw-ssh-key", {
 algorithm: "ED25519",
});

const vpc = new aws.ec2.Vpc("openclaw-vpc", {
 cidrBlock: "10.0.0.0/16",
 enableDnsHostnames: true,
 enableDnsSupport: true,
 tags: { Name: "openclaw-vpc" },
});

const gateway = new aws.ec2.InternetGateway("openclaw-igw", {
 vpcId: vpc.id,
 tags: { Name: "openclaw-igw" },
});

const subnet = new aws.ec2.Subnet("openclaw-subnet", {
 vpcId: vpc.id,
 cidrBlock: "10.0.1.0/24",
 mapPublicIpOnLaunch: true,
 tags: { Name: "openclaw-subnet" },
});

const routeTable = new aws.ec2.RouteTable("openclaw-rt", {
 vpcId: vpc.id,
 routes: [
 {
 cidrBlock: "0.0.0.0/0",
 gatewayId: gateway.id,
 },
 ],
 tags: { Name: "openclaw-rt" },
});

new aws.ec2.RouteTableAssociation("openclaw-rta", {
 subnetId: subnet.id,
 routeTableId: routeTable.id,
});

const securityGroup = new aws.ec2.SecurityGroup("openclaw-sg", {
 vpcId: vpc.id,
 description: "Security group for OpenClaw instance",
 ingress: [
 {
 description: "SSH access (fallback)",
 fromPort: 22,
 toPort: 22,
 protocol: "tcp",
 cidrBlocks: ["0.0.0.0/0"],
 },
 ],
 egress: [
 {
 fromPort: 0,
 toPort: 0,
 protocol: "-1",
 cidrBlocks: ["0.0.0.0/0"],
 },
 ],
 tags: { Name: "openclaw-sg" },
});

const keyPair = new aws.ec2.KeyPair("openclaw-keypair", {
 publicKey: sshKey.publicKeyOpenssh,
});

const ami = aws.ec2.getAmiOutput({
 owners: ["099720109477"],
 mostRecent: true,
 filters: [
 { name: "name", values: ["ubuntu/images/hvm-ssd-gp3/ubuntu-noble-24.04-amd64-server-*"] },
 { name: "virtualization-type", values: ["hvm"] },
 ],
});

const userData = pulumi
 .all([tailscaleAuthKey, anthropicApiKey, gatewayToken])
 .apply(([tsAuthKey, apiKey, gwToken]) => {
 return `#!/bin/bash
set -e

export DEBIAN_FRONTEND=noninteractive

# System updates
apt-get update
apt-get upgrade -y

# Install Docker
curl -fsSL https://get.docker.com | sh
systemctl enable docker
systemctl start docker
usermod -aG docker ubuntu

# Install NVM and Node.js for ubuntu user
sudo -u ubuntu bash << 'UBUNTU_SCRIPT'
set -e
cd ~

# Install NVM
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.1/install.sh | bash

# Load NVM
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"

# Install Node.js 22
nvm install 22
nvm use 22
nvm alias default 22

# Install OpenClaw
npm install -g openclaw@latest

# Add NVM to bashrc if not already there
if ! grep -q 'NVM_DIR' ~/.bashrc; then
 echo 'export NVM_DIR="$HOME/.nvm"' >> ~/.bashrc
 echo '[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"' >> ~/.bashrc
fi
UBUNTU_SCRIPT

# Set environment variables for ubuntu user
echo 'export ANTHROPIC_API_KEY="${apiKey}"' >> /home/ubuntu/.bashrc

# Install and configure Tailscale
echo "Installing Tailscale..."
curl -fsSL https://tailscale.com/install.sh | sh
tailscale up --authkey="${tsAuthKey}" --ssh || echo "WARNING: Tailscale setup failed. Run 'sudo tailscale up' manually."

# Enable systemd linger for ubuntu user (required for user services to run at boot)
loginctl enable-linger ubuntu

# Start user's systemd instance (required for user services during cloud-init)
systemctl start user@1000.service

# Run OpenClaw onboarding as ubuntu user (skip daemon install, do it separately)
echo "Running OpenClaw onboarding..."
sudo -H -u ubuntu ANTHROPIC_API_KEY="${apiKey}" GATEWAY_PORT="${gatewayPort}" bash -c '
export HOME=/home/ubuntu
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"

openclaw onboard --non-interactive --accept-risk \
 --mode local \
 --auth-choice apiKey \
 --gateway-port $GATEWAY_PORT \
 --gateway-bind loopback \
 --skip-daemon \
 --skip-skills || echo "WARNING: OpenClaw onboarding failed. Run openclaw onboard manually."
'

# Install daemon service with XDG_RUNTIME_DIR set
echo "Installing OpenClaw daemon..."
sudo -H -u ubuntu XDG_RUNTIME_DIR=/run/user/1000 bash -c '
export HOME=/home/ubuntu
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"

openclaw daemon install || echo "WARNING: Daemon install failed. Run openclaw daemon install manually."
'

# Configure gateway for Tailscale Serve (trustedProxies + skip device pairing + set token)
echo "Configuring gateway for Tailscale Serve..."
sudo -H -u ubuntu GATEWAY_TOKEN="${gwToken}" python3 << 'PYTHON_SCRIPT'
import json
import os
config_path = "/home/ubuntu/.openclaw/openclaw.json"
with open(config_path) as f:
 config = json.load(f)
config["gateway"]["trustedProxies"] = ["127.0.0.1"]
config["gateway"]["controlUi"] = {
 "enabled": True,
 "allowInsecureAuth": True
}
config["gateway"]["auth"] = {
 "mode": "token",
 "token": os.environ["GATEWAY_TOKEN"]
}
with open(config_path, "w") as f:
 json.dump(config, f, indent=2)
print("Configured gateway with trustedProxies, controlUi, and token")
PYTHON_SCRIPT

# Enable Tailscale HTTPS proxy (requires HTTPS to be enabled in Tailscale admin console)
echo "Enabling Tailscale HTTPS proxy..."
tailscale serve --bg ${gatewayPort} || echo "WARNING: tailscale serve failed. Enable HTTPS in your Tailscale admin console first."

echo "OpenClaw setup complete!"
`;
 });

const instance = new aws.ec2.Instance("openclaw-instance", {
 ami: ami.id,
 instanceType: instanceType,
 subnetId: subnet.id,
 vpcSecurityGroupIds: [securityGroup.id],
 keyName: keyPair.keyName,
 userData: userData,
 userDataReplaceOnChange: true,
 rootBlockDevice: {
 volumeSize: 30,
 volumeType: "gp3",
 },
 tags: { Name: "openclaw" },
});

export const publicIp = instance.publicIp;
export const publicDns = instance.publicDns;
export const privateKey = sshKey.privateKeyOpenssh;

// Construct the Tailscale MagicDNS hostname from the private IP
// AWS private IPs like 10.0.1.15 become hostnames like ip-10-0-1-15
const tailscaleHostname = instance.privateIp.apply(ip =>
 `ip-${ip.replace(/\./g, "-")}`
);

export const tailscaleUrlWithToken = pulumi.interpolate`https://${tailscaleHostname}.${tailnetDnsName}/?token=${gatewayToken}`;
export const gatewayTokenOutput = gatewayToken;

Deploying to Hetzner

Hetzner Cloud is a solid choice if you need European data residency or want to spend less money. Spoiler: it’s a lot less money.

Hetzner has similar concepts to AWS but different names. EC2 instances become Servers. Security groups become Firewalls. Same idea, different provider. The resource types come from @pulumi/hcloud.

Create a new project for Hetzner:

mkdir openclaw-hetzner && cd openclaw-hetzner
pulumi new typescript

Install the Hetzner provider:

npm install @pulumi/hcloud @pulumi/tls

The default server type cax21 is an ARM-based (Ampere) instance with 4 vCPUs and 8 GB RAM. ARM instances cost less for the same compute. If you need x86 architecture, use ccx13 or similar CCX series instead.

The Hetzner Pulumi program

Replace index.ts with the following:

import * as pulumi from "@pulumi/pulumi";
import * as hcloud from "@pulumi/hcloud";
import * as tls from "@pulumi/tls";

const config = new pulumi.Config();

const serverType = config.get("serverType") ?? "cax21";
const location = config.get("location") ?? "fsn1";
const anthropicApiKey = config.requireSecret("anthropicApiKey");
const model = config.get("model") ?? "anthropic/claude-sonnet-4";
const enableSandbox = config.getBoolean("enableSandbox") ?? true;
const gatewayPort = config.getNumber("gatewayPort") ?? 18789;
const browserPort = config.getNumber("browserPort") ?? 18791;

const tailscaleAuthKey = config.requireSecret("tailscaleAuthKey");
const tailnetDnsName = config.require("tailnetDnsName");

// Generate a random token for gateway authentication
const gatewayToken = new tls.PrivateKey("openclaw-gateway-token", {
 algorithm: "ED25519",
}).publicKeyOpenssh.apply(key => {
 const hash = require("crypto").createHash("sha256").update(key).digest("hex");
 return hash.substring(0, 48);
});

const sshKey = new tls.PrivateKey("openclaw-ssh-key", {
 algorithm: "ED25519",
});

const hcloudSshKey = new hcloud.SshKey("openclaw-sshkey", {
 publicKey: sshKey.publicKeyOpenssh,
});

const firewallRules: hcloud.types.input.FirewallRule[] = [
 {
 direction: "out",
 protocol: "tcp",
 port: "any",
 destinationIps: ["0.0.0.0/0", "::/0"],
 description: "Allow all outbound TCP",
 },
 {
 direction: "out",
 protocol: "udp",
 port: "any",
 destinationIps: ["0.0.0.0/0", "::/0"],
 description: "Allow all outbound UDP",
 },
 {
 direction: "out",
 protocol: "icmp",
 destinationIps: ["0.0.0.0/0", "::/0"],
 description: "Allow all outbound ICMP",
 },
 {
 direction: "in",
 protocol: "tcp",
 port: "22",
 sourceIps: ["0.0.0.0/0", "::/0"],
 description: "SSH access (fallback)",
 },
];

const firewall = new hcloud.Firewall("openclaw-firewall", {
 rules: firewallRules,
});

const userData = pulumi
 .all([tailscaleAuthKey, anthropicApiKey, gatewayToken])
 .apply(([tsAuthKey, apiKey, gwToken]) => {
 return `#!/bin/bash
set -e

export DEBIAN_FRONTEND=noninteractive

# System updates
apt-get update
apt-get upgrade -y

# Install Docker
curl -fsSL https://get.docker.com | sh
systemctl enable docker
systemctl start docker

# Create ubuntu user (Hetzner uses root by default)
useradd -m -s /bin/bash -G docker ubuntu || true

# Install NVM and Node.js for ubuntu user
sudo -u ubuntu bash << 'UBUNTU_SCRIPT'
set -e
cd ~

# Install NVM
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.1/install.sh | bash

# Load NVM
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"

# Install Node.js 22
nvm install 22
nvm use 22
nvm alias default 22

# Install OpenClaw
npm install -g openclaw@latest

# Add NVM to bashrc if not already there
if ! grep -q 'NVM_DIR' ~/.bashrc; then
 echo 'export NVM_DIR="$HOME/.nvm"' >> ~/.bashrc
 echo '[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"' >> ~/.bashrc
fi
UBUNTU_SCRIPT

# Set environment variables for ubuntu user
echo 'export ANTHROPIC_API_KEY="${apiKey}"' >> /home/ubuntu/.bashrc

# Install and configure Tailscale
echo "Installing Tailscale..."
curl -fsSL https://tailscale.com/install.sh | sh
tailscale up --authkey="${tsAuthKey}" --ssh || echo "WARNING: Tailscale setup failed. Run 'sudo tailscale up' manually."

# Enable systemd linger for ubuntu user (required for user services to run at boot)
loginctl enable-linger ubuntu

# Start user's systemd instance (required for user services during cloud-init)
systemctl start user@1000.service

# Run OpenClaw onboarding as ubuntu user (skip daemon install, do it separately)
echo "Running OpenClaw onboarding..."
sudo -H -u ubuntu ANTHROPIC_API_KEY="${apiKey}" GATEWAY_PORT="${gatewayPort}" bash -c '
export HOME=/home/ubuntu
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"

openclaw onboard --non-interactive --accept-risk \
 --mode local \
 --auth-choice apiKey \
 --gateway-port $GATEWAY_PORT \
 --gateway-bind loopback \
 --skip-daemon \
 --skip-skills || echo "WARNING: OpenClaw onboarding failed. Run openclaw onboard manually."
'

# Install daemon service with XDG_RUNTIME_DIR set
echo "Installing OpenClaw daemon..."
sudo -H -u ubuntu XDG_RUNTIME_DIR=/run/user/1000 bash -c '
export HOME=/home/ubuntu
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"

openclaw daemon install || echo "WARNING: Daemon install failed. Run openclaw daemon install manually."
'

# Configure gateway for Tailscale Serve (trustedProxies + skip device pairing + set token)
echo "Configuring gateway for Tailscale Serve..."
sudo -H -u ubuntu GATEWAY_TOKEN="${gwToken}" python3 << 'PYTHON_SCRIPT'
import json
import os
config_path = "/home/ubuntu/.openclaw/openclaw.json"
with open(config_path) as f:
 config = json.load(f)
config["gateway"]["trustedProxies"] = ["127.0.0.1"]
config["gateway"]["controlUi"] = {
 "enabled": True,
 "allowInsecureAuth": True
}
config["gateway"]["auth"] = {
 "mode": "token",
 "token": os.environ["GATEWAY_TOKEN"]
}
with open(config_path, "w") as f:
 json.dump(config, f, indent=2)
print("Configured gateway with trustedProxies, controlUi, and token")
PYTHON_SCRIPT

# Enable Tailscale HTTPS proxy (requires HTTPS to be enabled in Tailscale admin console)
echo "Enabling Tailscale HTTPS proxy..."
tailscale serve --bg ${gatewayPort} || echo "WARNING: tailscale serve failed. Enable HTTPS in your Tailscale admin console first."

echo "OpenClaw setup complete!"
`;
 });

const server = new hcloud.Server("openclaw-server", {
 serverType: serverType,
 location: location,
 image: "ubuntu-24.04",
 sshKeys: [hcloudSshKey.id],
 firewallIds: [firewall.id.apply(id => Number(id))],
 userData: userData,
 labels: {
 purpose: "openclaw",
 },
});

export const ipv4Address = server.ipv4Address;
export const privateKey = sshKey.privateKeyOpenssh;

// Construct the Tailscale MagicDNS hostname from the server name
// Hetzner servers use their name as the hostname
const tailscaleHostname = server.name;

export const tailscaleUrlWithToken = pulumi.interpolate`https://${tailscaleHostname}.${tailnetDnsName}/?token=${gatewayToken}`;
export const gatewayTokenOutput = gatewayToken;

You can find both programs in the Pulumi examples repo under openclaw/:

github.com/pulumi/examples

Cost comparison

Before deploying, let’s compare the costs between AWS and Hetzner for running OpenClaw 24/7:

	AWS (t3.medium)	Hetzner (cax21)
vCPUs	2	4
Memory	4 GB	8 GB
Storage	30 GB gp3 (+$2.40/mo)	80 GB NVMe (included)
Traffic	Pay per GB	20 TB included
Architecture	x86 (Intel/AMD)	ARM (Ampere)
Hourly price	$0.0416	€0.0104 (~$0.011)
Monthly price	~$33 (with storage)	€6.49 (~$7)
Annual cost	~$396	~$84

Hetzner gives you double the vCPUs, double the RAM, at less than a quarter of the price. The trade-off? ARM architecture instead of x86. But OpenClaw doesn’t care - it’s just Node.js and Docker.

Prices are for on-demand instances as of January 2026. AWS prices are for us-east-1; Hetzner prices exclude VAT. Both include standard networking and storage. Check AWS EC2 pricing and Hetzner Cloud pricing for current rates.

Running the deployment

With your ESC environment configured in Pulumi.dev.yaml, deploy with:

pulumi up

After deployment completes, you’ll see outputs similar to:

Outputs:
 gatewayTokenOutput : "786c099cc8f8bf20dbebf40b8b51b75cf5cdab25..."
 privateKey : [secret]
 publicDns : "ec2-x-x-x-x.compute-1.amazonaws.com"
 publicIp : "x.x.x.x"
 tailscaleUrlWithToken: "https://ip-10-0-1-x.tailxxxxx.ts.net/?token=786c099..."

The tailscaleUrlWithToken output provides the complete URL with authentication token. Copy and paste it into your browser to access the OpenClaw web UI.

Output names vary slightly between providers: AWS uses publicIp and publicDns, while Hetzner uses ipv4Address. The Tailscale hostname is derived from the instance’s private IP (AWS) or server name (Hetzner).

Automated onboarding

The Pulumi program runs OpenClaw’s non-interactive onboarding during instance provisioning. It uses your Anthropic API key from ESC, binds the gateway to loopback with Tailscale Serve as the HTTPS proxy, generates a secure gateway token (exported in Pulumi outputs), installs the daemon as a systemd user service, and configures trustedProxies and controlUi.allowInsecureAuth to skip device pairing when accessed via Tailscale.

The cloud-init script runs openclaw onboard --non-interactive with all necessary flags, then configures the gateway for secure Tailscale access. Your instance is ready as soon as provisioning finishes.

Access the web UI

Please wait 3-5 minutes after pulumi up completes. The cloud-init script runs after the instance launches and needs time to install Docker, Node.js, OpenClaw, and Tailscale, then run the onboarding process and start the daemon. Periodically refresh the page until it loads. If the page still doesn’t load after 5 minutes, see Verify the deployment to troubleshoot.

The easiest way to access the OpenClaw web UI is to use the tailscaleUrlWithToken output from Pulumi:

# Get the full URL with token
pulumi stack output tailscaleUrlWithToken

Copy and paste this URL into your browser. The URL includes both the Tailscale MagicDNS hostname and the authentication token, so you can access the web UI directly.

Finding your tailnet DNS name: Go to the Tailscale admin console and look under the DNS section for your tailnet name (e.g., tailxxxxx.ts.net). You can also find your machines and their MagicDNS hostnames in the Machines tab.

Token-based authentication provides an additional layer of security on top of Tailscale’s network-level authentication. Only devices on your tailnet can reach the URL, and the token prevents unauthorized access if someone gains access to your tailnet.

From the web UI, you can connect messaging channels (WhatsApp, Discord, Slack), configure skills and integrations, and manage settings.

Verify the deployment (optional)

If you encounter issues accessing the web UI, you can SSH into your instance to troubleshoot:

# Check your Tailscale admin console for the new machine
ssh ubuntu@<tailscale-ip>

# Check OpenClaw gateway status
systemctl --user status openclaw-gateway

Test your assistant

Open the gateway dashboard using the tailscaleUrlWithToken output and use the built-in chat to test your assistant:

Your personal AI assistant is now running 24/7 on your own infrastructure, accessible securely through Tailscale.

Redeploy OpenClaw with one command

With your infrastructure defined in Pulumi, you can move OpenClaw to a new region, rebuild it, or tear it down with pulumi destroy, without remembering which console buttons you clicked.

Get started

Security considerations

When self-hosting an AI assistant, security matters. OpenClaw’s rapid adoption meant thousands of instances spun up in days, and not everyone locked them down. The community noticed:

The tweet isn’t exaggerating. A quick Shodan search shows exposed gateways on port 18789 with shell access, browser automation, and API keys up for grabs:

Don’t let your instance be one of them.

Concern	Without Tailscale	With Tailscale
SSH access	Public (port 22 open)	Public fallback + Tailscale SSH
Gateway access	Public (port 18789 open)	Private (Tailscale only)
Browser control	Public (port 18791 open)	Private (Tailscale only)
API keys in transit	Exposed if gateway accessed over HTTP	Protected by Tailscale encryption
Attack surface	3 open ports	1 open port (SSH fallback)

SSH remains accessible as a fallback even with Tailscale enabled. This allows you to troubleshoot if Tailscale fails to connect. Once you’ve confirmed Tailscale is working, you can manually remove the SSH ingress rule from your security group for maximum security.

My recommendations:

Always use Tailscale for production
Rotate your auth keys periodically
Use Pulumi ESC for secrets instead of hardcoding
Enable Tailscale SSH to avoid managing keys manually
Monitor your Tailscale admin console for unauthorized devices
Remove the SSH fallback after confirming Tailscale works if you want zero public ports

What’s next?

Now that OpenClaw is running, you can install skills (voice generation, video creation, browser automation), set up scheduled tasks with cron, invite colleagues to your tailnet for shared access, or connect additional channels like WhatsApp and Discord.

For a deeper case study of running an AI assistant on Pulumi-managed infrastructure, see how we built Platybot, or read our KubeCon EU 2026 recap on the year AI moved into production on Kubernetes.

Conclusion

Deploying OpenClaw with infrastructure as code means you can reproduce your setup anytime, version control it, and tear it down with a single command. Adding Tailscale keeps it private - no exposed ports, no hoping you configured your firewall correctly at 2am.

If you run into issues or have questions, drop by the Pulumi Community Slack or GitHub Discussions.

New to Pulumi? Get started here.

Self-Verifying AI Agents: Vercel's Agent-Browser in the Ralph Wiggum Loop

Engin Diri — Tue, 20 Jan 2026 00:00:00 +0000

In my previous post about the Ralph Wiggum technique, Claude Code built a complete serverless URL shortener on AWS. The setup included Playwright MCP for end-to-end testing. It worked. But I kept wondering if there was something better for AI-driven browser automation. Then Vercel released agent-browser, and I had to try it.

Why browser automation matters for AI coding

When an AI coding agent builds a frontend, someone has to verify it works. Without browser automation, that someone is you. The AI finishes and says “done,” but you can’t trust that claim until you open a browser and click around yourself.

Browser automation changes this. The AI verifies its own work. It builds a component, launches a browser, tests the interaction, confirms behavior matches expectations. If something breaks, it fixes the code and tests again. The validation loop runs without you.

This matters more as AI agents take on larger tasks. A quick component fix might not need automated testing. But when an agent builds an entire dashboard from scratch, you need proof that the deployed application actually works.

The context problem with browser automation

Playwright MCP gives you powerful browser control, but it comes with overhead. Every screenshot, every DOM snapshot, every accessibility tree adds tokens to your context window. GitHub issue #889 documents a 6x token increase between versions 0.0.30 and 0.0.32. Users report single screenshots consuming over 15,000 tokens. Some exhausted their entire five-hour token allocation in just a few automation steps.

The problem is verbose output. Full accessibility trees contain every element on the page with all their properties. Console message logging adds more. Each piece of information might be useful for debugging, but together they overwhelm the context window.

This creates a frustrating tradeoff. You want rich browser interaction for thorough testing, but that richness costs tokens. Fewer tokens means fewer iterations before hitting limits.

Vercel’s “less is more” philosophy

Vercel ran into something similar with their D0 text-to-SQL agent. Their research documents what happened when they reduced tool complexity.

The original architecture used 17 specialized tools. Each tool handled one specific operation: create tables, insert rows, run queries, validate schemas. The approach seemed logical. Specialized tools should produce better results than general ones.

The numbers told a different story. With 17 tools, they got 80% success (4 out of 5 queries), averaging 274.8 seconds and ~102,000 tokens. The worst case took 724 seconds, burned 145,463 tokens, and still failed.

Then they rebuilt with just two tools: ExecuteCommand and ExecuteSQL. Success jumped to 100%. Average execution dropped to 77.4 seconds (3.5x faster). Token usage fell to ~61,000 (37% reduction). The worst case took 141 seconds, used 67,483 tokens, and succeeded.

Vercel’s takeaway: “We were constraining reasoning because we didn’t trust the model to reason.” Fewer tools meant the model could think more freely about how to accomplish tasks. The simplicity reduced confusion and context waste.

Applying “less is more” to browser automation

agent-browser takes the same approach. Instead of separate tools for clicking, typing, scrolling, and navigating, it has a unified CLI with one clever idea: the snapshot + refs system.

When you request a page snapshot, agent-browser returns something like this:

- button "Sign In" [ref=e1]
- textbox "Email" [ref=e2]
- textbox "Password" [ref=e3]
- link "Documentation" [ref=e4]

Those @e1, @e2 references are stable element identifiers. To click the sign-in button, you run click @e1. No CSS selectors. No XPath expressions. No waiting for the DOM to stabilize. The reference points to that exact element.

This cuts out the verbosity of full accessibility trees. You get just enough information to understand the page structure and interact with elements. The AI can reason about what it sees without drowning in metadata.

Installing agent-browser

Setup is simple:

npm install -g agent-browser && agent-browser install

This installs the CLI and downloads a bundled browser. You don’t need to install any additional browsers.

For Claude Code integration, install the official skill from the agent-browser package. You can either copy it from node_modules:

cp -r node_modules/agent-browser/skills/agent-browser .claude/skills/

Or download it directly:

mkdir -p .claude/skills/agent-browser
curl -o .claude/skills/agent-browser/SKILL.md \
 https://raw.githubusercontent.com/vercel-labs/agent-browser/main/skills/agent-browser/SKILL.md

The skill gives Claude Code better context than just running agent-browser --help.

Unlike Playwright MCP, there’s no server configuration. Agent-browser runs as a standalone CLI that Claude Code invokes directly through bash.

The experiment: adding analytics

I wanted a realistic test scenario, so I added an analytics dashboard to the url-shortener-saas project from my previous post. The feature tracks click events with metadata (browser, device, country, referrer) and displays them through interactive charts.

This follows the same “Ralph Wiggum” workflow from my previous post: write a feature-prompt.md that describes what you want, then let Claude Code implement, deploy with Pulumi, and verify with browser automation. Here’s a snippet from the feature prompt:

## Backend Requirements

### New DynamoDB Table: Analytics Events

Create a new DynamoDB table `url-shortener-analytics-{stack}` with:
- Partition Key: shortCode (String)
- Sort Key: timestamp (String, ISO 8601 format)
- Attributes: browser, deviceType, country, referrer, etc.

### New Lambda: analytics.ts

GET /api/analytics/{shortCode}
- Returns timeline, browsers, devices, countries, referrers
- Query params: from, to, granularity

## Success Criteria

1. `pulumi up` deploys successfully without errors
2. All existing E2E tests still pass
3. New analytics E2E tests pass
4. Charts render with real data after generating test clicks

Claude Code reads this, writes the Lambda handlers, updates the Pulumi infrastructure code, and deploys. The infrastructure changes are straightforward:

// Analytics events table
const analyticsTable = new aws.dynamodb.Table("analytics-table", {
 name: `${projectName}-analytics-${stack}`,
 billingMode: "PAY_PER_REQUEST",
 hashKey: "shortCode",
 rangeKey: "timestamp",
 attributes: [
 { name: "shortCode", type: "S" },
 { name: "timestamp", type: "S" },
 ],
 globalSecondaryIndexes: [{
 name: "by-date",
 hashKey: "shortCode",
 rangeKey: "timestamp",
 projectionType: "ALL",
 }],
});

After pulumi up succeeds, the agent needs to verify the deployed feature works. That’s where browser automation comes in. The feature prompt specifies verification requirements:

## End-to-End Verification

After deployment, use the `/agent-browser` skill to verify:
1. Analytics link appears in navigation
2. Analytics page loads without errors
3. Charts render (timeline, browser, device, geographic)
4. URL-specific analytics work from dashboard
5. Click data appears after generating test clicks

The test suite covered six scenarios: homepage load, URL shortening, dashboard view, analytics navigation, analytics overview, and date filter functionality. Nothing fancy, just the basics you’d want to verify after deploying.

I ran the exact same tests with both Playwright MCP and agent-browser to see how much context each consumed.

Agent-browser workflow in practice

Here’s what the CLI interaction looked like during my test run.

Navigate to the deployed application:

$ agent-browser open https://d1232drths1aav.cloudfront.net
✓ Snip.ly - Modern URL Shortener
 https://d1232drths1aav.cloudfront.net/

Snapshot to see available elements (the -i flag filters to interactive elements only):

$ agent-browser snapshot -i
- link "S Snip.ly" [ref=e1]
- link "Home" [ref=e2]
- link "Dashboard" [ref=e3]
- link "Analytics" [ref=e4]
- link "Docs" [ref=e5]
- button "Switch to dark mode" [ref=e6]
- link "Get Started" [ref=e7]
- textbox "Paste your long URL here..." [ref=e8]
- button "Shorten URL" [ref=e9]

That entire homepage snapshot is 280 characters. Playwright MCP returned 8,247 characters for the same page.

Fill the URL input and click the shorten button:

$ agent-browser fill @e8 "https://example.com/agent-browser-e2e-test"
✓ Done

$ agent-browser click @e9
✓ Done

Each action confirmation is 6 characters. Playwright MCP returns the full page state after every click - 12,891 characters when I clicked the shorten button.

Navigate to analytics:

$ agent-browser click @e4
✓ Done

$ agent-browser wait --load networkidle && agent-browser snapshot -i
✓ Done
- link "S Snip.ly" [ref=e1]
- link "Home" [ref=e2]
- link "Dashboard" [ref=e3]
- link "Analytics" [ref=e4]
- button "Last 7 days" [ref=e8]
- button "Last 30 days" [ref=e9]
- button "Last 90 days" [ref=e10]
- link "1 analytics-e2e-test https://example.com/analytics-test-verification 5" [ref=e11]
- link "2 test-url https://www.example.com/this-is-a-very-long-url... 4" [ref=e12]

The analytics snapshot shows date filter buttons and top URLs with click counts. 385 characters versus Playwright MCP’s 4,127.

Test the date filter:

$ agent-browser click @e9
✓ Done

$ agent-browser screenshot analytics-30-day-filter.png
✓ Screenshot saved to analytics-30-day-filter.png

The ref-based workflow is deterministic. Each command operates on a specific element from the snapshot. No guessing about selectors.

The numbers: 82.5% context reduction

Same six tests, both tools:

Metric	Playwright MCP	Agent-Browser	Reduction
Total response characters	31,117	5,455	82.5%
Largest single response	12,891	2,847	77.9%
Average response size	3,112	328	89.5%
Homepage snapshot	8,247	280	96.6%
Dashboard snapshot	12,891	2,847	77.9%

The difference is what each tool returns after every action.

Playwright MCP returns the full accessibility tree after every click:

### Page state
- Page URL: https://d1232drths1aav.cloudfront.net/dashboard
- Page Title: Snip.ly - Modern URL Shortener
- Page Snapshot:
- generic [ref=e2]:
 - banner [ref=e3]:
 - generic [ref=e4]:
 - link "S Snip.ly" [ref=e5] [cursor=pointer]:
 - /url: /
 - generic [ref=e6]: S
 - generic [ref=e7]: Snip.ly
 - navigation [ref=e8]:
 - link "Home" [ref=e9] [cursor=pointer]:
 - /url: /
 - text: Home
 ...
[12,891 characters continues...]

Agent-browser returns:

✓ Done

6 characters versus 12,891 for the same button click.

Six tests consumed ~31K characters with Playwright MCP versus ~5.5K with agent-browser. At roughly 4 characters per token, that’s ~7,800 tokens versus ~1,400. An AI agent could run 5.7x more tests in the same context budget.

What I noticed

Past the numbers, working with agent-browser felt different in ways that are hard to quantify.

The compact snapshots changed the rhythm of iteration. A typical page fit in a few hundred tokens instead of thousands, so Claude Code could run more test cycles before hitting limits. I spent less time worrying about context and more time actually testing.

Element refs removed a frustration I’d had with Playwright. CSS selectors break when someone changes a class name or restructures the DOM. With agent-browser, a button is just @e9. It doesn’t care about styling. That predictability was a relief.

What worked well:

Snapshots stay small enough that you don’t think about them
Refs mean no selector debugging
The CLI just works with Claude Code’s bash tool
No MCP server to configure

Agent-browser is early. Documentation is thin, and I read the source more than once to figure out edge cases.

Where I hit friction:

Modals that appear after API calls needed manual wait logic
Playwright’s waiting mechanisms are more mature
Hidden or dynamically loaded elements sometimes didn’t show up without explicit waits

For my URL shortener tests, agent-browser used fewer tokens per cycle, and the ref-based approach was more predictable than selector-driven automation.

Playwright MCP still wins on depth. Network interception, multi-tab handling, PDF generation, better waiting logic—if you need those, you need Playwright. For complex browser automation, it’s the more capable tool.

Deploy what your agent builds

Pulumi provisions the infrastructure your coding agent builds and tests, so a single pulumi up takes each verified change to the cloud.

Get started

When to use each

Agent-browser fits long autonomous sessions where context budget matters, basic navigation and verification tasks, and setups where you want to skip MCP configuration. CLI-based skills also avoid the schema overhead—tool definitions for multi-tool MCPs eat tokens even when idle.

Playwright MCP fits when you need the advanced stuff: network interception, PDF generation, multi-tab workflows, or sophisticated synchronization. It’s also the right choice if you have existing Playwright test suites. If the full MCP schema feels heavy, Playwright skills offer a lighter wrapper.

Start with agent-browser for AI validation loops. Move to Playwright when you outgrow it.

How Ralph Wiggum Built a Serverless SaaS with Pulumi

Engin Diri — Wed, 14 Jan 2026 00:00:00 +0000

I was about to do something that felt either genius or completely reckless: hand over my AWS credentials to an AI and step away from my computer. The technique is called “Ralph Wiggum,” named after the Simpsons character who eats glue and says “I’m in danger” while everything burns around him. And honestly, that felt about right for what I was attempting.

The problem with AI coding assistants

If you have spent any time with AI coding assistants, you know the frustration. You are in the middle of a task. It is going great. Claude is writing beautiful infrastructure code, understanding your architecture, making progress… and then it says “I have completed your request.”

You stare at the screen. “No. No, you have not. You maybe did a third of what I asked.”

So you reprompt it. It does a little more. Then it stops again. You are babysitting. The whole point of using an AI assistant was to save time, but if you have to check on it every five minutes, you are not saving anything. You are just a very expensive supervisor.

This is the problem Geoffrey Huntley decided to solve.

What if we just do not let it stop?

Geoffrey Huntley, a developer based in rural Australia (who, according to internet lore, lives on a property with goats), had a deceptively simple idea: what if every time Claude Code finishes and tries to exit, we just feed it the prompt again?

He named the technique “Ralph Wiggum” because the character embodies a kind of childlike persistence. Ralph repeatedly fails, makes silly mistakes, yet stubbornly continues in an endless loop until he eventually succeeds. Sound familiar? That is exactly how AI coding agents work. They make mistakes. They try again. They iterate.

The philosophy behind Ralph is beautifully stated: “Better to fail predictably than succeed unpredictably.” Every failure is just data for the next iteration.

How the Ralph loop works

At its core, the Ralph Wiggum technique is almost embarrassingly simple. It is a bash loop:

while true; do
 cat PROMPT.md | claude --print --dangerously-skip-permissions
done

That is it. That is the whole thing. You write your instructions in a PROMPT.md file, and the loop keeps feeding it to Claude Code over and over.

But here is why it works: each time Claude starts up, it looks at the project. It sees the files that exist, the code that is already written, the git history. It is not starting from zero. It picks up where it left off, sees what still needs to be done, and keeps going.

The official Claude Code plugin formalizes this with some important additions:

Completion promises: You tell Claude to output a specific string like <promise>COMPLETE</promise> when it has genuinely finished
Max iterations: A safety limit so your API bill does not reach infinity
Stop hooks: The mechanism that intercepts exit attempts and re-injects the prompt

To install it:

/plugin install ralph-wiggum@ghuntley

For end-to-end testing of deployed infrastructure, you will also want the Playwright MCP server. This lets Claude interact with your deployed application in a real browser:

claude mcp add playwright npx @playwright/mcp@latest

Why Pulumi is perfect for Ralph

Infrastructure as code has something most application code does not: objective success criteria. Your Lambda either deploys or it does not. Your DynamoDB table either exists or it does not. Tests either pass or they fail.

This makes Pulumi an ideal candidate for autonomous AI development:

Clear completion signals: pulumi preview and pulumi up provide unambiguous feedback
Iterative feedback: Failed deployments tell Claude exactly what went wrong
Testable outputs: You can write integration tests that verify your infrastructure actually works
Registry access: With the Pulumi MCP server, Claude has real-time access to documentation and examples

The experiment

I decided to build a serverless URL shortener on AWS. Not because the world needs another URL shortener, but because it is the perfect test case: multiple AWS services, clear requirements, and objective success criteria. But I did not want just a basic demo. I wanted a full SaaS-like experience with a polished frontend.

Here is the PROMPT.md file I wrote:

# Build a Production-Ready URL Shortener SaaS on AWS

Using Pulumi TypeScript, create a complete URL shortener SaaS with a polished frontend experience.

## Infrastructure Requirements
- DynamoDB table for storing URL mappings (shortCode -> originalUrl, clickCount, createdAt)
- Lambda functions for: creating short URLs, redirecting to original URLs, getting stats
- API Gateway REST API exposing the Lambda functions
- S3 bucket for the React frontend with static website hosting
- CloudFront distribution with HTTPS for the frontend and API
- Use the Pulumi ESC environment pulumi-idp/auth for the AWS credentials

## Frontend Requirements (Use /frontend-design skill)
Create a polished, production-ready SaaS website with:

### Landing Page
- Hero section with catchy headline and URL input form
- Feature highlights (fast redirects, analytics, custom aliases)
- Pricing section (Free tier, Pro tier, Enterprise tier)
- Testimonials section with 3 fake but realistic customer reviews
- Footer with links to Docs, Privacy, Terms

### Dashboard Page
- URL shortening form with optional custom alias
- List of created short URLs with click counts
- Copy-to-clipboard functionality
- Delete URL option

### Documentation Page
- Getting started guide
- API reference (POST /shorten, GET /{code}, GET /stats/{code})
- Rate limits and usage policies
- FAQ section

### Design Requirements
- Modern, clean aesthetic (no generic AI look)
- Responsive design (mobile, tablet, desktop)
- Dark mode support
- Smooth animations and transitions
- Consistent color scheme and typography

## API Requirements
- POST /shorten: accepts { url: string, alias?: string }, returns { shortCode, shortUrl }
- GET /{shortCode}: redirects to original URL (301)
- GET /stats/{shortCode}: returns { originalUrl, clickCount, createdAt }

## Success Criteria
- All unit tests pass
- All integration tests pass
- pulumi preview shows no errors
- pulumi up deploys successfully

## End-to-End Verification (Required)
After deployment, use Playwright MCP to verify the live site:
1. Open the CloudFront URL and verify the landing page loads with all sections
1. Navigate to the docs page and verify content renders
1. Create a short URL using the dashboard
1. Verify the short URL redirects correctly (301 status)
1. Check the stats show the click was recorded
1. Test responsive design by resizing the viewport
1. Take screenshots of landing page, dashboard, and docs as proof

Only output <promise>COMPLETE</promise> after ALL of the above including E2E tests pass.

The key addition here is the end-to-end verification section. Without it, Claude reports success after pulumi up finishes, but you have no guarantee the deployed infrastructure actually works. By requiring Playwright to test the live URLs, you catch issues like the CloudFront 403 error I mentioned earlier.

Before running the loop, I started Claude Code with permission bypass mode to prevent it from stopping to ask for approval on every file write or command:

claude --permission-mode bypassPermissions

There are other ways to handle permissions, but this is the simplest for autonomous execution. Then I ran the Ralph loop:

/ralph-wiggum:ralph-loop PROMPT.md --max-iterations 25 --completion-promise "COMPLETE"

Then I stepped away and let it run unsupervised. For complex projects, you could even let it run overnight while you sleep.

Here is Claude Code in action during the Ralph loop, fixing CloudFront configuration issues and writing unit tests:

The results

The complete source code for this project is available on GitHub: url-shortener-saas

When Claude finished, it had:

Successfully built:

A complete Pulumi program with proper resource organization
DynamoDB table with a GSI for querying by creation date
Three Lambda functions with proper error handling
API Gateway with CORS configuration
S3 bucket with static website hosting
CloudFront distribution with proper cache behaviors

The frontend was genuinely impressive:

A polished landing page with hero section, feature cards, and pricing tiers
Working testimonials section with three fictional but believable customer reviews
A functional dashboard where you could create and manage short URLs
A documentation page with API reference and getting started guide
Dark mode toggle that actually worked
Responsive design that looked good on mobile

Home screen

Dashboard

Documentation

Here is a snippet of what it generated:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

const stack = pulumi.getStack();
const projectName = "url-shortener";

// DynamoDB Table
const urlTable = new aws.dynamodb.Table("url-table", {
 name: `${projectName}-urls-${stack}`,
 billingMode: "PAY_PER_REQUEST",
 hashKey: "shortCode",
 attributes: [{ name: "shortCode", type: "S" }],
 tags: {
 Environment: stack,
 Project: projectName,
 ManagedBy: "Pulumi",
 },
});

// Lambda function for URL shortening
const shortenFunction = new aws.lambda.Function("shorten-function", {
 name: `${projectName}-shorten-${stack}`,
 runtime: aws.lambda.Runtime.NodeJS20dX,
 handler: "shorten.handler",
 role: lambdaRole.arn,
 timeout: 30,
 memorySize: 256,
 code: new pulumi.asset.AssetArchive({
 ".": new pulumi.asset.FileArchive("./lambda"),
 }),
 environment: {
 variables: {
 TABLE_NAME: urlTable.name,
 },
 },
});

// CloudFront Distribution with S3 and API Gateway origins
const distribution = new aws.cloudfront.Distribution("frontend-distribution", {
 enabled: true,
 defaultRootObject: "index.html",
 origins: [
 {
 domainName: frontendBucketWebsite.websiteEndpoint,
 originId: "S3Origin",
 customOriginConfig: {
 httpPort: 80,
 httpsPort: 443,
 originProtocolPolicy: "http-only",
 originSslProtocols: ["TLSv1.2"],
 },
 },
 {
 domainName: pulumi.interpolate`${api.id}.execute-api.${aws.config.region}.amazonaws.com`,
 originId: "APIGatewayOrigin",
 customOriginConfig: {
 httpPort: 80,
 httpsPort: 443,
 originProtocolPolicy: "https-only",
 originSslProtocols: ["TLSv1.2"],
 },
 },
 ],
 defaultCacheBehavior: {
 targetOriginId: "S3Origin",
 viewerProtocolPolicy: "redirect-to-https",
 allowedMethods: ["GET", "HEAD", "OPTIONS"],
 cachedMethods: ["GET", "HEAD"],
 forwardedValues: {
 queryString: false,
 cookies: { forward: "none" },
 },
 },
 orderedCacheBehaviors: [{
 pathPattern: "/api/*",
 targetOriginId: "APIGatewayOrigin",
 viewerProtocolPolicy: "https-only",
 allowedMethods: ["DELETE", "GET", "HEAD", "OPTIONS", "PATCH", "POST", "PUT"],
 cachedMethods: ["GET", "HEAD"],
 forwardedValues: {
 queryString: true,
 headers: ["Authorization", "Content-Type"],
 cookies: { forward: "none" },
 },
 minTtl: 0,
 defaultTtl: 0,
 maxTtl: 0,
 }],
 viewerCertificate: {
 cloudfrontDefaultCertificate: true,
 },
});

Where it struggled:

The first few iterations had the Lambda code inline instead of in separate files
It initially forgot to set up IAM permissions for the Lambda to access DynamoDB
The CloudFront configuration took three attempts to get the cache behaviors right

The funny parts:

At iteration 12, it decided to completely refactor the project structure. Then at iteration 13, it refactored it back
One commit message simply read: “fix the fix that fixed the previous fix”
It wrote a test, ran the test, fixed the code to pass the test, then realized the test was wrong

Pair your coding agent with Pulumi

You can point Claude Code, Cursor, Codex, or any coding agent at Pulumi through the Pulumi MCP server and Agent Skills, and let infrastructure as code give it objective success criteria. Start a project to try it.

Get started

Best practices for Ralph with Pulumi

After running several experiments, here is what I have learned:

Write extremely specific prompts. Vague instructions lead to vague implementations. Be explicit about every requirement, every edge case, every success criterion.

Use pulumi preview as your feedback loop. Add this to your success criteria:

Success Criteria:
- pulumi preview completes with no errors
- pulumi preview shows expected resource count (approximately X resources)

Set realistic iteration limits. For infrastructure projects, I have found 20-30 iterations is usually sufficient. More than that and you are probably missing something in your prompt.

Include test requirements. Claude is surprisingly good at test-driven development when you tell it to write tests first:

## Development Approach
1. Write unit tests for each Lambda function first
1. Implement Lambda functions to pass the tests
1. Write integration tests that verify the deployed infrastructure
1. Only output COMPLETE when all tests pass

Mention cost awareness. Add a note like “Use PAY_PER_REQUEST billing for DynamoDB” or “Use the smallest Lambda memory allocation that works.” Claude will optimize for what you tell it to optimize for.

The honest assessment

Is this the future of infrastructure development? Probably not entirely. But it is a genuinely useful technique for specific scenarios:

Great for:

Greenfield projects where you have clear requirements
Large migrations or refactors
Building proof-of-concept infrastructure while you focus on other work
Tasks where iteration is more valuable than perfection on the first try

Not great for:

Production infrastructure changes (please do not let an AI modify your production AWS account unsupervised)
Complex architecture decisions that require human judgment
Anything involving sensitive data or security configurations
Small tweaks or quick fixes (overkill)

The API costs were reasonable for what I got. For a complete infrastructure project built autonomously, a few dollars in Claude API calls is a bargain compared to my hourly rate.

Try it yourself

If you want to experiment with Ralph Wiggum and Pulumi:

Install Claude Code and the Ralph Wiggum plugin
Set up your Pulumi account and AWS credentials
Write a detailed PROMPT.md with clear success criteria
Start with a small project and low iteration limits
Review the git history when it finishes

The combination of autonomous AI loops and infrastructure as code feels like a glimpse of where development is heading. Not replacing developers, but changing how we use our time. Instead of babysitting an AI through each step, you hand it a well-defined problem and come back to working infrastructure.

Just make sure you have billing alerts set up. Ralph Wiggum does not know when to stop spending your money.

Or try Pulumi Neo: Ralph built-in

If setting up the Ralph Wiggum plugin feels like too much ceremony, Pulumi has something similar built right into Pulumi Neo.

Neo Tasks work on the same principle: you describe what you want, and Neo plans and executes the infrastructure changes. The key is auto mode. When you set a task to auto mode, Neo runs without requesting approvals. It plans, executes, validates with pulumi preview, and iterates until the task is complete.

The experience is remarkably similar to Ralph:

Persistent execution: Tasks continue running even if you close your browser. Come back later and Neo shows you what it accomplished while you were away.
Iterative refinement: Failed deployments inform the next attempt, just like Ralph re-reading the project state.
Clear completion: Neo knows when the infrastructure matches your requirements.

The main difference is that Neo runs in Pulumi Cloud rather than on your local machine, and it is purpose-built for infrastructure tasks. You get the autonomous loop experience without needing to configure plugins or bash loops.

For teams already using Pulumi Cloud, Neo with auto mode might be the fastest path to autonomous infrastructure development.

Resources

How to Move to the Gateway API: post ingress-nginx Retirement

Engin Diri — Fri, 02 Jan 2026 11:35:51 +0100

The upcoming retirement of ingress-nginx in early 2026 gives infrastructure teams both a deadline and an opportunity to rethink traffic management. Configuring the Ingress API often meant relying on controller-specific annotations that varied between implementations. The Gateway API offers a cleaner, standardized alternative. This post investigates the practical reality of this migration and explores why kgateway emerges as a robust solution for the future.

With ingress-nginx entering its sunset phase in early 2026, the Kubernetes community faces a decision point. While the controller served as the default standard for a decade, its architecture now struggles to meet modern requirements. The transition to the Gateway API offers a chance to adopt a standard designed for contemporary traffic patterns.

The Gateway API addresses the portability issues of its predecessor by establishing a standardized, expressive approach that behaves consistently across implementations. A technical evaluation of the available options points to kgateway as a particularly strong candidate for production workloads.

Why ingress-nginx is retiring

The Kubernetes SIG Network and Security Response Committee was direct in its announcement regarding the project’s future. After March 2026, ingress-nginx will cease to receive releases, bug fixes, or security patches. The repositories will become read-only, leaving existing deployments functional but unmaintained.

This decision stems from both resource constraints and technical debt. The project relied on a very small group of maintainers working primarily in their spare time. Furthermore, features that once provided flexibility, such as “snippets” for arbitrary NGINX configuration injection, now pose significant security liabilities. With no path to modernize the codebase, retirement was inevitable.

It is worth noting that the Ingress API itself remains supported. The retirement affects only the ingress-nginx controller. NGINX as a web server continues unchanged. However, clusters relying on this specific controller must prepare for a transition.

To identify affected resources, checking the cluster for specific pods can reveal the scope of the dependency:

kubectl get pods --all-namespaces --selector app.kubernetes.io/name=ingress-nginx

Understanding the Gateway API

The Gateway API represents a fundamental shift in traffic management concepts. Some elements will look familiar to Ingress users, but the underlying philosophy addresses the “baggage” that the previous standard accumulated.

The core improvement lies in expressiveness. Advanced routing requirements (header matching, weighted traffic splitting, and traffic policies) are now native parts of the specification. This eliminates the need for the non-portable annotations that plagued the Ingress ecosystem.

The API also introduces a role-oriented structure that mirrors actual organizational workflows.

GatewayClass resources are managed by infrastructure teams to define the underlying controller.
Gateway resources are created by platform teams to specify entry points and listeners.
HTTPRoute resources are owned by application teams to define service traffic rules.

This separation allows for genuine self-service. Application teams can manage their routing logic without requiring broad cluster privileges.

The relationship between these resources defines the traffic flow:

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
 name: kgateway
spec:
 controllerName: kgateway.dev/kgateway

A Gateway references this class to establish the entry point:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
 name: my-gateway
 namespace: default
spec:
 gatewayClassName: kgateway
 listeners:
 - name: http
 port: 8080
 protocol: HTTP
 allowedRoutes:
 namespaces:
 from: All

An HTTPRoute then attaches to the gateway:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: my-route
 namespace: default
spec:
 parentRefs:
 - name: my-gateway
 hostnames:
 - "example.com"
 rules:
 - matches:
 - path:
 type: PathPrefix
 value: /api
 backendRefs:
 - name: api-service
 port: 8080

Beyond standard HTTP, the API supports GRPCRoute, TCPRoute, UDPRoute, and TLSRoute, offering a protocol diversity that the original Ingress spec lacked.

Cross-namespace routing and ReferenceGrant

The Gateway API enables a security model where routes can cross namespace boundaries, but only with explicit permission. The ReferenceGrant resource controls these connections. Without a grant, an HTTPRoute in one namespace cannot reference a Service in another.

apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
 name: allow-routes-from-apps
 namespace: gateway-system
spec:
 from:
 - group: gateway.networking.k8s.io
 kind: HTTPRoute
 namespace: my-app
 to:
 - group: ""
 kind: Service

This mechanism allows platform teams to strictly control which namespaces can attach to shared gateways.

Evaluating Gateway API implementations

Several implementations have emerged to support the new standard, each with a distinct philosophy.

NGINX Gateway Fabric leverages NGINX as the data plane, offering continuity for teams in the F5 ecosystem. It provides impressive throughput and integrates with enterprise security tools, making it a logical choice for existing NGINX shops.

Traefik focuses on simplicity. As a single binary with no external dependencies, it aligns well with environments that prioritize developer experience and rapid iteration. Its declarative configuration is particularly friendly to GitOps workflows.

Envoy Gateway is built on the Envoy Proxy and operates under a community-driven governance model. It appeals to those seeking strong conformance to the Gateway API spec without vendor lock-in. Its companion project, Envoy AI Gateway, adds support for LLM routing.

kgateway, formerly Gloo Gateway, brings seven years of production history to the table. Donated to the CNCF in early 2025, it combines an Envoy data plane with a control plane optimized for scale. Its internal architecture uses the “krt” framework to handle massive route tables efficiently, avoiding the performance bottlenecks often seen in snapshot-based systems.

Aspect	NGINX Gateway Fabric	Traefik	Envoy Gateway	kgateway
Base Technology	NGINX (C)	Custom Go	Envoy Proxy (C++)	Envoy Proxy (C++)
Maturity	Production-ready	Mature	GA (v1.0+)	Battle-tested (7+ years as Gloo)
Setup Complexity	Medium	Low	Low	Low
Commercial Support	F5 Enterprise	Traefik Labs	Multiple vendors	Solo.io
AI/LLM Support	No	No	Via Envoy AI Gateway	Native (Agentgateway)
Learning Curve	Familiar for NGINX users	Gentle	Moderate	Moderate
Community Size	Growing	Large	Large	Established
Best For	Teams with NGINX expertise, high-throughput requirements	Teams prioritizing simplicity and rapid deployment	Standards-focused teams, multi-vendor environments	Enterprise scale, AI workloads, Istio integration

Decision framework

NGINX Gateway Fabric suits teams needing F5 support or deep NGINX tuning.
Traefik fits best where simplicity and operational ease are paramount.
Envoy Gateway works for those valuing community governance and strict standards adherence.
kgateway is the choice for enterprise scale, native AI gateway needs, and Istio integration.

This exploration uses kgateway for its proven track record and native handling of both traditional microservices and AI traffic.

Migrating from Ingress to Gateway API

The official migration guide outlines the transition process. It’s less translation, more restructuring.

The process typically involves:

Defining a Gateway resource: Unlike Ingress, listeners must be explicitly defined. This provides granular control over ports and protocols.
Creating HTTPRoute resources: These replace Ingress routing rules. You can split complex Ingress resources into multiple HTTPRoutes for better management.
Configuring filters: Annotations for headers or redirects are replaced by standardized filters within the HTTPRoute spec.

The ingress2gateway tool can automate much of the initial conversion, providing a baseline of resources to review and refine.

Migration planning

Scope the migration effort first:

1. Assessment Identify the volume and complexity of existing resources.

# Count your Ingress resources across all namespaces
kubectl get ingress -A | wc -l

# List all Ingress resources with their hosts
kubectl get ingress -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}: {.spec.rules[*].host}{"\n"}{end}'

2. Parallel Operations Running kgateway alongside ingress-nginx allows for incremental migration. Services can be moved one by one, validating the new configuration without risking the entire cluster’s traffic.

3. Rollback Strategy Keeping the original Ingress manifests ensures that traffic can be quickly reverted to the old controller if issues arise during the transition.

Why kgateway stands out

kgateway distinguishes itself through its maturity and architectural decisions. Having started as Gloo Gateway in 2018, it has hardened through years of production use.

Its control plane uses the krt framework, originally developed for Istio. This allows it to track dependencies precisely and recalculate only the parts of the configuration that have changed. In large clusters where pods churn constantly, this incremental approach prevents the control plane from becoming a bottleneck, enabling it to handle over 10,000 routes efficiently.

For teams managing AI workloads, kgateway includes the Agentgateway component. This Rust-based data plane is built for LLM traffic, supporting the Model Context Protocol (MCP) and providing token-based rate limiting. It unifies the management of AI and traditional traffic under a single control plane.

Additional features include:

Route Delegation: Allows large routing tables to be split across teams with clear inheritance.
Security Integration: Seamless mTLS with Istio ambient mesh and external authorization support.
Traffic Management: Advanced capabilities like traffic mirroring and session affinity are available without complex Envoy configuration.

Putting it all together with Pulumi

Seeing these concepts in code clarifies the architecture. The following Pulumi program demonstrates a complete setup: a DigitalOcean Kubernetes cluster, the kgateway installation via Helm, Gateway API resources, and an httpbin application to validate the configuration.

The program provisions infrastructure in a specific sequence. The cluster comes first, followed by the Gateway API CRDs, then kgateway itself. Once the control plane is running, the program creates a GatewayClass, a Gateway listener, and an HTTPRoute that directs traffic to the sample application. A ReferenceGrant permits the cross-namespace reference between the HTTPRoute and the backend Service.

import * as pulumi from "@pulumi/pulumi";
import * as digitalocean from "@pulumi/digitalocean";
import * as k8s from "@pulumi/kubernetes";

// Configuration
const config = new pulumi.Config();
const clusterName = config.get("clusterName") || "kgateway-demo-cluster";
const region = config.get("region") || "fra1";
const nodeSize = config.get("nodeSize") || "s-2vcpu-4gb";
const nodeCount = config.getNumber("nodeCount") || 2;
const k8sVersion = config.get("k8sVersion") || "1.32.10-do.2";

// Create a DigitalOcean Kubernetes cluster
const cluster = new digitalocean.KubernetesCluster(clusterName, {
 name: clusterName,
 region: region,
 version: k8sVersion,
 nodePool: {
 name: "default-pool",
 size: nodeSize,
 nodeCount: nodeCount,
 },
});

// Create a Kubernetes provider using the cluster's kubeconfig
const k8sProvider = new k8s.Provider("k8s-provider", {
 kubeconfig: cluster.kubeConfigs[0].rawConfig,
});

// Install Gateway API CRDs (standard channel)
const gatewayApiCrds = new k8s.yaml.ConfigFile("gateway-api-crds", {
 file: "https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.0/standard-install.yaml",
}, { provider: k8sProvider });

// Create kgateway-system namespace
const kgatewayNamespace = new k8s.core.v1.Namespace("kgateway-system", {
 metadata: {
 name: "kgateway-system",
 },
}, { provider: k8sProvider });

// Install kgateway CRDs via Helm
const kgatewayCrds = new k8s.helm.v3.Release("kgateway-crds", {
 chart: "oci://cr.kgateway.dev/kgateway-dev/charts/kgateway-crds",
 version: "v2.1.2",
 namespace: kgatewayNamespace.metadata.name,
 createNamespace: false,
}, { provider: k8sProvider, dependsOn: [gatewayApiCrds, kgatewayNamespace] });

// Install kgateway control plane via Helm
const kgateway = new k8s.helm.v3.Release("kgateway", {
 chart: "oci://cr.kgateway.dev/kgateway-dev/charts/kgateway",
 version: "v2.1.2",
 namespace: kgatewayNamespace.metadata.name,
 createNamespace: false,
}, { provider: k8sProvider, dependsOn: [kgatewayCrds] });

// Create hello-world namespace
const helloWorldNamespace = new k8s.core.v1.Namespace("hello-world", {
 metadata: {
 name: "hello-world",
 },
}, { provider: k8sProvider });

// Deploy hello world application (using httpbin as shown in kgateway docs)
const helloWorldLabels = { app: "httpbin" };

const helloWorldDeployment = new k8s.apps.v1.Deployment("httpbin", {
 metadata: {
 name: "httpbin",
 namespace: helloWorldNamespace.metadata.name,
 },
 spec: {
 replicas: 1,
 selector: {
 matchLabels: helloWorldLabels,
 },
 template: {
 metadata: {
 labels: helloWorldLabels,
 },
 spec: {
 containers: [{
 name: "httpbin",
 image: "mccutchen/go-httpbin:v2.15.0",
 ports: [{
 containerPort: 8080,
 name: "http",
 }],
 }],
 },
 },
 },
}, { provider: k8sProvider, dependsOn: [helloWorldNamespace] });

const helloWorldService = new k8s.core.v1.Service("httpbin", {
 metadata: {
 name: "httpbin",
 namespace: helloWorldNamespace.metadata.name,
 },
 spec: {
 selector: helloWorldLabels,
 ports: [{
 port: 8000,
 targetPort: 8080,
 name: "http",
 }],
 },
}, { provider: k8sProvider, dependsOn: [helloWorldDeployment] });

// Create GatewayClass for kgateway
const gatewayClass = new k8s.apiextensions.CustomResource("kgateway-gateway-class", {
 apiVersion: "gateway.networking.k8s.io/v1",
 kind: "GatewayClass",
 metadata: {
 name: "kgateway",
 },
 spec: {
 controllerName: "kgateway.dev/kgateway",
 },
}, { provider: k8sProvider, dependsOn: [kgateway] });

// Create Gateway with LoadBalancer service (matching kgateway docs pattern)
const gateway = new k8s.apiextensions.CustomResource("http-gateway", {
 apiVersion: "gateway.networking.k8s.io/v1",
 kind: "Gateway",
 metadata: {
 name: "http",
 namespace: kgatewayNamespace.metadata.name,
 },
 spec: {
 gatewayClassName: "kgateway",
 listeners: [{
 name: "http",
 port: 8080,
 protocol: "HTTP",
 allowedRoutes: {
 namespaces: {
 from: "All",
 },
 },
 }],
 },
}, { provider: k8sProvider, dependsOn: [gatewayClass] });

// Create HTTPRoute for httpbin application
const httpRoute = new k8s.apiextensions.CustomResource("httpbin-route", {
 apiVersion: "gateway.networking.k8s.io/v1",
 kind: "HTTPRoute",
 metadata: {
 name: "httpbin",
 namespace: helloWorldNamespace.metadata.name,
 },
 spec: {
 parentRefs: [{
 name: "http",
 namespace: kgatewayNamespace.metadata.name,
 }],
 hostnames: [
 "*.nip.io",
 ],
 rules: [{
 backendRefs: [{
 name: "httpbin",
 port: 8000,
 }],
 }],
 },
}, { provider: k8sProvider, dependsOn: [gateway, helloWorldService] });

// Create a reference grant to allow the HTTPRoute to reference the service cross-namespace
const referenceGrant = new k8s.apiextensions.CustomResource("httpbin-reference-grant", {
 apiVersion: "gateway.networking.k8s.io/v1beta1",
 kind: "ReferenceGrant",
 metadata: {
 name: "allow-gateway-to-httpbin",
 namespace: helloWorldNamespace.metadata.name,
 },
 spec: {
 from: [{
 group: "gateway.networking.k8s.io",
 kind: "HTTPRoute",
 namespace: helloWorldNamespace.metadata.name,
 }],
 to: [{
 group: "",
 kind: "Service",
 }],
 },
}, { provider: k8sProvider, dependsOn: [gatewayApiCrds] });

// Create a LoadBalancer Service for the Gateway proxy
const gatewayProxyService = new k8s.core.v1.Service("gateway-proxy", {
 metadata: {
 name: "gateway-proxy",
 namespace: kgatewayNamespace.metadata.name,
 },
 spec: {
 type: "LoadBalancer",
 selector: {
 "gateway.networking.k8s.io/gateway-name": "http",
 },
 ports: [{
 name: "http",
 port: 8080,
 targetPort: 8080,
 protocol: "TCP",
 }],
 },
}, { provider: k8sProvider, dependsOn: [gateway] });

// Extract the LoadBalancer IP from our managed Service
const gatewayIP = gatewayProxyService.status.apply(status => {
 const ingress = status?.loadBalancer?.ingress;
 if (ingress && ingress.length > 0) {
 return ingress[0].ip || ingress[0].hostname || "pending";
 }
 return "pending";
});

// Exports
export const clusterNameOutput = cluster.name;
export const clusterEndpoint = cluster.endpoint;
export const kubeconfig = pulumi.secret(cluster.kubeConfigs[0].rawConfig);

// Export the LoadBalancer IP and nip.io URL
export const gatewayLoadBalancerIP = gatewayIP;
export const httpbinUrl = gatewayIP.apply(ip =>
 ip !== "pending" ? `http://httpbin.${ip}.nip.io:8080/get` : "Waiting for LoadBalancer IP..."
);

The demo uses nip.io, a wildcard DNS service that resolves *.IP.nip.io hostnames to the specified IP address. This eliminates the need to configure separate DNS records during testing. Once deployed, the httpbinUrl output provides a ready-to-use endpoint for validating the gateway configuration.

Migrate to the Gateway API

You can provision your cluster, install kgateway, and define your Gateway API resources in one Pulumi program, then migrate services incrementally and review each change. Start a project to plan your migration.

Get started

The short-term option: Chainguard’s ingress-nginx fork

For organizations that cannot immediately migrate to the Gateway API, there is another path forward. Chainguard has forked ingress-nginx as part of their EmeritOSS program, providing continued maintenance after the upstream project’s retirement.

This fork is not about adding new features. Chainguard is explicit that they are maintaining stability, not continuing development. Their commitment includes keeping dependencies updated, addressing CVEs on a best-efforts basis, and providing commercial container images with low vulnerability counts and SLAs. FIPS-compliant versions are also available for regulated environments.

The chainguard-forks/ingress-nginx repository on GitHub provides the maintained codebase. For teams running ingress-nginx in production, this fork offers a viable bridge while evaluating the Gateway API or other alternatives. However, this buys time rather than solving the problem permanently. The architectural limitations of the Ingress API remain, and the eventual move to a more expressive standard like the Gateway API is still the recommended path forward.

Looking ahead

The retirement of ingress-nginx marks the end of an era and the beginning of a more structured approach to Kubernetes networking. The Gateway API offers a robust framework that reflects how teams actually build and secure applications today.

While March 2026 provides a comfortable runway, early planning prevents rushed decisions. Setting up a test environment with kgateway to validate the new API constructs is a prudent next step.

Moving to the Gateway API means building a networking foundation ready for the next decade of infrastructure challenges.

Get Started with Pulumi

Use Pulumi's open-source SDK to create, deploy, and manage infrastructure on any cloud.

Resources

Gateway API Documentation – Comprehensive guides and examples.
Migration Guide – Official steps for transitioning from Ingress.
ingress2gateway – Tooling to automate resource conversion.
kgateway Documentation – Installation and configuration details.
Designing kgateway for Scalability – Deep dive into the krt framework.
Kubernetes Blog: ingress-nginx Retirement – Full context on the deprecation.

From 'Works on My Machine' to Production-Ready: Building AI Agents with Amazon Bedrock AgentCore

Engin Diri — Mon, 15 Dec 2025 00:00:00 +0200

Every developer building AI agents knows the gap between a working prototype and production deployment. Your fraud detection agent works perfectly on your laptop, but how do you deploy it with proper authentication, memory persistence, observability, and guardrails? This post walks through a complete journey from local development to production-ready AI agents using Amazon Bedrock AgentCore, the Strands SDK, and Pulumi.

TL;DR: This post shows how to deploy AI agents from local prototype to production using:

Strands SDK for agent development
Amazon Bedrock AgentCore for managed runtime with Firecracker isolation
AgentCore Memory for short-term and long-term context
MCP Gateway for secure tool integration with OAuth2
Pulumi for infrastructure as code

Complete code: GitHub

The Challenge: Bridging the Gap

Building AI agents is getting easier. Tools like the Strands Agents SDK let you create functional agents in just a few lines of code. Production deployment introduces challenges:

Stateless to Stateful: How does your agent remember previous conversations?
Single-tenant to Multi-tenant: How do you isolate different users’ data?
Implicit Trust to Zero Trust: How do you authenticate requests and enforce boundaries?
Console Logs to Observability: How do you trace requests across distributed systems?
Ad-hoc Tools to Managed Services: How do you expose tools securely at scale?

The approach is progressive: start simple, add complexity only when needed, and use infrastructure as code to make it reproducible.

Agents Beyond Chat: Event-Driven AI

When most people think of AI agents, they imagine chatbots: interactive systems responding to user questions. But many production agents never interact with end users directly. Instead, they’re triggered by event streams: Kafka topics, SQS queues, webhook callbacks, or scheduled jobs.

Our fraud detection agent is a prime example. It doesn’t wait for customers to ask questions. It’s triggered automatically when the payment processing system detects a new transaction. The “user” is another system, and the “conversation” is a structured event payload.

This event-driven pattern is becoming the new stack for AI agents. As Kai Waehner describes in his analysis of agentic AI with Kafka and Flink:

“Protocols like A2A and MCP are essential for standardizing agent behavior and communication. But without an event-driven substrate like Kafka and a stream-native runtime like Flink, these agents remain stuck in isolated interactions.”

The emerging architecture looks like this:

In production, you might use Amazon MSK (Managed Kafka), Amazon SQS, or EventBridge to deliver transaction events to your agent. The agent processes each event autonomously, takes action (blocking cards, creating tickets), and publishes results back to another topic.

What We’re Building: A Fraud Detection Agent

Our demo is a fraud detection agent that:

Receives transaction alerts with user ID, amount, merchant, and location
Gathers context from tools (user profiles, transaction history)
Analyzes for fraud patterns (impossible travel, velocity attacks, amount anomalies)
Takes protective action (blocking cards) when fraud is detected

We’ll build this progressively:

Stage	What Changes	Key Additions
Local Development	Running on your laptop	Strands SDK, Claude Opus 4.5, mock tools
First Deployment	Running in AgentCore	Containerized, Firecracker isolation, guardrails
Production-Ready	Full enterprise features	Memory, MCP Gateway, X-Ray observability

Local Development with Strands SDK

The Strands Agents SDK is an open-source framework from AWS that takes a model-driven approach to agent development. Instead of hardcoding complex workflows, you let the LLM reason about tool usage autonomously.

Here’s our complete local agent in about 30 lines:

# agent.py
from strands import Agent
from strands.models import BedrockModel
from tools import get_user_profile, get_recent_transactions, block_credit_card

# Configure Bedrock model with cross-region inference for Claude Opus 4.5
# Note: Cross-region inference may have different pricing than single-region
bedrock_model = BedrockModel(
 model_id="us.anthropic.claude-opus-4-5-20251101-v1:0",
 region_name="us-east-1",
)

# Define the Fraud Detection Agent
fraud_agent = Agent(
 model=bedrock_model,
 tools=[get_user_profile, get_recent_transactions, block_credit_card],
 system_prompt="""
 You are a Senior Fraud Analyst Agent at a financial institution.
 Your mission is to protect customers from fraudulent transactions in real-time.

 When you receive a transaction alert:
 1. Gather context by checking the user's profile and recent transaction history.
 2. Analyze the transaction for fraud indicators:
 - Impossible travel: Can the user physically be at the new location?
 - Unusual amounts: Is this significantly larger than typical spending?
 - High-risk merchants: Electronics stores, gift cards, wire transfers.
 - Geographic anomalies: Transactions far from home location.
 3. Make a decision:
 - If fraud indicators are present, BLOCK the card immediately.
 - If the transaction appears legitimate, allow it to proceed.
 4. Document your reasoning clearly.

 Prioritize customer protection. When in doubt, block the card.
 """,
)

Tools are simple Python functions decorated with @tool:

from strands import tool

@tool
def get_user_profile(user_id: str) -> dict:
 """Get user profile including home location and account status."""
 return {
 "user_id": user_id,
 "name": "John Doe",
 "home_location": "London, UK",
 "account_status": "active",
 }

@tool
def block_credit_card(user_id: str, reason: str) -> dict:
 """Block a user's credit card and create a fraud ticket."""
 return {
 "status": "blocked",
 "ticket_id": f"FRAUD-{user_id}-001",
 "message": f"Card blocked for user {user_id}: {reason}",
 }

Running the Local Demo

Here’s how we simulate an incoming transaction event:

# demo_simulation.py
from agent import fraud_agent

# The "Trigger" Event (Simulated Payload from Kafka/SQS)
new_transaction_event = """
ALERT: New Transaction Attempt
User ID: user_123
Amount: $2000
Merchant: Electronics Store
Location: Tokyo, Japan
Time: 09:15
"""

def run_demo():
 print("--- INCOMING EVENT STREAM ---")
 print(new_transaction_event)
 print("--- AGENT ACTIVATED ---")

 # Run the agent - processes the event autonomously
 result = fraud_agent(new_transaction_event)

 print("--- ANALYSIS COMPLETE ---")

if __name__ == "__main__":
 run_demo()

When we run this, the agent autonomously gathers context, analyzes the transaction, and takes action:

$ python demo_simulation.py
--- INCOMING EVENT STREAM ---
ALERT: New Transaction Attempt
User ID: user_123
Amount: $2000
Merchant: Electronics Store
Location: Tokyo, Japan
Time: 09:15
-----------------------------------
AGENT ACTIVATED. Analyzing...
Tool #1: get_user_profile
Tool #2: get_recent_transactions
## 🚨 FRAUD ANALYSIS REPORT
**Customer:** John Doe (user_123)
### Analysis:
#### 1. ❌ IMPOSSIBLE TRAVEL DETECTED
- **Last Transaction:** 09:00 in **London, UK**
- **Current Transaction:** 09:15 in **Tokyo, Japan**
- **Time Elapsed:** 15 minutes
- **Distance:** ~9,500 km (5,900 miles)
- **Minimum Flight Time:** ~12 hours
**Verdict:** It is physically impossible for John Doe to travel from London
to Tokyo in 15 minutes. This is a critical fraud indicator.
#### 2. ⚠️ HIGH-RISK MERCHANT
- Electronics stores are frequently targeted for fraud due to high resale value
#### 3. ⚠️ HIGH-VALUE TRANSACTION
- $2,000 at an electronics store represents a substantial purchase
### Decision: 🛑 BLOCK CARD IMMEDIATELY
Tool #3: block_credit_card
## ✅ ACTION COMPLETED
| Action | Status |
|--------|--------|
| Card Blocked | ✅ Confirmed |
| Ticket ID | TICKET-999 |
--- ANALYSIS COMPLETE ---

Notice how the agent reasoned about the event using tools and took autonomous action. This works great for development, but it lacks everything needed for production: authentication, persistence, isolation, monitoring, and governance.

First Deployment with AgentCore Runtime

Our local prototype works, but where do you actually run it? Lambda times out after 15 minutes. ECS and Kubernetes add operational overhead. And AI agents have different requirements than typical microservices: they reason through multi-step workflows, sometimes taking minutes to analyze a complex fraud case. You need something built for this workload.

Amazon Bedrock AgentCore fills this gap. It’s a managed runtime specifically designed for AI agents, with support for tasks running up to 8 hours. No timeout workarounds, no cluster management.

The security model is particularly relevant for fraud detection. AgentCore runs each session in a dedicated microVM using AWS Firecracker (the same isolation technology behind Lambda). When your agent processes one customer’s transactions, it’s completely isolated from other sessions. No shared memory, no cross-tenant contamination.

The deployment model is straightforward: containerize your agent and point AgentCore at the image. It works with Strands, LangGraph, CrewAI, or custom code. The same agent we built locally runs unchanged. We just wrap it in FastAPI and add a Dockerfile.

Cold Start and Latency

For fraud detection, latency matters. AgentCore’s Firecracker microVMs introduce a cold start penalty on the first invocation, typically 2-5 seconds to spin up the container. Subsequent invocations within the same session are fast (sub-second overhead beyond model inference time).

For real-time fraud detection, this cold start is acceptable because:

The LLM reasoning itself takes 5-15 seconds for complex analysis
Most production deployments use session pooling to keep warm instances available
The alternative (always-on containers) trades latency for cost

If sub-second total latency is critical, consider Lambda for simple rule-based checks and reserve AgentCore for complex reasoning that justifies the cold start overhead.

Containerizing the Agent

First, we wrap our agent in a FastAPI endpoint:

# agent.py (production version)
from fastapi import FastAPI, Request
from strands import Agent
from strands.models import BedrockModel
import os

app = FastAPI()

bedrock_model = BedrockModel(
 model_id="us.anthropic.claude-opus-4-5-20251101-v1:0",
 region_name="us-east-1",
 guardrail_id=os.environ.get("BEDROCK_GUARDRAIL_ID"),
 guardrail_version=os.environ.get("BEDROCK_GUARDRAIL_VERSION"),
 guardrail_trace="enabled",
)

fraud_agent = Agent(
 model=bedrock_model,
 tools=[get_user_profile, get_recent_transactions, block_credit_card],
 system_prompt="...",
)

@app.post("/invocations")
async def invoke(request: Request):
 body = await request.json()
 prompt = body.get("input", {}).get("prompt", "")
 result = fraud_agent(prompt)
 return {"output": {"message": str(result)}}

Adding Guardrails

Amazon Bedrock Guardrails help ensure your agent stays on task. We configure content filters and topic restrictions using Pulumi:

fraud_detection_guardrail = bedrock.Guardrail(
 "fraud-detection-guardrail",
 name="fraud-detection-guardrail",
 description="Ensure agent only handles fraud detection tasks",
 blocked_input_messaging="This request is outside the scope of fraud detection.",
 blocked_outputs_messaging="This response was blocked.",
 content_policy_config=bedrock.GuardrailContentPolicyConfigArgs(
 filters_configs=[
 bedrock.GuardrailContentPolicyConfigFiltersConfigArgs(
 type="SEXUAL", input_strength="HIGH", output_strength="HIGH",
 ),
 bedrock.GuardrailContentPolicyConfigFiltersConfigArgs(
 type="VIOLENCE", input_strength="HIGH", output_strength="HIGH",
 ),
 bedrock.GuardrailContentPolicyConfigFiltersConfigArgs(
 type="PROMPT_ATTACK", input_strength="HIGH", output_strength="NONE",
 ),
 # ... additional filters
 ],
 ),
 topic_policy_config=bedrock.GuardrailTopicPolicyConfigArgs(
 topics_configs=[
 bedrock.GuardrailTopicPolicyConfigTopicsConfigArgs(
 name="entertainment-media",
 type="DENY",
 definition="Questions about movies, TV shows, celebrities...",
 examples=["What is LOTR?", "Tell me about Star Wars"],
 ),
 bedrock.GuardrailTopicPolicyConfigTopicsConfigArgs(
 name="general-assistance",
 type="DENY",
 definition="Requests for coding help, math problems...",
 examples=["Write me a poem", "Help me with Python code"],
 ),
 ],
 ),
)

Deploying with Pulumi

The infrastructure is straightforward: ECR repository, IAM roles, and AgentCore Runtime:

# Create ECR repository
repo = ecr.Repository(
 "strands-agent-repo",
 name="fraud-detection-agent",
 force_delete=True,
)

# Build and push Docker image
image = docker.Image(
 "strands-agent-image",
 image_name=repo.repository_url.apply(lambda url: f"{url}:latest"),
 build=docker.DockerBuildArgs(
 context="../fraud-detection-agent",
 platform="linux/arm64",
 ),
 registry=docker.RegistryArgs(
 server=repo.repository_url,
 username=auth_token.user_name,
 password=auth_token.password,
 ),
)

# Deploy AgentCore Runtime
agent_runtime = bedrock.AgentcoreAgentRuntime(
 "fraud-detection-agent-runtime",
 agent_runtime_name="fraud_detection_agent",
 description="Fraud Detection Agent with Guardrails",
 role_arn=agent_role.arn,
 agent_runtime_artifact=bedrock.AgentcoreAgentRuntimeAgentRuntimeArtifactArgs(
 container_configuration=bedrock.AgentcoreAgentRuntimeAgentRuntimeArtifactContainerConfigurationArgs(
 container_uri=image.image_name,
 ),
 ),
 network_configuration=bedrock.AgentcoreAgentRuntimeNetworkConfigurationArgs(
 network_mode="PUBLIC",
 ),
 environment_variables={
 "BEDROCK_GUARDRAIL_ID": fraud_detection_guardrail.guardrail_id,
 "BEDROCK_GUARDRAIL_VERSION": fraud_detection_guardrail.version,
 },
)

Testing the Basic Deployment

After pulumi up, we can invoke the deployed agent via the AWS SDK:

# invoke_agent.py
import boto3
import json

def invoke_agent(agent_runtime_arn: str, prompt: str) -> dict:
 client = boto3.client("bedrock-agentcore", region_name="us-east-1")

 response = client.invoke_agent_runtime(
 agentRuntimeArn=agent_runtime_arn,
 runtimeSessionId="session_abc123...",
 payload=json.dumps({"input": {"prompt": prompt}}),
 qualifier="DEFAULT",
 )

 return json.loads(response["response"].read())

Running the same fraud scenario against the deployed agent:

$ ./test.sh
Selecting Pulumi stack 'dev'...
Getting agent runtime ARN from Pulumi...
Agent Runtime ARN: arn:aws:bedrock-agentcore:us-east-1:052848974346:runtime/fraud_detection_agent-W7zFZT3Vcn
Invoking agent with fraud detection alert...
========================================
Sending prompt to agent...
Prompt: ALERT: New Transaction Attempt
User ID: user_123
Amount: $2000
Merchant: Electronics Store
Location: Tokyo, Japan
Time: 09:15
--------------------------------------------------
Agent Response:
{
"output": {
"message": {
"role": "assistant",
"content": [
{
"text": "**Analysis:** Impossible travel confirmed. User was in London at 09:00
and this transaction attempted in Tokyo at 09:15. Physically impossible.
Combined with high-risk merchant and $2000 amount, this is fraudulent.
Card blocked (TICKET-999)."
}
]
},
"timestamp": "2025-12-15T04:17:48.762738+00:00",
"model": "fraud-detection-agent"
}
}

The agent now runs in a managed, isolated environment with guardrails. Next, we add memory and external tool integration.

Production-Ready: Memory and MCP Gateway

Our basic deployment handles individual transactions, but fraud detection in the real world requires context. If a customer’s card was blocked five minutes ago, the agent shouldn’t try to block it again. If there’s a pattern of suspicious activity over the past week, the agent should factor that into its analysis.

This is where AgentCore Memory and the MCP Gateway come in.

AgentCore Memory: Short-term and Long-term

Memory in AgentCore works at two levels. Short-term memory stores raw conversation events within a session. If the agent blocks a card in one message, it remembers this in subsequent messages during the same session.

Long-term memory is more interesting. It uses semantic extraction to persist facts across sessions. The system runs your conversations through a smaller model (we use Claude Haiku) to extract key facts, then stores those facts in a namespace you define. When a new session starts, the agent can query for relevant facts about that user. Even if the customer calls back tomorrow, the agent knows their card was blocked yesterday.

Here’s how we configure the memory infrastructure:

# Create AgentCore Memory with 30-day event expiry
fraud_detection_memory = bedrock.AgentcoreMemory(
 "fraud-detection-memory",
 name="fraud_detection_memory",
 description="Memory for fraud detection agent",
 event_expiry_duration=30, # Events expire after 30 days
 memory_execution_role_arn=memory_execution_role.arn,
)

# Custom semantic extraction for fraud-related facts
FRAUD_EXTRACTION_PROMPT = """
Extract key fraud-related facts from this conversation about the user. Focus on:
1. Credit card status (blocked/active, ticket IDs if blocked)
2. Fraud alerts and patterns detected for this user
3. Transaction history patterns (locations, amounts, frequencies)
4. User profile details relevant to fraud detection
5. Previous fraud investigations and their outcomes

Format as concise facts that can help identify fraud patterns across sessions.
"""

# Create long-term memory strategy
fraud_semantic_strategy = bedrock.AgentcoreMemoryStrategy(
 "fraud-semantic-strategy",
 name="fraud_semantic_extraction",
 memory_id=fraud_detection_memory.id,
 type="CUSTOM",
 namespaces=["/fraud-detection/users/{actorId}"],
 configuration={
 "type": "SEMANTIC_OVERRIDE",
 "extraction": {
 "append_to_prompt": FRAUD_EXTRACTION_PROMPT,
 "model_id": "us.anthropic.claude-3-5-haiku-20241022-v1:0",
 },
 "consolidation": {
 "model_id": "us.anthropic.claude-3-5-haiku-20241022-v1:0",
 },
 },
)

The namespace pattern /fraud-detection/users/{actorId} ensures each user’s facts are isolated. Claude Haiku handles the extraction efficiently, keeping costs low while maintaining accuracy.

Long-term memory extraction is asynchronous. Plan for a 20-40 second delay between storing conversations and having facts available for retrieval. Use short-term memory for immediate needs while long-term memories are being processed.

In the agent code, we integrate memory using Strands hooks:

from bedrock_agentcore.memory import MemoryClient

memory_client = MemoryClient(region_name="us-east-1")

# Retrieve long-term facts before processing
memories = memory_client.retrieve_memories(
 memory_id=memory_id,
 namespace=f"/fraud-detection/users/{actor_id}",
 query=f"fraud detection facts for user {actor_id}"
)

# Store conversation after processing
memory_client.create_event(
 memory_id=memory_id,
 actor_id=actor_id,
 session_id=session_id,
 messages=[
 (user_message, "USER"),
 (assistant_message, "ASSISTANT")
 ]
)

MCP Gateway: Tool Integration at Scale

The tools we’ve used so far are baked into the agent container. That works for simple cases, but what happens when you have dozens of tools across multiple teams? Or when a tool needs to call an external service that requires its own authentication?

The Model Context Protocol (MCP) is an open standard for AI systems to communicate with tools. AgentCore Gateway provides a managed MCP endpoint that handles authentication (JWT inbound, OAuth2 outbound) and lets agents discover tools using natural language queries rather than knowing exact tool names.

We deploy an MCP server for risk scoring with three tools:

from mcp.server.fastmcp import FastMCP

mcp = FastMCP("Fraud Risk Scoring Server")

@mcp.tool()
def calculate_risk_score(user_id: str, amount: float, merchant: str, location: str) -> dict:
 """Calculate comprehensive fraud risk score (0-100)."""
 # Risk calculation logic
 return {
 "risk_score": 75,
 "factors": ["amount_anomaly", "high_risk_merchant"],
 "recommendation": "BLOCK"
 }

@mcp.tool()
def get_fraud_indicators(user_id: str) -> dict:
 """Get known fraud indicators for a user."""
 return {
 "indicators": ["velocity_alert", "device_inconsistency"],
 "severity": "HIGH"
 }

@mcp.tool()
def check_merchant_reputation(merchant_name: str) -> dict:
 """Check merchant risk rating and fraud history."""
 return {
 "risk_rating": "HIGH",
 "fraud_reports": 47,
 "chargeback_rate": 0.15
 }

The Gateway infrastructure requires careful attention to authentication. We use two separate Cognito user pools to handle the dual authentication flow:

Inbound authentication (Agent → Gateway): The agent authenticates to the Gateway using JWT tokens from the first Cognito pool. This ensures only authorized agents can discover and call tools.

# Gateway with MCP protocol and semantic search
agentcore_gateway = bedrock.AgentcoreGateway(
 "fraud-detection-gateway",
 name="fraud-detection-mcp-gateway",
 description="Gateway for fraud detection MCP tools",
 role_arn=gateway_role.arn,
 authorizer_type="CUSTOM_JWT",
 authorizer_configuration={
 "customJwtAuthorizer": {
 "discoveryUrl": gateway_discovery_url,
 "allowedClients": [gateway_app_client.id],
 },
 },
 protocol_type="MCP",
 protocol_configuration={
 "mcp": {
 "supportedVersions": ["2025-03-26"],
 "searchType": "SEMANTIC",
 "instructions": "Gateway for fraud detection risk scoring tools.",
 },
 },
)

The searchType: "SEMANTIC" configuration enables natural language tool discovery. Instead of the agent needing to know exact tool names, it can query for “tools that check merchant risk” and the Gateway returns relevant matches.

Outbound authentication (Gateway → MCP Server): When the Gateway calls the MCP server, it needs its own credentials. The OAuth2 Credential Provider manages this second authentication layer:

# OAuth2 Credential Provider for Gateway -> MCP Server auth
oauth2_credential_provider = bedrock.AgentcoreOauth2CredentialProvider(
 "mcp-runtime-oauth2-provider",
 name="fraud-mcp-runtime-oauth2",
 credential_provider_vendor="CustomOauth2",
 oauth2_provider_config={
 "customOauth2ProviderConfig": {
 "oauthDiscovery": {"discoveryUrl": mcp_runtime_discovery_url},
 "clientId": mcp_runtime_app_client.id,
 "clientSecret": mcp_runtime_app_client.client_secret,
 },
 },
)

Connecting the pieces: Finally, we create a Gateway Target. This links the Gateway to the MCP server endpoint. It also specifies which credential provider to use for outbound calls:

# Connect Gateway to MCP Server
gateway_target = bedrock.AgentcoreGatewayTarget(
 "mcp-risk-server-target",
 name="mcp-risk-server-target",
 gateway_identifier=agentcore_gateway.gateway_id,
 target_configuration={
 "mcp": {"mcpServer": {"endpoint": mcp_server_endpoint_url}},
 },
 credential_provider_configuration={
 "oauth": {
 "providerArn": oauth2_credential_provider.credential_provider_arn,
 "scopes": [mcp_runtime_scope_string],
 },
 },
)

This three-part configuration (Gateway, Credential Provider, Target) separates concerns cleanly: the Gateway handles inbound auth and protocol translation, the Credential Provider manages outbound secrets, and the Target defines routing.

Observability with X-Ray and OpenTelemetry

Production systems need observability. We configure CloudWatch Logs for application logs and X-Ray for distributed tracing:

# Agent application logs
agent_log_group = cloudwatch.LogGroup(
 "agent-runtime-log-group",
 name=pulumi.Output.concat(
 "/aws/vendedlogs/bedrock-agentcore/", agent_runtime.agent_runtime_id
 ),
 retention_in_days=30,
)

agent_logs_delivery = cloudwatch.LogDelivery(
 "agent-logs-delivery",
 delivery_source_name=agent_logs_delivery_source.name,
 delivery_destination_arn=agent_logs_delivery_destination.arn,
)

# X-Ray traces
agent_traces_delivery_destination = cloudwatch.LogDeliveryDestination(
 "agent-traces-delivery-destination",
 name="fraud-agent-traces-dest",
 delivery_destination_type="XRAY",
)

agent_traces_delivery = cloudwatch.LogDelivery(
 "agent-traces-delivery",
 delivery_source_name=agent_traces_delivery_source.name,
 delivery_destination_arn=agent_traces_delivery_destination.arn,
)

In the container, we use AWS Distro for OpenTelemetry for auto-instrumentation:

CMD ["uv", "run", "opentelemetry-instrument", "uvicorn", "agent:app", "--host", "0.0.0.0", "--port", "8080"]

This gives us end-to-end tracing across agent invocations, memory operations, and Gateway calls with zero code changes.

AWS has announced that X-Ray SDKs will be deprecated in favor of OpenTelemetry by February 2027. Using ADOT from the start ensures your observability stack is future-proof.

Running the Demo

Deploy the Infrastructure

cd advanced-bedrock-deployment/infra
uv sync
uv run pulumi up

This deploys approximately 40 AWS resources including:

ECR repositories for agent and MCP server
AgentCore Memory with semantic extraction strategy
Two Cognito user pools for authentication
AgentCore Gateway with MCP protocol
CloudWatch log groups and X-Ray trace delivery
IAM roles and policies

Test Short-term Memory

./test.sh demo

This runs 5 scenarios in the same session, demonstrating short-term memory:

======================================================================
FRAUD DETECTION DEMO - SHORT-TERM MEMORY (Same Session)
======================================================================
SCENARIO 1: John Doe - Impossible Travel Detection
Expected: FRAUD - impossible travel from London to Tokyo in 15 minutes
Actor ID: user_123
Session ID: session_cf171803eb324a2aa9ac456b3013fe45...
----------------------------------------------------------------------
Agent Response:
{
"output": {
"message": {
"text": "**FRAUD CONFIRMED - CARD BLOCKED**
| Finding | Details |
|---------|---------|
| **User** | John Doe |
| **Ticket** | TICKET-123-999 |
| **Fraud Type** | Impossible Travel + Spending Anomaly |
**Analysis:** Transaction in Tokyo 15 minutes after London purchase
is physically impossible. The $2,000 amount is 10x the user's typical
spending range ($50-200). Card blocked immediately."
},
"memory_enabled": true,
"long_term_facts_retrieved": 3
}
}
======================================================================
SCENARIO 3: John Doe - Follow-up Transaction (Short-term Memory Test)
Expected: FRAUD - card should already be blocked (SHORT-TERM MEMORY)
Actor ID: user_123
Session ID: session_cf171803eb324a2aa9ac456b3013fe45... (SAME SESSION!)
----------------------------------------------------------------------
Agent Response:
{
"output": {
"message": {
"text": "**Card for user_123 (John Doe) is ALREADY BLOCKED
(Ticket: TICKET-123-999). No further action needed.**
This appears to be another fraudulent attempt on an already-compromised
account. The card was previously blocked due to impossible travel fraud.
The current transaction follows the same pattern: Gift Card Kiosk
purchases are a common fraud tactic for converting stolen card access
to untraceable value."
},
"memory_enabled": true,
"long_term_facts_retrieved": 3
}
}
======================================================================
SHORT-TERM MEMORY DEMO SUMMARY
======================================================================
[OK] John Doe - Impossible Travel Detection
[OK] Jane Smith - Normal Transaction
[OK] John Doe - Follow-up Transaction (Short-term Memory Test)
[OK] Alice Chen - Impossible Travel Detection
[OK] Jane Smith - Second Normal Transaction

The agent remembers within the same session that John’s card is already blocked, so there’s no need to block it twice.

Test Long-term Memory

./test.sh longterm-fresh

This demonstrates cross-session memory with two phases:

======================================================================
PHASE 1: INITIAL FRAUD DETECTION
These interactions create long-term memory facts about users.
======================================================================
SCENARIO: John Doe - Initial Fraud (Card Blocked)
Session ID: session_xyz789... (NEW SESSION)
→ Card blocked, fact stored to long-term memory
======================================================================
WAITING FOR LONG-TERM MEMORY EXTRACTION...
The semantic strategy needs time to process conversations into facts.
Waiting 30 seconds...
======================================================================
======================================================================
PHASE 2: NEW SESSIONS - LONG-TERM MEMORY RETRIEVAL
These are COMPLETELY NEW sessions. The agent should remember
fraud-related facts about users from Phase 1.
======================================================================
SCENARIO: John Doe - NEW SESSION (Long-term Memory Test)
Session ID: session_new456... (DIFFERENT SESSION!)
----------------------------------------------------------------------
Agent Response:
{
"output": {
"message": "CARD ALREADY BLOCKED. According to my records,
this user's card was previously blocked due to fraud.
Ticket FRAUD-user_123-001 exists from a previous investigation.
The card remains blocked.",
"memory_enabled": true,
"long_term_facts_retrieved": 2
}
}

Even in a completely new session, the agent retrieves facts from long-term memory and knows John’s card was blocked yesterday.

Test MCP Gateway

./test.sh gateway-fresh

This runs 6 diverse fraud scenarios using both local tools and MCP Gateway tools:

======================================================================
LOADING MCP GATEWAY CONFIGURATION...
======================================================================
Gateway URL: https://fraud-detection-mcp-gateway-xxx.gateway.bedrock-agentcore.us-east-1.amazonaws.com/mcp
Token Endpoint: https://fraud-gateway-xxx.auth.us-east-1.amazoncognito.com/oauth2/token
Scope: fraud-detection-gateway/invoke
MCP Gateway configuration loaded successfully!
======================================================================
FRAUD DETECTION DEMO - MCP GATEWAY INTEGRATION
======================================================================
LOCAL TOOLS:
- get_user_profile: User details and home location
- get_recent_transactions: Last known transaction
- block_credit_card: Block card and create ticket
MCP GATEWAY TOOLS (via Risk Scoring Service):
- calculate_risk_score: Fraud risk score 0-100
- get_fraud_indicators: Known fraud indicators
- check_merchant_reputation: Merchant risk rating
======================================================================
SCENARIO 1: HIGH-RISK MERCHANT - Jane at CryptoExchange123
MCP Gateway: ENABLED
----------------------------------------------------------------------
Agent Response:
{
"output": {
"message": {
"text": "**✅ CARD BLOCKED** - Ticket: TICKET-456-999
**Summary:** Jane Smith's card blocked due to a $5,000 crypto exchange
transaction that is 10x her typical spend and outside her normal merchant
categories. Location matched home, but amount and merchant anomalies
indicate likely fraud or compromised credentials."
},
"mcp_gateway_enabled": true,
"mcp_tools_count": 4
}
}
======================================================================
SCENARIO 5: COMBINED SIGNALS - Alice's Multi-Factor Fraud
MCP Gateway: ENABLED
----------------------------------------------------------------------
Agent Response:
{
"output": {
"message": {
"text": "## 🚨 FRAUD CONFIRMED - CARD BLOCKED
**User:** Alice Chen (user_321)
**Ticket ID:** TICKET-321-999
**Fraud Indicators Detected:**
| Factor | Finding |
|--------|---------|
| ⛔ Impossible Travel | Singapore (10:00) → Los Angeles (11:30) = 90 min |
| ⛔ Amount Anomaly | $2,500 vs typical $50-300 (8x+ normal) |
| ⛔ Merchant Mismatch | Cash advance vs usual food/retail |
| ⛔ High-Risk Merchant | QuickCashAdvance (cash advance services) |
**Action Taken:** Card blocked immediately. This transaction exhibits
classic signs of stolen card credentials being used for rapid cash extraction."
},
"mcp_gateway_enabled": true,
"mcp_tools_count": 4
}
}
======================================================================
MCP GATEWAY DEMO SUMMARY
======================================================================
[OK] 1. HIGH-RISK MERCHANT - Jane at CryptoExchange123
Gateway: YES (4 tools)
[OK] 2. VELOCITY ATTACK - Bob's Rapid Transactions
Gateway: YES (4 tools)
[OK] 3. AMOUNT ANOMALY - Alice's Large Purchase
Gateway: YES (4 tools)
[OK] 4. KNOWN FRAUD PATTERNS - John's Indicator Check
Gateway: YES (4 tools)
[OK] 5. COMBINED SIGNALS - Alice's Multi-Factor Fraud
Gateway: YES (4 tools)
[OK] 6. CLEAN TRANSACTION - Jane's Normal Purchase
Gateway: YES (4 tools)

The MCP Gateway tools provide additional context (merchant reputation, risk scores, fraud indicators) that helps the agent make better decisions.

Ship your AI agents to production

You can define the runtime, memory, gateway, and guardrails for your agents as infrastructure as code, and pair Pulumi with the coding agent you already use through the Pulumi MCP server. Start a project to deploy your own.

Get started

Architecture Summary

Here’s the complete architecture:

Key Takeaways

Event-driven first: Production agents are triggered by Kafka, SQS, or webhooks, not chat interfaces
Start simple: Build with Strands SDK locally, add AgentCore complexity only when needed
Guardrails early: Content filtering and topic restrictions prevent prompt injection
Memory has two levels: Short-term (session) and long-term (semantic extraction across sessions)
MCP for tools at scale: Gateway handles auth and discovery; you focus on tool logic
Observability required: X-Ray + CloudWatch are essential for debugging distributed AI
IaC for reproducibility: Pulumi deploys 40+ resources with proper dependencies

Cost Considerations

When evaluating AgentCore for production, consider these cost factors:

AgentCore Runtime charges per session-second. For fraud detection agents that typically run 10-30 seconds per transaction, this is cost-effective compared to maintaining always-on ECS containers. However, if your agents process thousands of transactions per second, the per-session costs add up. Consider batching or using Lambda for simple, fast agents.

Model costs dominate most deployments. Claude Opus 4.5 (used for reasoning) costs significantly more than Claude Haiku (used for memory extraction). We deliberately use Haiku for semantic extraction to keep memory costs low while maintaining accuracy. For high-volume production, consider whether Haiku or even smaller models could handle your primary reasoning.

Memory and Gateway have their own pricing tiers. AgentCore Memory charges for storage and retrieval operations. The Gateway adds overhead for OAuth2 token management and routing. For simple agents that don’t need cross-session memory or external tools, skip these components entirely.

Cognito user pools for authentication have a free tier (50,000 MAU) that covers most development and moderate production workloads. Beyond that, costs scale with active users.

Start with a basic deployment and add Memory/Gateway only when you have a clear need. The infrastructure cost savings of skipping unnecessary components often exceed the engineering cost of adding them later.

Conclusion

The fraud detection agent in this post grew from 30 lines of local Python to a production deployment with guardrails, dual-layer memory, MCP Gateway integration, and full observability, all defined as infrastructure as code.

The gap between prototype and production doesn’t have to be a chasm. AgentCore provides the managed runtime, Pulumi makes it reproducible, and the progressive approach lets you add complexity only when your use case demands it.

Get Started with Pulumi

Use Pulumi's open-source SDK to create, deploy, and manage infrastructure on any cloud.

AI Predictions for 2026: A DevOps Engineer's Guide

Engin Diri — Thu, 11 Dec 2025 00:00:00 +0000

The IDE is dying, and so is tool calling. OpenAI is not going to win. And next year, you’re going to be shipping code that you’ve never reviewed before, even as an experienced engineer.

These are bold claims, but the way we use AI in 2026 for coding and agents is going to look completely different. In this post, I want to cover my predictions and why they matter right now for DevOps engineers. Some of these are definitely hot takes, but that’s what makes this conversation worth having.

IDEs are dead (at least as we know them)

Traditional IDEs where the code is the focus of the interface are simply going to become irrelevant. We’re moving toward agent manager interfaces where we can kick off agents in parallel to work on different features in a codebase or even work on different projects at the exact same time.

We’re already seeing this transition. Google Antigravity combines a familiar AI-powered coding experience with a new agent-first interface. You can deploy agents that autonomously plan, execute, and verify complex tasks across your editor, terminal, and browser. Cursor 2.0 lets you run up to eight agents in parallel on a single prompt, using git worktrees or remote machines to prevent file conflicts. Each agent operates in its own isolated copy of your codebase.

AWS validated this direction at re:Invent 2025 by announcing “frontier agents” including Kiro for autonomous coding, along with dedicated security and DevOps agents. These agents maintain state, log actions, operate with policy guardrails, and integrate directly with CI/CD pipelines.

For infrastructure specifically, Pulumi Neo represents this same shift. Instead of writing code or running CLI commands for every operation, you describe what you need in natural language and Neo handles the implementation. It works across your entire infrastructure, understanding dependencies and creating execution plans that go through pull requests for review.

For DevOps engineers, this means your pipelines need to accommodate AI-generated code at scale. Multiple agents working simultaneously need isolated, reproducible environments. More generated code means more artifacts to track, version, and deploy.

The different paths to AI dominance

A lot of people think that in the future, a single large language model will have a monopoly and be the best at absolutely everything. But what’s really going to happen is different providers will specialize and focus on being the best at different things.

Google is going down the generalist route with Gemini, aiming to be the jack-of-all-trades. Anthropic is focusing on being the best for coding. You can see this in the benchmarks: when Opus 4.5 came out, the first benchmark they highlighted was for software engineering, because that’s what Anthropic is focusing on.

Amazon is carving out its own niche with the Nova model family, announced at re:Invent 2025. The Nova 2 lineup includes specialized models: Pro for complex reasoning, Sonic for real-time voice conversations, and Omni for simultaneous text, audio, and video processing. With Nova Forge, organizations can build custom frontier models by combining their proprietary data with AWS open weight models. The re:Invent message was clear: leveraging your first-party data is now fundamental to going beyond generic AI. We’re talking about 30-40% increases in accuracy when you bring your own data into the equation.

But here’s the hot take: I don’t think OpenAI is going to come out on top with any kind of specialization. They’ve disappointed time and time again with GPT-5 and GPT-4.5. With 4.5, they seemed to try to be the creative specialist, but it just didn’t work. The GPT-5 launch in August 2025 was described as “barely better than last month’s flavor of the month” and on some metrics it’s actually worse than earlier models.

For DevOps teams, this specialization means you’ll need infrastructure that’s model-agnostic and supports multiple AI backends. Plan for secrets management across multiple LLM providers and design your systems to swap models based on the task at hand.

The local AI breakthrough

2026 will be the year of local AI. We didn’t see that much this year besides DeepSeek at the start of 2025, which was a big deal. We had a couple of new models like Qwen 3, but nothing that fundamentally changed the game. Now we’re starting to see new hardware that makes it obvious we’re going to be able to run very large models on smaller devices.

There’s new AI chips that can run upwards of 120 billion parameter large language models on the edge, which would be a complete game-changer. Right now, hardware requirements are one of the biggest challenges for scaling local AI. If we can solve the hardware problem, we get 100% data privacy and zero-millisecond latency for our agents.

AWS is addressing this with Trainium3 UltraServers, their 3nm AI chips delivering 4.4x more compute than the previous generation. More significantly, AWS AI Factories allow organizations to deploy racks of Trainium chips and NVIDIA GPUs directly into their own data centers, addressing data sovereignty concerns while keeping AI inference close to the data.

For DevOps, this opens the door to zero-latency inference in CI/CD pipelines, complete data privacy for sensitive codebases, and reduced cloud costs for AI-heavy workloads.

Engineers as system architects

We’re finally going to get to the point where we’re not the coders. We delegate that entirely to our coding agents, and we become the system architects. This mirrors the evolution of other engineering disciplines. Civil engineers don’t fabricate the steel beams; they design the structure and verify the integrity.

I think of this as a three-step process:

Define: Set your objectives and the system that your agents will operate under
Orchestrate: Delegate the coding to your agents
Validate: Ensure the quality of the outputs and the overall system

We’re still in the loop. We are the final say in whatever is created, but we’re delegating the grunt work to our coding agents.

This is exactly the model that Pulumi Neo implements for infrastructure. When you give Neo a complex request, it creates a task plan outlining the steps it will take to accomplish your goal. This plan provides transparency into Neo’s approach and gives you the opportunity to adjust the strategy before execution begins. Neo operates in different modes: Review mode where everything requires approval, Balanced mode where only deployments need sign-off, or Auto mode for full autonomy. You define the boundaries, Neo orchestrates the work, and you validate through pull requests and previews.

For DevOps engineers, this shift means building robust validation infrastructure becomes critical. When AI writes the code, you need automated testing pipelines, security scanning, and verification systems that can operate at the speed of AI-generated changes.

Code execution is replacing tool calling

Here’s a key insight that kept coming up at re:Invent: models are no longer the bottleneck. Context is. Our agents are going to change a lot next year because code execution is starting to replace tool calling. The problem with tool calling right now is that all the capabilities you give an agent take up context upfront. When you try to give a lot of different tools to an agent, you completely overwhelm it.

Anthropic’s research on code execution with MCP addresses exactly this problem. Code execution is a massive token reduction, faster, and more flexible. You’re giving the agent the ability to generate its own capabilities at runtime by writing code to interact with APIs. A workflow that previously consumed about 150,000 tokens when tools were passed directly through the model was reimplemented with code execution and used only about 2,000 tokens. That’s a 98.7% reduction.

AWS embraced this pattern with Amazon Bedrock AgentCore, which now includes code interpretation capabilities. AgentCore supports any agent framework (CrewAI, LangGraph, OpenAI SDK) and provides memory, browser tools, and observability features that make code execution practical at enterprise scale.

For DevOps, this means you need sandboxed, secure execution environments for AI-generated code. Running agent-generated code requires appropriate isolation, resource limits, and monitoring.

Progressive disclosure and composable skills

The best part about code execution flexibility is it unlocks progressive disclosure. All I mean by that is: you have a lot of capabilities for an agent, but you don’t actually give all of them upfront. Instead, you allow the agent to discover capabilities and then leverage them in a more flexible way.

For each capability, you just have a bit of metadata or description that loads upfront. When the agent decides to leverage that capability, then you load the full instructions. Now you can practically scale to infinity because all capabilities don’t have to be loaded at runtime.

Claude Skills is a good example of this pattern. Skills are organized folders of instructions, scripts, and resources that agents can discover and load dynamically. At session start, the agent scans available skills and populates the system prompt with just a brief name and description (around 100 tokens). The full skill prompt loads only after Claude selects it, preventing context bloat while maintaining discoverability.

Kiro Powers addresses the same problem. Connecting five MCP servers can consume over 50,000 tokens, roughly 40% of an AI model’s context window, before you even type your first request. Powers bundle MCP servers, steering files, and hooks into units that load dynamically based on conversation context. Mention “payment” and the Stripe power activates. Datadog, Figma, and others have powers available.

For DevOps, this translates to modular infrastructure definitions, on-demand capability loading, and efficient resource utilization. Think about how you can apply this pattern to your own automation.

Equip your agents with Pulumi skills

Give Claude Code, Cursor, or any coding agent working Pulumi patterns through Agent Skills and the Pulumi MCP server, so it generates infrastructure that deploys.

Get started

Agent-to-agent protocols are finally happening

Agent-to-agent protocols are where AI agents operate in a peer network, discover each other’s capabilities in real time, and interact autonomously. When Google released their A2A protocol earlier this year, there was a ton of buzz. A lot of people thought it was going to be the next big standard, like the next MCP. But then it kind of fell to the wayside.

The big reason is the chicken-and-egg problem. For A2A to be useful, you need many people to adopt it at the same time. Otherwise, if you build an A2A-compatible agent, it has no other agents to talk to. The whole value proposition is lost unless you already have a big network to attach to.

But that’s finally changing. The Linux Foundation launched the A2A project in June 2025, and adoption is accelerating. Adobe, Microsoft, SAP, ServiceNow, and S&P Global are all implementing A2A. In July 2025, Google released version 0.3 of the A2A protocol with a more stable interface critical to accelerating enterprise adoption.

Machines paying machines

My next big prediction is that machines paying machines is going to become a very big thing. Coinbase released the x402 protocol for exactly this: building AI agents that you expose over the internet but require payment whenever someone else interacts with them.

This goes really well with agent-to-agent protocols. You can create a peer network where you monetize your agents. They all leverage each other but make payments whenever they take advantage of another agent’s capabilities. Cryptocurrency is the perfect solution for this kind of machine-to-machine network because it enables micropayments to be made quickly and globally.

The x402 protocol has achieved 156,000 weekly transactions with 492% growth since launching in May 2025. It’s now integrated with Anthropic’s MCP Protocol, Google Gemini, OpenAI Codex, and other platforms. Stablecoins like USDC enable charging per request, per service, or per second of usage at near-zero transaction costs, enabling payments as low as $0.001 per request.

Artifact reviews instead of diff reviews

When we want to do a rigorous code review, traditionally, we look line by line at all the changes. But coding agents are getting to the point where they can prove their code works through artifacts. Instead of reviewing line by line, we can look at browser recordings, full working demos of a backend API, and other artifacts.

Google Antigravity is a perfect example. As part of its coding process, it can autonomously spin up your website, visit it, scroll through it, take screenshots, and record everything. Agents generate artifacts, including tangible deliverables like task lists, implementation plans, screenshots, and browser recordings. You can verify the agent’s logic at a glance.

Amazon Nova Act takes this further. It enables AI agents to automate browser-based tasks like form filling, QA testing, and workflow validation with over 90% reliability. The service includes built-in observability through live viewing, CloudTrail logging, and session replay, making it possible to review what an agent actually did rather than parsing through code changes.

Shipping code you’ve never read

For the last prediction, we’re tying everything together. We’ve talked about reviewing artifacts instead of diffs, creating systems instead of coding, and the new capabilities for agents with code execution.

We’re going to get to the point very quickly where we’re shipping code that we have never read before. And I’m not talking about people who vibe code. Even experienced engineers are going to trust their systems so much that they have the ability to review the code but they’re not going to. We’re just going to ship to production after reviewing the artifacts.

I presented on this exact topic at the Tel Aviv Pulumi User Group meetup at Qodo HQ back in October, where I demonstrated how Pulumi Neo’s autonomous decision-making capabilities can handle infrastructure tasks that we traditionally managed manually. Qodo is doing fascinating work in this space with their agentic development tools, building systems that let you trust the output without necessarily reviewing every line.

I’m not saying we’re taking the human completely out of the loop. I’m saying we’re going to have a lot of trust in our systems and a validation process that includes us, but that doesn’t necessarily have to be us actually looking at the code. Tools like Pulumi Neo create pull requests with clear documentation of changes, run previews to validate infrastructure modifications, and provide the transparency needed to ship with confidence.

The path forward

The predictions I’ve outlined point to a fundamental shift in how software gets built and deployed. For DevOps engineers, this isn’t a threat but an opportunity to become more strategic and less operational. We’re entering the battle of the agentic frameworks, where the winners will be those who can build faster, cheaper agentic applications through their platforms.

The immediate reality is that your CI/CD pipelines need to accommodate AI-generated code at scale, your secrets management needs to handle multiple LLM providers, and your execution environments need proper sandboxing for agent-generated code. These aren’t future concerns; they’re requirements for working effectively with the AI tools available today.

Looking further out, the engineers who thrive will be those who embrace the system architect role. Define clear objectives and constraints for your AI agents. Build validation frameworks that can verify outcomes without requiring line-by-line code review. Design infrastructure that’s modular enough to load capabilities on demand.

The technology to make this happen already exists. Agent orchestration platforms are shipping. Code execution is replacing tool calling. Progressive disclosure patterns are proven. The question isn’t whether these changes are coming; it’s whether you’ll be ready when they arrive.

Start building for the AI-driven future today

If you want to experience what this future looks like right now, Pulumi Neo is the place to start. Neo lets you make natural language requests for routine infrastructure tasks, analysis, and management. Instead of writing code for every operation, you describe what you need, and Neo handles the implementation, creating task plans, running previews, and opening pull requests for your review.

Whether you’re looking to update outdated resources across your infrastructure, analyze your cloud spend, or automate complex multi-step workflows, Neo provides the agent-first experience that’s defining the next generation of DevOps tooling.

Get started with Pulumi Neo and see how AI-powered infrastructure automation can transform your workflow.
Project: Build a production-ready AI agent by moving from local prototypes to deployed infrastructure with Amazon Bedrock AgentCore, the Strands SDK, and Pulumi.

Grounded AI: Why Neo Knows Your Infrastructure

Engin Diri — Fri, 24 Oct 2025 00:00:00 +0000

Ask a generic LLM to “fix my broken deployment,” and you’ll get generic advice. Ask Pulumi Neo the same question, and you’ll get a fix plan grounded in your actual infrastructure state.

The difference isn’t about better prompts or newer models. It’s about what the AI actually knows. Generic LLMs have been trained on the internet. Neo has been trained on your infrastructure.

This distinction matters more than you’d think.

The grounding problem

Most AI tools treat infrastructure like a text generation problem. You describe what you want, the model produces code, and you hope it works.

This approach fails more often than it succeeds because the AI has no connection to your actual infrastructure. It doesn’t know what resources you’ve already deployed, what dependencies exist, or what policies you need to follow. It’s generating code from patterns it learned on the internet, not from understanding your environment.

Pulumi Neo takes a different approach. It’s trained on your infrastructure context: your programs, your state, your resource relationships. When you ask Neo about drift, it doesn’t guess. It queries your actual deployment history. When it suggests a fix, it reasons from your program graph, not generic examples.

Neo doesn’t just autocomplete code. It understands state, resources, dependencies, and cloud behavior. When something fails, Neo explains why, not just what.

This is what makes it grounded AI: it’s anchored to the reality of your infrastructure, not floating in a sea of internet-trained probabilities.

What makes Neo different

The foundation matters more than you’d think.

Traditional AI tools fail in DevOps because they operate in a vacuum. Ask a generic LLM to “fix my broken deployment,” and you’ll get generic advice that ignores your specific infrastructure, your state, your constraints.

The difference with systems like Pulumi Neo comes down to what powers them: not just data, but context.

Think of it this way: a data lake is a massive repository of information sitting inert, waiting to be queried. A context lake is something else entirely. It’s a structured repository that aggregates knowledge about your domain and feeds it to AI systems.

A context lake contains things that matter:

Your infrastructure programs, resource definitions, API schemas, service dependencies. In Pulumi’s case, this includes your program graph, component models, stack configurations.

Real-time metrics, deployment history, drift detection, policy violations. The live pulse of your infrastructure as it actually runs in production.

Ownership information, compliance policies, access controls, quality signals. The organizational context that determines what changes are safe, who can make them, and why they matter.

Pulumi’s approach builds on this principle. Your infrastructure programs, state files, resource metadata, policy definitions. All of this becomes queryable context. Neo doesn’t hallucinate solutions because it’s grounded in your actual infrastructure. It knows what you’ve deployed, how resources relate to each other, what dependencies exist, what’s drifted.

This is the architectural shift that makes AI-powered DevOps actually work. AI agents are only as effective as the context they can access and the guardrails you have in place to keep them in check. You’re not just automating actions anymore. You’re automating understanding.

Grounded reasoning in practice

Early DevOps automated actions. Modern AI agents automate decisions. But grounded AI does something different: it automates understanding.

When you combine observability data, infrastructure-as-code programs, and deployment history into a context lake, the AI doesn’t just predict what might happen. It reasons about what is happening based on your actual infrastructure state.

Neo can watch infrastructure drift, identify whether it stems from a manual change or a misconfigured resource, and generate a fix plan that maps back to your Pulumi program. That feedback isn’t guesswork. It’s grounded in the same infrastructure metadata developers already use. The context lake ensures Neo isn’t making probabilistic predictions. It’s reasoning from your specific infrastructure truth.

This is what separates grounded AI from generic LLMs. Neo doesn’t just automate deployment. It understands the intent behind it.

The human layer stays critical

Grounded AI doesn’t replace engineering judgment. Every engineer who’s tried to prompt a generic LLM to “write a Pulumi program for me” knows how quickly hallucinations creep in. You still need human context: the judgment to choose the right platform, the discipline to model dependencies, the awareness of compliance and cost.

That’s where grounded AI like Neo fits best. It’s not a chatbot for infrastructure. It’s an extension of your own reasoning.

Neo learns from the same program graph and state data you already manage, drawing from a continuously updated context lake of your infrastructure reality. Its recommendations stay grounded in your environment, not floating in a generic prompt window trained on the public internet.

The opportunity isn’t about replacing engineers. It’s about creating smarter feedback loops between human expertise and AI that actually understands your infrastructure.

What grounded AI enables

The first generation of DevOps automated deployment. Grounded AI automates understanding.

Infrastructure becomes queryable. AI-powered observability correlates incidents before they cascade. CI/CD becomes continuous reasoning rather than continuous execution. Platform teams spend less time fighting YAML and more time guiding systems that reason alongside them.

The context lake architecture makes this possible. Instead of static documentation and scattered tribal knowledge, your infrastructure context becomes something AI agents can actually query and reason over.

Grounded AI won’t replace engineers. But it will change what engineering means: less time translating intent into YAML, more time reasoning about systems at a higher level.

Try it yourself

Want to see what infrastructure cognition looks like in practice? Get started with Neo and ask it about your infrastructure. Watch it reason through drift detection, generate fix plans, or explain complex resource relationships in plain English. For a wider survey of the category, see our guide to the best AI infrastructure tools.

Neo is available today for teams using Pulumi Cloud. The cognitive layer isn’t coming. It’s already here.

Golden Paths in IDPs: A Complete Guide to Reusable Infrastructure with Pulumi Components and Templates

Engin Diri — Wed, 20 Aug 2025 10:00:00 +0200

Welcome to the second post in our IDP Best Practices series. In this article, we explore how to create golden paths, pre-architected, reusable infrastructure patterns that help standardize and accelerate cloud development.

Modern cloud platforms offer endless options, over 200 AWS services, sprawling Azure catalogs, and countless DevOps tools. The result? Developers face decision fatigue and inconsistent implementations. Golden paths solve this by providing ready-to-use, production-grade infrastructure that encodes your organization’s best practices, security policies, and operational standards.

In this guide, you’ll learn how to build golden paths for your Internal Developer Platform using two core Pulumi constructs: Components, reusable infrastructure building blocks, and Templates, predefined, deployable patterns. You’ll see how to create infrastructure abstractions that are written once, shared across teams, and consumed in any language, turning weeks of setup into minutes of developer-ready infrastructure.

The complete code examples from this post are available on GitHub.

The Platform Engineering Layer Cake: A Model for IDPs

To understand where golden paths fit into an Internal Developer Platform (IDP), think in layers. This three-tier model structures your platform to deliver increasing levels of abstraction, reuse, and developer value:

Layer 1: Infrastructure Layer

This is the foundation: raw cloud resources include VMs, databases, networks, and storage, which are the fundamental building blocks from AWS, Azure, GCP, and other providers. Pulumi gives you programmatic access to these resources through native providers, but working at this level requires deep infrastructure knowledge.

Layer 2: Platform Layer - Components

This is where the magic happens. Pulumi Components take those raw resources and package them into higher-level abstractions. Instead of manually configuring 20+ AWS resources for a secure web application, you create a component that handles all that complexity and exposes just the configuration that matters:

const app = new SecureWebApplication("my-app", {
 instanceType: "t3.medium",
 minReplicas: 2,
 maxReplicas: 10,
 enableWAF: true,
 environment: "production"
});

Part 1: Building Reusable Infrastructure Components

Reusable infrastructure components act as the foundation of your platform. They encapsulate complexity while remaining composable and scalable. Let’s build a real-world example: a component that deploys containerized microservices to AWS Fargate.

The Power of Multi-Language Components

Here’s what makes Pulumi components revolutionary: write once, consume anywhere. Your platform team can author components in TypeScript, but application teams can consume them in Python, Go, .NET, Java, or even YAML. This separation of concerns is crucial for scaling platform engineering across diverse teams.

Building a Microservice Component Step-by-Step

Let’s create a component that encapsulates everything needed to deploy a production-ready microservice:

// microservice-component.ts
import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";

export interface MicroserviceArgs {
 appPath: string; // Path to application code
 cpu?: number; // CPU units (256, 512, 1024, etc.)
 memory?: number; // Memory in MB
 port?: number; // Application port
 desiredCount?: number; // Number of tasks
}

export class MicroserviceComponent extends pulumi.ComponentResource {
 public readonly url: pulumi.Output<string>;
 public readonly serviceName: pulumi.Output<string>;
 public readonly clusterName: pulumi.Output<string>;

 constructor(name: string, args: MicroserviceArgs, opts?: pulumi.ComponentResourceOptions) {
 super("custom:infrastructure:Microservice", name, {}, opts);

 // Set defaults
 const cpu = args.cpu || 256;
 const memory = args.memory || 512;
 const port = args.port || 8080;
 const desiredCount = args.desiredCount || 2;

 // Create ECR repository for Docker images
 const repository = new awsx.ecr.Repository(`${name}-repo`, {
 forceDelete: true
 }, { parent: this });

 // Build and push Docker image
 const image = new awsx.ecr.Image(`${name}-image`, {
 repositoryUrl: repository.url,
 context: args.appPath,
 platform: "linux/amd64",
 }, { parent: this });

 // Create Application Load Balancer
 const loadBalancer = new awsx.lb.ApplicationLoadBalancer(`${name}-lb`, {
 defaultTargetGroup: {
 port: port,
 protocol: "HTTP",
 healthCheck: {
 path: "/health",
 interval: 30,
 timeout: 5,
 healthyThreshold: 2,
 unhealthyThreshold: 3,
 }
 }
 }, { parent: this });

 // Create ECS cluster
 const cluster = new aws.ecs.Cluster(`${name}-cluster`, {}, { parent: this });

 // Create Fargate service
 const service = new awsx.ecs.FargateService(`${name}-service`, {
 cluster: cluster.arn,
 taskDefinitionArgs: {
 container: {
 name: name,
 image: image.imageUri,
 cpu: cpu,
 memory: memory,
 essential: true,
 portMappings: [{
 containerPort: port,
 targetGroup: loadBalancer.defaultTargetGroup,
 }],
 },
 },
 desiredCount: desiredCount,
 assignPublicIp: true,
 }, { parent: this });

 // Export outputs
 this.url = pulumi.interpolate`http://${loadBalancer.loadBalancer.dnsName}`;
 this.serviceName = service.service.name;
 this.clusterName = cluster.name;

 // Register outputs
 this.registerOutputs({
 url: this.url,
 serviceName: this.serviceName,
 clusterName: this.clusterName,
 });
 }
}

Documenting for Discoverability and Adoption

To make your component easy to use, add comprehensive documentation and examples to a README.md file:

# MicroserviceComponent

Abstraction for resources needed when using AWS container services.

A component to abstract the details related to:
- Creating a docker image and pushing it to AWS ECR.
- Deploy to ECS Fargate using the docker image.

# Inputs

* appPath: Path to local folder containing the app and Dockerfile.
* port: Port to expose via an ALB.
* cpu (Optional): CPU capacity. Defaults to 256 (i.e. 0.25 vCPU).
* memory (Optional): Memory capacity. Defaults to 512 (i.e. 0.5GB).
* containerName (Optional): Name of the container. Defaults to "my-app".

# Outputs

* publicUrl: The DNS name for the loadbalancer fronting the app.

# Usage
## Specify Package in `Pulumi.yaml`

Add the following to your `Pulumi.yaml` file:
Note: If no version is specified, the latest version will be used.

... omit for brevity ...

This documentation will help developers understand how to use your component effectively, including required inputs, outputs, and example usage.

Publishing Your Component via Private Registry

Once your component is ready, publish it to your Pulumi Private Registry:

# Tag your component version
git tag v1.0.1

# Publish to your organization's registry
pulumi package publish https://registry.pulumi.com/myorg/microservice-component

Now any team in your organization can discover and use your component, regardless of their language preference. All they need is to navigate to the Components section in the Pulumi IDP and search for microservice-component.

Part 2: From Building Blocks to Golden Paths

While components are powerful, they’re just the starting point. Golden path templates layer on opinionated scaffolding, workflows, and compliance best practices that guide developers to production.

Golden Path Maturity: From Zero to Product-Grade Platforms

Teams evolve from ad hoc deployments to mature, productized templates with versioning, metrics, and governance. These maturity levels reflect how deeply your platform enables safe, consistent delivery.

Organizations typically progress through three stages of golden path maturity:

Stage 1: No Golden Paths

Every team reinvents the wheel
Inconsistent practices across projects
High cognitive load on developers
Security and compliance gaps

Stage 2: Dawn of Templates

Basic cookie-cutter templates emerge
Some standardization begins
Manual processes still dominate
Limited support and evolution

Stage 3: Templates as Products

Golden paths have dedicated owners
Regular release cycles and versioning
Migration guides for updates
Metrics track adoption and success
Continuous improvement based on feedback

What Makes a Golden Path Golden?

Drawing from Spotify’s pioneering work, golden paths share these characteristics:

Pre-architected and Supported: The platform team owns and supports the path
Optional but Recommended: Developers can deviate, but staying on the path ensures support
Transparent Abstractions: The implementation is visible, not a black box
Extensible: Teams can add project-specific resources without breaking the pattern
Evolutionary: Templates improve based on feedback and new requirements

Building a Go Microservice Golden Path

Let’s create a complete golden path for Go microservices that includes:

Application scaffolding with best practices
Infrastructure deployment using our component
CI/CD pipeline configuration
Observability setup
Security controls

Step 1: Template Structure

go-microservice-boilerplate/
├── Pulumi.yaml # Infrastructure definition
├── src/ # Application code
│ ├── main.go # Go microservice
│ ├── Dockerfile # Container definition
│ └── go.mod # Dependencies
└── README.md # Documentation

Step 2: Application Scaffolding

The src/main.go file contains a minimal Go microservice with all the best practices and compliance guardrails baked in. This helps developers to quickly get started and extend the application without worrying too much about selecting the right libraries or frameworks.

// source/main.go
package main

import (
 "context"
 "flag"
 "net/http"
 "os"
 "os/signal"
 "time"

 "github.com/labstack/echo/v4"
 "github.com/labstack/echo/v4/middleware"
 "go.opentelemetry.io/contrib/instrumentation/github.com/labstack/echo/otelecho"
 "go.opentelemetry.io/otel"
 "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
 "go.opentelemetry.io/otel/sdk/resource"
 sdktrace "go.opentelemetry.io/otel/sdk/trace"
 semconv "go.opentelemetry.io/otel/semconv/v1.34.0"
)

func initTracer() (*sdktrace.TracerProvider, error) {
 exporter, err := otlptracehttp.New(context.Background())
 if err != nil {
 return nil, err
 }

 resource, err := resource.Merge(
 resource.Default(),
 resource.NewWithAttributes(
 semconv.SchemaURL,
 semconv.ServiceNameKey.String("${PROJECT}"),
 semconv.ServiceVersionKey.String("1.0.0"),
 ),
 )
 if err != nil {
 return nil, err
 }

 tp := sdktrace.NewTracerProvider(
 sdktrace.WithBatcher(exporter),
 sdktrace.WithResource(resource),
 sdktrace.WithSampler(sdktrace.AlwaysSample()),
 )

 otel.SetTracerProvider(tp)
 return tp, nil
}

func echoHandler(c echo.Context) error {
 message := c.QueryParam("message")
 if message == "" {
 message = "Hello from Go microservice!"
 }

 return c.JSON(http.StatusOK, map[string]string{
 "echo": message,
 "service": "${PROJECT}",
 "timestamp": time.Now().UTC().Format(time.RFC3339),
 })
}

func healthHandler(c echo.Context) error {
 return c.JSON(http.StatusOK, map[string]string{
 "status": "healthy",
 "service": "${PROJECT}",
 })
}

func main() {
 healthCheck := flag.Bool("health-check", false, "Run health check and exit")
 flag.Parse()

 if *healthCheck {
 resp, err := http.Get("http://localhost:8080/health")
 if err != nil || resp.StatusCode != http.StatusOK {
 os.Exit(1)
 }
 os.Exit(0)
 }

 tp, err := initTracer()
 if err != nil {
 panic(err)
 }
 defer func() {
 if err := tp.Shutdown(context.Background()); err != nil {
 panic(err)
 }
 }()

 e := echo.New()

 e.Use(middleware.Logger())
 e.Use(middleware.Recover())
 e.Use(middleware.CORS())
 e.Use(otelecho.Middleware("${PROJECT}"))

 e.GET("/echo", echoHandler)
 e.GET("/health", healthHandler)

 go func() {
 if err := e.Start(":8080"); err != nil && err != http.ErrServerClosed {
 e.Logger.Fatal("shutting down the server")
 }
 }()

 quit := make(chan os.Signal, 1)
 signal.Notify(quit, os.Interrupt)
 <-quit
 ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
 defer cancel()
 if err := e.Shutdown(ctx); err != nil {
 e.Logger.Fatal(err)
 }
}

Step 3: Infrastructure Template with YAML

Here’s how you define the infrastructure using Pulumi YAML. You can instantly spot one of the features of Pulumi Components. We consume the micorservice-component in YAML without knowing that it was authored in a different language. This is the power of Pulumi Components: write once, consume anywhere.

And we see another feature of Pulumi Components: As it is a first-class citizen, we can embed the component and add additional resources like auto-scaling policies, alarms, and more directly in the YAML file. This allows us to create a complete golden path for deploying Go microservices on AWS ECS with auto-scaling capabilities.

# Pulumi.yaml
name: ${PROJECT}
description: ${DESCRIPTION}
runtime: yaml
packages:
 component-microservice: https://github.com/smithrobs/component-microservice@v1.0.1
template:
 description: A template for deploying a Go microservice on AWS ECS with auto-scaling capabilities.
resources:
 microserviceComponent:
 type: component-microservice:MicroserviceComponent
 properties:
 appPath: ./src
 port: 8080
 containerName: ${PROJECT}
 ecsTarget:
 type: aws:appautoscaling:Target
 name: ecs_target
 properties:
 maxCapacity: 4
 minCapacity: 1
 resourceId: service/${microserviceComponent.clusterName}/${microserviceComponent.serviceName}
 scalableDimension: ecs:service:DesiredCount
 serviceNamespace: ecs
 ecsPolicyUP:
 type: aws:appautoscaling:Policy
 name: ecs_policy_up
 properties:
 name: ecs_policy_up
 policyType: StepScaling
 resourceId: ${ecsTarget.resourceId}
 scalableDimension: ${ecsTarget.scalableDimension}
 serviceNamespace: ${ecsTarget.serviceNamespace}
 stepScalingPolicyConfiguration:
 adjustmentType: ChangeInCapacity
 cooldown: 60
 metricAggregationType: Maximum
 stepAdjustments:
 - metricIntervalUpperBound: 0
 scalingAdjustment: 1
 ecsPolicyDown:
 type: aws:appautoscaling:Policy
 name: ecs_policy_down
 properties:
 name: ecs_policy_down
 policyType: StepScaling
 resourceId: ${ecsTarget.resourceId}
 scalableDimension: ${ecsTarget.scalableDimension}
 serviceNamespace: ${ecsTarget.serviceNamespace}
 stepScalingPolicyConfiguration:
 adjustmentType: ChangeInCapacity
 cooldown: 60
 metricAggregationType: Maximum
 stepAdjustments:
 - metricIntervalLowerBound: 0
 scalingAdjustment: -1
 serviceCPUHighUtilization:
 type: aws:cloudwatch:MetricAlarm
 name: service_cpu_high_utilization
 properties:
 name: service_cpu_high_utilization
 comparisonOperator: GreaterThanOrEqualToThreshold
 evaluationPeriods: 2
 metricName: CPUUtilization
 namespace: AWS/ECS
 period: 60
 statistic: Average
 threshold: 80
 alarmActions:
 - ${ecsPolicyUP.arn}
 dimensions:
 ClusterName: ${microserviceComponent.clusterName}
 ServiceName: ${microserviceComponent.serviceName}
 serviceCPULowUtilization:
 type: aws:cloudwatch:MetricAlarm
 name: service_cpu_low_utilization
 properties:
 name: service_cpu_low_utilization
 comparisonOperator: LessThanOrEqualToThreshold
 evaluationPeriods: 2
 metricName: CPUUtilization
 namespace: AWS/ECS
 period: 60
 statistic: Average
 threshold: 10
 alarmActions:
 - ${ecsPolicyDown.arn}
 dimensions:
 ClusterName: ${microserviceComponent.clusterName}
 ServiceName: ${microserviceComponent.serviceName}

outputs:
 publicUrl: ${microserviceComponent.publicUrl}

Step 4: Documentation and Examples

Add a README.md file to your template directory to provide clear instructions on how to use the template, including prerequisites, configuration options, and example commands:

# Go Microservice Golden Path

A **golden path** template that gets your Go microservice from code to production on AWS in minutes, not weeks.

## What You Get

This golden path provides everything your development team needs to deploy production-ready Go microservices:

✅ **Production-ready Go microservice** with Echo framework and OpenTelemetry tracing
✅ **AWS infrastructure that scales** - ECS with auto-scaling from 1-4 instances
✅ **Security by default** - Hardened containers, IAM roles, security groups
✅ **Monitoring built-in** - Health checks, load balancer monitoring, CloudWatch alarms
✅ **One-command deployment** - `pulumi up` handles everything
✅ **No AWS expertise required** - Complex ECS setup abstracted away

## For Development Teams: What to Expect

### Your Experience

**Day 1: Getting Started**

- Clone this repo, run `pulumi up`
- Your service is live on AWS in 5-10 minutes
- Public URL provided automatically - no manual setup needed

**Day 2-N: Development Workflow**

- Write your Go code in `microservice/main.go`
- Test locally with `go run main.go`
- Deploy changes with `pulumi up`
- AWS automatically rebuilds and redeploys your container

**Production Operations**

- Service automatically scales with CPU load (80% up, 10% down)
- Health checks ensure unhealthy containers are replaced
- Load balancer distributes traffic across healthy instances
- Distributed tracing helps debug issues across services

### What's Handled For You

You **don't** need to learn or configure:

- ECS clusters, services, and task definitions
- Application Load Balancers and target groups
- Auto-scaling policies and CloudWatch alarms
- Security groups and IAM roles
- Container registries and image building
- Health check configuration

You **do** focus on:

- Writing your Go application logic
- Adding your business endpoints
- Testing your service locally
- Deploying with confidence

Making Templates Available in Pulumi IDP

Publishing your template to Pulumi IDP enables true self-service. Head to Settings → Integrations → Organization Template Sources and add your template repository.

Once published, developers can discover and deploy it directly from the IDP interface.

Now developers can deploy through multiple interfaces:

CLI Deployment:

pulumi new https://github.com/myorg/go-microservice-boilerplate

No-Code Deployment: Navigate to Pulumi IDP → Templates → Deploy with Pulumi → Configure and launch

Pulumi Deployment: Navigate to Pulumi IDP → Templates → Deploy with Pulumi → Configure and launch

Build golden paths with Pulumi

Package your infrastructure patterns into reusable components and templates, publish them to a private registry, and let teams deploy production-ready stacks in minutes.

Get started

Best Practices for Reusable Infrastructure Components and Templates

Well-designed components and templates are the foundation of scalable, self-service infrastructure. These best practices ensure your abstractions are maintainable, discoverable, and production-ready.

1. Design for Day 2 Operations from Day 1

Think beyond deployment. A golden path must also support ongoing operations:

How will teams safely update their infrastructure?
What happens during scaling events?
How do you handle disaster recovery?
What metrics and logs are needed for troubleshooting?

2. Expose Complexity Progressively

Provide sensible defaults that work, while enabling customization for advanced use cases:

interface ComponentArgs {
 // Required - what users must provide
 appName: string;

 // Common customizations with good defaults
 instanceType?: string; // default: "t3.micro"
 replicas?: number; // default: 2

 // Advanced options for power users
 networkConfig?: NetworkConfig;
 securityPolicies?: SecurityPolicy[];
 customMetrics?: MetricDefinition[];
}

4. Use Semantic Versioning Everywhere

Clear versioning for both components and templates indicate stability and reliability:

Major versions (1.0.0 → 2.0.0): Breaking changes
Minor versions (1.0.0 → 1.1.0): New features, backward compatible
Patch versions (1.0.0 → 1.0.1): Bug fixes

This lets teams adopt updates with confidence and control.

5. Test Your Abstractions

Don’t ship black boxes. Create automated tests to validate key functionality and resource creation. Focus on:

Smoke tests that validate resource existence
Output validation to ensure correctness
Integration tests to confirm end-to-end behavior

import { expect } from "chai";
import * as pulumi from "@pulumi/pulumi";

describe("MicroserviceComponent", () => {
 it("should create required resources", async () => {
 const component = new MicroserviceComponent("test", {
 appPath: "./test-app",
 port: 3000
 });

 const resources = await pulumi.runtime.allResources();
 expect(resources).to.include("aws:ecs/cluster:Cluster");
 expect(resources).to.include("aws:ecs/service:Service");
 expect(resources).to.include("aws:lb/loadBalancer:LoadBalancer");
 });
});

How to Measure Golden Path Success

Golden paths aren’t complete until they deliver measurable value. Use these KPIs to assess performance and drive iteration:

Adoption Metrics

Template usage rate: Percentage of new projects using golden paths
Component reuse: Number of stacks consuming shared components
Time to first deployment: Deployment time and frequency from code to production

Quality Metrics

Security compliance rate: Percentage of deployments passing security policies
Stability: Deployment incident frequency
Mean time to recovery (MTTR): Time to recover from production issues

Developer Experience Metrics

Developer satisfaction: Survey teams about their platform experience
Support ticket volume: Ticket volume related to infrastructure
Contribution: Number of PRs or issues submitted to platform templates

Real-World Results: Success Stories

Organizations using golden paths report significant improvements in speed and reliability:

Snowflake reduced deployment time from 1.5 weeks to less than a day
Mercedes-Benz decreased infrastructure provisioning from weeks to minutes
Starburst Data cut deployment time from 2 weeks to 3 hours

These results show that golden paths are not just developer tools, they’re a competitive advantage. To learn more, download the whitepaper: The Golden Path to Cloud Success: Your IDP Roadmap.

Common Pitfalls and How to Avoid Them

Pitfall 1: Over-Abstraction

Problem: Creating components so abstract they’re unusable Solution: Start with concrete use cases, then generalize based on actual patterns

Pitfall 2: Insufficient Escape Hatches

Problem: Golden paths become golden cages Solution: Always provide ways to extend or override default behavior

Pitfall 3: Poor Versioning Strategy

Problem: Breaking changes without migration paths Solution: Maintain backward compatibility and provide clear upgrade guides

Pitfall 4: Lack of Ownership

Problem: Templates become orphaned and outdated Solution: Assign clear ownership and establish maintenance schedules

The Future of Golden Paths: AI and Beyond

The future of golden paths is intelligent, cross-cloud, and fully integrated with modern workflows.

AI-Enhanced Templates

Imagine templates that adapt based on your application’s actual behavior, automatically tuning resources and configurations for optimal performance and cost.

Cross-Cloud Portability

Components that abstract not just resources but entire cloud providers, enabling true multi-cloud golden paths.

GitOps-Native Workflows

Templates that include not just infrastructure but complete GitOps pipelines, from code commit to production deployment.

Your First Steps to Golden Paths

Ready to get started? Here’s your action plan:

Audit Current Patterns: Document the infrastructure patterns your teams use most frequently
Build one reusable component: Pick your most common pattern and build a reusable component
Create Your First Template: Build a complete golden path for one project type
Gather Feedback: Deploy with a pilot team and iterate based on their experience
Scale Gradually: Expand your component library and template catalog based on demand
Measure and Iterate: Track adoption and continuously improve based on metrics

Conclusion: From Fragmentation to Flow

Golden paths aren’t about restricting creativity or enforcing rigid standards. They’re about removing friction from the development process, encoding expertise into reusable patterns, and empowering developers to move at the speed of business.

By building a library of components and templates, you transform your Internal Developer Platform from a collection of tools into a force multiplier for your entire engineering organization. You give developers the gift of not having to solve solved problems, while maintaining the flexibility to innovate where it matters.

Start small. Solve one problem well. Expand from there. With the right foundation, your developer platform will evolve into a system of self-service, speed, and stability.

Ready to Build Your Golden Paths?

Code examples: Explore golden path examples on GitHub
Build with Components: Pulumi Components documentation
Enable Self-Service: Pulumi IDP documentation
Join the Community: Connect with platform engineers in our Pulumi Slack

Next in our series: Policy as Code for Safer IDPs. You will learn how to add automated guardrails that make sure every deployment meets your security and compliance standards without slowing down development.

How to Build an Internal Developer Platform: Strategy, Best Practices, and Self-Service Infrastructure

Mitch Gerdisch — Thu, 14 Aug 2025 05:13:02 +0200

Welcome to the first post in our IDP Best Practices series. In this guide, we’ll walk through the strategic foundations for designing an Internal Developer Platform that empowers developers without sacrificing governance, security, or operational control.

At Pulumi, we’ve worked with hundreds of teams facing the same core challenge: How do you give developers the infrastructure access they need, while maintaining the governance and security your organization requires?

That tension is at the heart of every IDP conversation. Teams want to move faster and innovate, but also need to stay compliant, control costs, and maintain operational stability.

The good news? You can do both, with a clear strategy and the right approach. This series shares proven best practices for designing, building, and scaling IDPs using Pulumi.

These lessons come from real-world implementations across industries and company sizes—and are built to grow with you.

Understanding the Platform Engineering Layers in Your Internal Developer Platform

When we work with customers on their platform strategy, we often start by referring to the “platform engineering layer cake.” Here is a quick walkthrough of each layer and how Pulumi IDP constructs fit this layer cake approach:

Layer 1: Infrastructure Layer - This is your raw cloud resources: VMs, databases, networks, storage. These are the fundamental building blocks that exist in AWS, Azure, GCP, and other cloud providers.

Layer 2: Platform Layer - This is where Pulumi Components live. Components take those raw infrastructure resources and package them into higher-level abstractions that encapsulate best practices, security policies, and organizational standards. For example, instead of manually configuring 15 different AWS resources to create a secure web application, you create a component that handles all that complexity and exposes just the configuration options that matter to your developers.

Layer 3: Developer Experience Layer - This is where Pulumi Templates and the Private Registry come into play. Templates provide ready-to-deploy patterns that developers can customize, while the private registry makes everything discoverable and manageable at scale.

In our workshop, we focused specifically on those top two layers because that’s where the transformation happens. This is where you turn raw infrastructure into something developers can actually use productively without becoming infrastructure experts themselves.

Here’s how this maps to what you’re actually building:

Infrastructure Layer → Raw Cloud Resources

Your foundation is still the same: AWS EC2 instances, Azure Virtual Networks, GCP Cloud Functions. Pulumi’s providers give you access to these resources using real programming languages instead of YAML or proprietary DSLs.

Platform Layer → Pulumi Components

This is where you create your reusable building blocks. A component might encapsulate:

A complete microservice infrastructure pattern (load balancer, auto-scaling group, database, monitoring)
A secure data pipeline (storage, processing, access controls, encryption)
A standardized Kubernetes application deployment (ingress, service, deployment, secrets management)

Components are written once in any supported language (TypeScript, Python, Go, .NET, Java) but can be consumed by teams using any of these languages. This is crucial. Your platform team might write components in Go, but your application teams can consume them in Python or TypeScript.

Developer Experience Layer → Templates + Private Registry

Templates are complete, deployable projects that use your components. They provide:

Getting started patterns: “Deploy a new microservice”
Reference architectures: “Deploy a three-tier web application”
Environment patterns: “Set up a complete dev/staging/prod pipeline”

The Private Registry is your distribution mechanism. It’s where components and templates become discoverable, versioned, and manageable. Developers don’t need to hunt through Git repositories or Slack channels to find the right building blocks. Everything is catalogued, documented, and accessible through familiar package management workflows.

IDP Example: Building a Web Application Platform Step-by-Step

Let’s make this concrete with an example from our workshops. Say you want to enable teams to deploy secure web applications:

Infrastructure Layer: You need an Application Load Balancer, EC2 Auto Scaling Group, RDS database, VPC with proper subnets, Security Groups, IAM roles, CloudWatch monitoring, and S3 bucket for static assets. That’s about 20+ individual AWS resources that need to be configured correctly and securely.

Platform Layer (Components): You create a WebApplication component that encapsulates all this complexity. Your component exposes simple configuration options like:

const app = new WebApplication("my-app", {
 instanceType: "t3.medium",
 minSize: 2,
 maxSize: 10,
 databaseSize: "small",
 environment: "production"
});

Behind the scenes, your component handles all the security configurations, networking setup, monitoring, and best practices. You publish this component to your private registry.

Developer Experience Layer (Templates): You create templates like:

web-app-starter: A simple web application using your WebApplication component
microservice-template: API service with database and monitoring
full-stack-template: Frontend, backend, and database in a complete environment

Developers can now run pulumi new web-app-starter and get a production-ready, secure web application in minutes instead of weeks. The template uses your component, which handles all the underlying infrastructure complexity.

This layered approach solves the core challenge we see every platform team struggling with: how do you provide developers with the infrastructure they need without creating operational bottlenecks or compromising security standards.

Why Internal Developer Platforms Matter in 2025 and Beyond

While DevOps brought us incredible advances in how we ship and maintain software, we’ve watched many organizations struggle with the unintended consequences: tool sprawl, Day 2 operational pain, and developers who are drowning in infrastructure complexity.

That’s why platform teams have emerged. You exist to solve these challenges by building tools and workflows that enable your internal customers (the developers) to provision infrastructure and deploy software without getting blocked. But here’s the thing I’ve learned from working with hundreds of platform teams: success isn’t just about the tools you choose. It’s about the strategy behind how you implement them.

5 Core Components of a Successful Internal Developer Platform

Through years of working with platform teams and analyzing what separates successful IDPs from those that struggle to gain adoption, we’ve identified five essential components that every platform strategy needs. These aren’t theoretical concepts. They’re battle-tested patterns that work across organizations of all sizes and industries.

1. Abstractions

Let me be clear about something: abstractions aren’t about dumbing down infrastructure for developers. The abstraction is really where you’re hiding the complexity of the underlying infrastructure from your end users, but you’re doing it intentionally to provide appropriate interfaces for different personas in your organization.

We’ve seen too many platform teams get this wrong by either over-abstracting (creating black boxes that developers can’t customize) or under-abstracting (exposing too much complexity). The sweet spot is creating component resources that encapsulate your infrastructure patterns and best practices into reusable building blocks that can be consumed across different programming languages and deployment scenarios, while still providing escape hatches when needed.

2. Blueprints

Blueprints are your templatized, well-architected patterns that developers can use to bootstrap their infrastructure. But here’s what I want you to understand about blueprints: they’re not just starting points that you throw over the wall to developers. They’re carefully designed patterns that embody your organizational best practices and architectural decisions.

When we see successful platform teams, their templates serve as both accelerators and guardrails. They give developers a fast path to production-ready infrastructure while ensuring that everything they deploy follows your organization’s standards. That’s the power of good blueprint design: speed and compliance working together, not against each other.

3. Workflows

Here’s where many platform teams get trapped: they try to build one workflow to rule them all. But in our experience working with organizations across every industry, you need to support multiple consumption patterns because you have multiple types of users. You need:

No-code workflows for users who want point-and-click deployment (think product managers or junior developers who just need to spin up a database)
Low-code workflows using tools like Pulumi YAML for configuration-driven infrastructure (perfect for developers who understand infrastructure but don’t want to write Go or TypeScript)
Full-code workflows for developers who need maximum flexibility and want to leverage the full power of general-purpose programming languages

The key insight? These aren’t three different platforms. They’re three different interfaces to the same underlying components and templates.

4. Security guardrails

This is probably the most important distinction we can share with you: you want to have guardrails, not gates. Too many platform teams create approval processes and manual checkpoints that completely undermine the self-service promise they’re trying to deliver.

Guardrails are different. They start with the best practices and security built into the reusable abstractions. But even the most secure solutions need a safety net. This safety net is policy-as-code that runs automatically, preventing violations before deployment happens. When we work with customers on implementing Pulumi CrossGuard, we’re not creating new friction. We’re embedding security, compliance, and cost controls directly into the deployment process. The developer gets immediate feedback, and you get the assurance that nothing goes to production without meeting your standards.

5. Self-service

Everything we’ve talked about so far builds toward this moment: enabling developers to provision and manage infrastructure independently. But self-service isn’t just about providing a UI. It’s about carefully orchestrating all the other components to create experiences that are both powerful and intuitive.

When we walked through the demo in our workshop, what you saw was the culmination of thoughtful platform design. Developers can discover available services, deploy infrastructure through whatever interface makes sense for them, and manage their resources over time, all without requiring tickets or manual intervention from platform teams. That’s the promise of a well-designed IDP: developer autonomy without operational chaos.

Put your IDP strategy into practice

Build reusable components, golden paths, and self-service workflows on Pulumi, and give developers autonomy while you keep governance and costs in check.

Get started

How to Implement Your IDP Strategy

Supporting Different Personas in Your Internal Developer Platform Design

One of the biggest mistakes we see platform teams make: they try to build for their most sophisticated users first. But here’s what I’ve learned from working with hundreds of organizations: you need to think about all your personas from day one.

In most organizations, you’re serving:

Infrastructure engineers who want full control and flexibility (they’ll use the full-code workflows)
Application developers who need infrastructure but prefer higher-level abstractions (perfect for low-code approaches)
Non-technical users who want simple, point-and-click deployment (your no-code users)

The breakthrough insight that separates successful platform teams from struggling ones is this: all these personas can be served by the same underlying components and templates, just consumed through different interfaces. You don’t need to build three different platforms—you need to build one platform with three different consumption models.

Why a Private Registry Is Key to IDP Adoption

Here’s something we always tell platform teams: if you don’t solve the discoverability problem, your beautiful components and templates will sit unused in Git repositories where no one can find them. That’s why the private registry isn’t just a nice-to-have. It’s the foundation that makes everything else work.

When you establish a private registry as your source of truth for components, templates, providers, and policies, you’re solving two critical problems at once. First, discoverability: developers can actually find and explore what’s available. Second, lifecycle management: you can see where each package is being used, track version drift, and understand the impact of changes before you make them.

The workflow is beautifully simple: you publish standardized building blocks with a single pulumi publish command, and developers discover and consume these assets through familiar package management workflows they already know.

Aligning Your IDP to Organizational Context and Services

One pattern we see in every successful IDP implementation is that the platform reflects how the organization actually works. Your developers don’t think in terms of individual stacks or resources. They think in terms of services, applications, and business functionality.

That’s why we built Pulumi Services - to let teams logically group stacks and ESC environments in ways that make sense to your organization. Pulumi Services improve organization by helping your team model infrastructure in a way that’s familiar - simplifying Day 2 operations by providing additional context like links to relevant dashboards, Slack channels and more. When something breaks at 2 AM, your on-call engineer needs to understand dependencies, track usage, and manage infrastructure in the context of the business services that are actually impacted.

How to Measure the Success of Your Internal Developer Platform

Here’s something that might surprise you: the most successful platform teams we work with don’t measure their success by how many cool features they’ve built. They measure success by the business outcomes they’re driving:

Reduced time-to-market through faster infrastructure provisioning (because at the end of the day, your platform exists to help the business move faster)
Improved developer productivity by eliminating infrastructure bottlenecks (happy developers ship more features)
Enhanced security posture through consistent policy enforcement (security that doesn’t slow people down is security that actually gets used)
Lower operational overhead through standardization and automation (because your platform team’s time is better spent on innovation than toil)

The numbers speak for themselves. We’ve worked with customers like Snowflake who reduced deployment times from one and a half weeks to less than a day. Starburst Data cut their infrastructure deployments from two weeks to just three hours. These aren’t vanity metrics. They’re business transformations.

IDP Strategy: 5 Key Steps to Build a Strong Foundation

If you’re just starting your IDP journey, here’s the roadmap we give to every platform team we work with:

Start with user research: Before you write a single line of code, go talk to your developers. Understand your different personas and their actual needs, not what you think they need. Do your users like to work from the command line? Do they prefer interacting with a UI?
Identify common patterns: Look for infrastructure patterns that get repeated across teams. These are your goldmine—the patterns that, once abstracted and templatized, will provide immediate value.
Begin with components: Build reusable infrastructure building blocks before you worry about fancy UIs or workflow orchestration. You need solid foundations before you can build the house.
Implement progressive disclosure: Give people simple interfaces for common use cases, but always provide escape hatches for complex scenarios. The moment your abstraction becomes a prison, developers will route around it.
Measure and iterate: Track adoption and gather feedback religiously. Your platform is only as good as its adoption rate, and adoption only happens when you’re solving real problems for real people.

The Future of IDPs and Platform Engineering

Trends come and go, but platform engineering is here to stay. It’s not just another tool or methodology. This is how modern organizations scale infrastructure to match the speed of innovation. The companies that win are the ones that treat their Internal Developer Platform (IDP) as a strategic foundation, not just a tool.

Your job isn’t just to manage infrastructure and choose the right tools - it’s to enable developers to move faster while keeping governance, security, and costs under control. Get that balance right, and your platform becomes a force multiplier across your entire engineering department.

The fastest way to get there? Focus on these five essential components:

Abstractions
Blueprints
Workflows
Security guardrails
Self-service

Together, they turn your IDP into more than just infrastructure automation, they create a platform that developers trust and your business relies on.

Ready to Build a Successful IDP that Scales?

Pulumi makes it easy to go from static IaC to dynamic, self-service infrastructure with real programming languages and built-in guardrails.

Explore Pulumi IDP or sign up for free to get started today.

Backstage vs Pulumi IDP: Why Infrastructure-First Wins!

Engin Diri — Wed, 13 Aug 2025 00:00:00 +0000

Developers are losing days every month to infrastructure bottlenecks, compliance hurdles, and inconsistent environments. Platform engineering promised to fix that, yet too many platforms fail before they deliver real impact.

In this comparison of Backstage vs Pulumi IDP, we’ll explore why choosing the right architectural approach matters more than the tool itself.

Quick comparison: Backstage vs Pulumi IDP

Backstage is an open-source developer portal framework from Spotify that focuses on service catalogs and documentation. It requires significant setup time (12-18 months) and ongoing maintenance (3-5 FTEs).

Pulumi IDP is an infrastructure-first internal developer platform that embeds governance, cost control, and security from day one. It can be deployed in hours with minimal ongoing maintenance.

The problem isn’t whether you choose Backstage or Pulumi IDP. The real question is: Will your platform treat infrastructure as a first-class concern, or as an afterthought?

According to Gartner, 80% of large enterprises will deploy internal developer platforms within the next two years. But without a strong infrastructure foundation, those platforms risk becoming expensive, hard-to-maintain bottlenecks instead of accelerators.

Why platform engineering efforts fail: Common mistakes

Platform teams consistently make four critical mistakes:

Neglecting Infrastructure Foundations
Over-focusing on the developer portal experience while letting infrastructure governance and scalability lag behind.
Ignoring Cost Efficiency
Prioritizing productivity at the expense of cloud cost management.
Example: One company considered giving every developer their own cluster, a scaling nightmare both technically and financially.
Treating Security & Governance as Add-Ons
Bolting on compliance after launch leads to constant firefighting and security debt.
Building for Yesterday’s Architecture
Designing general-purpose platforms that can’t keep up with Kubernetes-native workloads.

Portal-first vs infrastructure-first platform engineering

There are two dominant approaches:

Portal-First typified by Backstage, which starts with developer experience and service catalogs.
Infrastructure-First exemplified by Pulumi IDP, which builds the platform foundation around scalable, policy-driven infrastructure.

Platform Engineering Feature	Backstage (Portal-First)	Pulumi IDP (Infrastructure-First)
Primary Focus	Developer portal & catalog	Scalable, policy-driven infrastructure
Setup Time	12-18 months	Hours
Maintenance	3-5 FTE ongoing	Minimal ongoing overhead
Infra Provisioning	Requires separate tooling	Built-in
Cost Awareness	Depends on infra layer	Native
Security & Governance	Add-on	Built-in from day one

Backstage: The portal-first platform engineering approach

Backstage is an open-source framework for building developer portals. Originally built by Spotify, it excels at unifying tools, services, and documentation.

Strengths

Mature Ecosystem with over 1,200 integrations and a large community.
Excellent Service Catalog providing centralized metadata, ownership, and discovery.
Docs-as-Code that keeps documentation close to the codebase.
Self-Service Templates that standardize scaffolding for new projects.

Challenges

High Maintenance Overhead requiring 3-5 FTEs just to keep it running in large orgs.
Complex Setup with Yarn version headaches, TypeScript complexity, plugin upkeep.
Infra as an Afterthought that lacks built-in provisioning, often pushing infra concerns down the road.
Hidden Costs resulting in multi-million dollar TCO over three years for a 300-dev team when factoring in staffing and infra complexity.

Pulumi IDP: the infrastructure-first platform engineering approach

Pulumi IDP brings Infrastructure Platform Engineering (IPE) to the forefront by embedding scalable, secure, cost-aware infrastructure into the platform from day one.

Strengths

Infrastructure as a First-Class Concern with built-in multitenancy, isolation, and governance.
Efficiency & Cost Awareness through golden paths and reusable components optimized from the start.
Policy-Driven by Design that enforces security, compliance, and cost policies automatically.
Cloud-Native Ready ideal for ephemeral workloads, Kubernetes, and multi-cloud.

Considerations

Newer Platform with a smaller plugin ecosystem than Backstage’s.
Infra-Heavy Focus that’s best fit for infra-intensive orgs; overpowered if you only need a catalog and docs.

Real-world platform engineering outcomes: Backstage vs Pulumi IDP

Pulumi IDP Adoption: CLEAR leveraged Pulumi IDP’s prebuilt capabilities instead of building infra automation from scratch, speeding up delivery while keeping governance tight.

Backstage vs Pulumi IDP decision guide

When to choose Backstage

Choose Backstage for platform engineering if you:

Have a dedicated platform team with strong React/TypeScript skills
Primarily need service cataloging and documentation
Already have robust infrastructure provisioning and governance tools
Have 18+ months and budget for ongoing maintenance (3-5 FTEs)

When to choose Pulumi IDP

Choose Pulumi IDP for platform engineering if you:

Need infrastructure governance from day one
Want cost management and policy automation built in
Must scale efficiently across Kubernetes and multi-cloud
Prefer to avoid building everything from scratch
Need to deploy quickly (hours vs months)

Build your platform on infrastructure

Pulumi IDP embeds governance, cost control, and security into your platform from day one, giving your teams self-service golden paths with policy enforced automatically.

Get started

The shift to infrastructure platform engineering (IPE)

Platform engineering isn’t failing because the idea is flawed. It’s failing because too many efforts start with the portal and leave infrastructure for later.

Infrastructure Platform Engineering flips that model, ensuring the platform’s foundation is:

Hybrid approaches are also possible. Pulumi offers a Backstage plugin so teams can pair Backstage’s portal features with Pulumi IDP’s infrastructure power.

Bottom line: infrastructure-first wins for platform engineering

If your platform is slow, costly, or hard to govern, it’s not a developer problem. It’s an infrastructure problem. Treating infrastructure as a first-class concern from day one isn’t optional. It’s the difference between a platform that scales and one that stalls.

Frequently asked questions

What is the main difference between Backstage and Pulumi IDP?

Backstage is a portal-first developer platform that focuses on service catalogs and documentation, requiring separate tools for infrastructure provisioning. Pulumi IDP is an infrastructure-first platform that embeds governance, cost control, and infrastructure automation from the start.

How long does it take to implement Backstage vs Pulumi IDP?

Backstage typically requires 12-18 months for full implementation with ongoing maintenance by 3-5 FTEs. Pulumi IDP can be deployed in hours with minimal ongoing maintenance overhead.

Can I use Backstage and Pulumi IDP together?

Yes, Pulumi offers a Backstage plugin that allows teams to combine Backstage’s portal features with Pulumi IDP’s infrastructure capabilities for a hybrid approach.

What are the costs of running Backstage vs Pulumi IDP?

Backstage’s TCO for a 300-developer team can reach multiple millions over three years when factoring in staffing and infrastructure complexity. Pulumi IDP has lower operational costs due to built-in automation and minimal maintenance requirements.

Which platform is better for Kubernetes environments?

Pulumi IDP is purpose-built for cloud-native and Kubernetes environments with built-in support for ephemeral workloads and multi-cloud deployments. Backstage requires additional tooling for Kubernetes infrastructure management.

Learn more about Pulumi IDP and how infrastructure-first platform engineering avoids common failure patterns.

I Tried Jenkins in 2025 with Pulumi: Here's How It Went

Engin Diri — Tue, 29 Jul 2025 10:35:00 +0200

It’s funny how technology has a way of sneaking back into your life just when you think you’ve moved on for good. Jenkins and I have quite the history. Think of it as that reliable but slightly temperamental friend from your college days who you haven’t seen in years.

A Blast from the Jenkins Past

The last time Jenkins and I were on speaking terms was during my tenure at my former workplace, back when the CI/CD landscape looked very different than it does today. We weren’t just casual acquaintances either. We were deep in the trenches together, orchestrating builds, managing deployments, and occasionally cursing at each other when things went sideways (which, let’s be honest, happened more often than I’d like to admit).

Those were the days when Jenkins was transitioning from its scrappy, plugin-heavy adolescence into something resembling maturity. I was part of a significant modernization effort that would make any DevOps engineer’s heart race: migrating our entire Jenkins infrastructure from Jenkins 1.x to the shiny new Jenkins 2.0. If you’ve never experienced a Jenkins major version upgrade in a production environment, imagine performing heart surgery while the patient is running a marathon. Technically possible, but requiring nerves of steel and copious amounts of coffee.

The migration wasn’t just about version numbers. Jenkins 2.0 brought with it the Pipeline plugin as a first-class citizen, transforming how we thought about continuous integration. Gone were the days of clicking through endless GUI configurations and praying that someone had documented the build process correctly. Instead, we embraced the revolutionary concept of “Pipeline as Code”, storing our build definitions in Jenkinsfile scripts that lived alongside our application code.

But the real game-changer came when we decided to containerize our Jenkins worker nodes using Docker. This was back when Docker was still the new kid on the block, and the idea of running build agents in containers felt both cutting-edge and slightly reckless. We implemented the Docker plugin for Jenkins, which allowed us to spin up fresh, isolated environments for each build. No more “but it works on my machine” syndrome. Every build got its own pristine container, complete with exactly the dependencies it needed.

The transformation was remarkable. What used to take us hours of manual environment setup and dependency management became as simple as defining a Dockerfile. Our build agents went from being precious, hand-crafted snowflakes that required constant care and feeding to becoming ephemeral, reproducible environments that could be destroyed and recreated at will.

Here’s what a typical Jenkinsfile looked like during that era. Notice how we could seamlessly combine different technologies and services, each running in their own containers:

pipeline {
 agent none
 stages {
 stage('Back-end') {
 agent {
 docker { image 'maven:3.9.10-eclipse-temurin-21-alpine' }
 }
 steps {
 sh 'mvn clean compile test'
 junit 'target/surefire-reports/*.xml'
 }
 }
 stage('Front-end') {
 agent {
 docker { image 'node:22.17.0-alpine3.22' }
 }
 steps {
 sh 'npm install && npm test'
 }
 }
 stage('Integration Tests') {
 agent any
 steps {
 script {
 docker.image('postgres:13').withRun('-e POSTGRES_PASSWORD=test') { db ->
 docker.image('redis:7-alpine').withRun() { redis ->
 docker.build('myapp:test').inside("--link ${db.id}:postgres --link ${redis.id}:redis") {
 sh 'npm run integration-tests'
 }
 }
 }
 }
 }
 }
 }
}

This pipeline definition encapsulated everything we’d learned about containerized CI/CD: isolated environments, service orchestration, and the beautiful simplicity of treating infrastructure as disposable. Each stage could use exactly the tools it needed, without worrying about conflicts or environmental drift.

Fast forward to 2025, and here I am, staring at a Jenkins instance again. But this time, there’s a twist: instead of the traditional XML configurations and GUI-heavy setup processes I remember, I’m armed with Pulumi. This tool promises to bring infrastructure-as-code practices to Jenkins deployment itself. The irony isn’t lost on me: using modern infrastructure tooling to deploy a CI/CD platform that helped define what modern infrastructure tooling could become.

So buckle up for this nostalgic yet forward-looking journey. We’re about to find out whether Jenkins has kept up with the times, and more importantly, whether the combination of Jenkins and Pulumi can create something greater than the sum of its parts.

How I’d Set Up Jenkins in 2025: The Kubernetes Way

Fast forward to 2025, and the infrastructure landscape has evolved dramatically. While my old Jenkins setup was running on virtual machines with Docker containers as build agents, the modern approach demands something more cloud-native. Today, if you’re serious about CI/CD at scale, Kubernetes isn’t just an option. It’s practically a requirement.

The beauty of running Jenkins on Kubernetes lies in its perfect marriage of concepts. Jenkins already understood the value of ephemeral build agents, and Kubernetes takes that philosophy to its logical conclusion: everything is ephemeral, everything is disposable, and everything can be recreated on demand. Your Jenkins controller becomes just another workload in your cluster, and your build agents become pods that spin up, do their work, and disappear without a trace.

But here’s where it gets interesting. Instead of clicking through installation wizards or crafting complex shell scripts, I’m going to use Pulumi to define my entire Jenkins infrastructure as code. This isn’t just about following the latest trends. It’s about bringing the same “everything as code” philosophy that made Jenkins 2.0 so compelling to the infrastructure layer itself.

The Pulumi Advantage

Why Pulumi for this task? After years of wrestling with YAML manifests and Helm charts, I’ve come to appreciate Pulumi’s approach to infrastructure as code. With Pulumi, I can leverage real programming languages (in this case, Python) to define my infrastructure, complete with loops, conditionals, and all the abstractions that make complex deployments manageable.

Let me show you what a modern Jenkins deployment looks like when we combine Pulumi’s infrastructure-as-code capabilities with Kubernetes’ orchestration power. This isn’t just theoretical. This is production-ready code that you could deploy in any Kubernetes cluster.

Setting Up Our Local Kubernetes Cluster with Kind

Before we dive into the Pulumi code, we need a Kubernetes cluster to deploy to. For this demonstration, I’m using Kind (Kubernetes in Docker), which creates a local Kubernetes cluster using Docker containers as nodes. It’s perfect for development, testing, and learning scenarios like this.

Kind has become the de facto standard for local Kubernetes development because it’s lightweight, fast, and provides a genuine multi-node Kubernetes experience without the overhead of virtual machines. Plus, it’s what the Kubernetes project itself uses for testing, so you know it’s reliable.

Creating the Cluster:

Once Kind is installed, creating a cluster is straightforward:

kind create cluster --name jenkins-cluster

This command spins up a single-node Kubernetes cluster running in a Docker container. The beauty of Kind is that it configures your kubectl context automatically, so you’re immediately ready to start deploying workloads.

You can verify your cluster is running:

kubectl cluster-info --context kind-jenkins-cluster
kubectl get nodes

For our Jenkins deployment, the default single-node cluster is sufficient, but Kind also supports multi-node clusters if you need to test more complex scenarios. The cluster will persist until you explicitly delete it with

kind delete cluster --name jenkins-cluster

Now that we have our Kubernetes cluster ready, let’s deploy Jenkins using Pulumi.

import pulumi
import pulumi_kubernetes as k8s
from pulumi_kubernetes.helm.v3 import Chart, ChartOpts, FetchOpts

# Configuration - making it parameterizable
config = pulumi.Config()
namespace_name = config.get("namespace", "jenkins")
storage_class = config.get("storageClass", "standard") # Kind uses 'standard' by default
jenkins_admin_password = config.require_secret("adminPassword")

# Create a dedicated namespace for Jenkins
jenkins_namespace = k8s.core.v1.Namespace(
 "jenkins-namespace",
 metadata=k8s.meta.v1.ObjectMetaArgs(
 name=namespace_name,
 labels={"app": "jenkins", "managed-by": "pulumi"}
 )
)

# Create persistent storage for Jenkins
jenkins_pvc = k8s.core.v1.PersistentVolumeClaim(
 "jenkins-pvc",
 metadata=k8s.meta.v1.ObjectMetaArgs(
 name="jenkins-pv-claim",
 namespace=jenkins_namespace.metadata.name,
 ),
 spec=k8s.core.v1.PersistentVolumeClaimSpecArgs(
 access_modes=["ReadWriteOnce"],
 storage_class_name=storage_class,
 resources=k8s.core.v1.ResourceRequirementsArgs(
 requests={"storage": "20Gi"}
 )
 )
)

# Create a service account for Jenkins with proper RBAC
jenkins_service_account = k8s.core.v1.ServiceAccount(
 "jenkins-service-account",
 metadata=k8s.meta.v1.ObjectMetaArgs(
 name="jenkins",
 namespace=jenkins_namespace.metadata.name,
 )
)

# Create cluster role for Jenkins to manage pods (for Kubernetes plugin)
jenkins_cluster_role = k8s.rbac.v1.ClusterRole(
 "jenkins-cluster-role",
 metadata=k8s.meta.v1.ObjectMetaArgs(
 name="jenkins-cluster-role",
 ),
 rules=[
 k8s.rbac.v1.PolicyRuleArgs(
 api_groups=[""],
 resources=["pods", "pods/exec", "pods/log", "persistentvolumeclaims"],
 verbs=["get", "list", "watch", "create", "update", "patch", "delete"]
 ),
 k8s.rbac.v1.PolicyRuleArgs(
 api_groups=["apps"],
 resources=["deployments", "replicasets"],
 verbs=["get", "list", "watch", "create", "update", "patch", "delete"]
 )
 ]
)

# Bind the cluster role to the service account
jenkins_cluster_role_binding = k8s.rbac.v1.ClusterRoleBinding(
 "jenkins-cluster-role-binding",
 metadata=k8s.meta.v1.ObjectMetaArgs(
 name="jenkins-cluster-role-binding",
 ),
 role_ref=k8s.rbac.v1.RoleRefArgs(
 api_group="rbac.authorization.k8s.io",
 kind="ClusterRole",
 name=jenkins_cluster_role.metadata.name
 ),
 subjects=[
 k8s.rbac.v1.SubjectArgs(
 kind="ServiceAccount",
 name=jenkins_service_account.metadata.name,
 namespace=jenkins_namespace.metadata.name
 )
 ]
)

# Create a secret for Jenkins admin credentials
jenkins_admin_secret = k8s.core.v1.Secret(
 "jenkins-admin-secret",
 metadata=k8s.meta.v1.ObjectMetaArgs(
 name="jenkins-admin-secret",
 namespace=jenkins_namespace.metadata.name,
 ),
 string_data={
 "jenkins-admin-user": "admin",
 "jenkins-admin-password": jenkins_admin_password,
 }
)

# Deploy Jenkins using the official Helm chart from https://charts.jenkins.io
jenkins_chart = Chart(
 "jenkins",
 ChartOpts(
 chart="jenkins",
 version="5.8.72", # Using latest stable version
 namespace=jenkins_namespace.metadata.name,
 fetch_opts=FetchOpts(
 repo="https://charts.jenkins.io" # Official Jenkins Helm repository
 ),
 values={
 "controller": {
 "admin": {
 "existingSecret": jenkins_admin_secret.metadata.name,
 "userKey": "jenkins-admin-user",
 "passwordKey": "jenkins-admin-password",
 },
 "resources": {
 "requests": {
 "cpu": "500m",
 "memory": "1Gi"
 },
 "limits": {
 "cpu": "2000m",
 "memory": "4Gi"
 }
 },
 "installPlugins": [
 "kubernetes",
 "workflow-aggregator",
 "git",
 "configuration-as-code",
 "blueocean" # Modern Jenkins UI
 ],
 "JCasC": { # Jenkins Configuration as Code
 "enabled": True,
 "defaultConfig": True,
 "configScripts": {
 "kubernetes-config": '''
 jenkins:
 clouds:
 - kubernetes:
 name: "kubernetes"
 serverUrl: "https://kubernetes.default:443"
 namespace: "''' + namespace_name + '''"
 credentialsId: ""
 jenkinsTunnel: "jenkins-agent:50000"
 jenkinsUrl: "http://jenkins:8080"
 maxRequestsPerHostStr: 64
 retentionTimeout: 5
 connectTimeout: 10
 templates:
 - name: "pulumi-agent"
 namespace: "''' + namespace_name + '''"
 label: "pulumi"
 nodeUsageMode: NORMAL
 containers:
 - name: "pulumi"
 image: "pulumi/pulumi:latest"
 alwaysPullImage: true
 workingDir: "/home/jenkins/agent"
 command: ""
 args: ""
 resourceRequestCpu: "500m"
 resourceRequestMemory: "1Gi"
 resourceLimitCpu: "2000m"
 resourceLimitMemory: "4Gi"
 serviceAccount: "jenkins"
 - name: "default"
 namespace: "''' + namespace_name + '''"
 label: "jenkins-agent"
 nodeUsageMode: NORMAL
 containers:
 - name: "jnlp"
 image: "jenkins/inbound-agent:latest"
 alwaysPullImage: true
 workingDir: "/home/jenkins/agent"
 command: ""
 args: ""
 resourceRequestCpu: "500m"
 resourceRequestMemory: "512Mi"
 resourceLimitCpu: "1000m"
 resourceLimitMemory: "1Gi"
 '''
 }
 }
 },
 "serviceAccount": {
 "create": False,
 "name": jenkins_service_account.metadata.name
 },
 "persistence": {
 "existingClaim": jenkins_pvc.metadata.name
 },
 "agent": {
 "enabled": True,
 "namespace": jenkins_namespace.metadata.name
 }
 }
 )
)

# Export useful information
pulumi.export("namespace", jenkins_namespace.metadata.name)
pulumi.export("jenkins_url", "http://jenkins.{}.svc.cluster.local:8080".format(namespace_name))
pulumi.export("admin_user", "admin")

In the browser, you can access Jenkins at http://localhost:8080 (or the appropriate URL for your cluster).

What Makes This Approach Special

This Pulumi program does several things that would be much more cumbersome with traditional approaches:

Infrastructure as Code: Every aspect of the Jenkins deployment is version-controlled and reproducible. No more “it works on my machine” scenarios with infrastructure.

Kubernetes-Native: The deployment leverages Kubernetes’ native capabilities for service discovery, persistent storage, and RBAC, making Jenkins a first-class citizen in your cluster.

Configuration as Code: Using Jenkins Configuration as Code (JCasC), the Jenkins instance comes pre-configured with the Kubernetes plugin and agent templates, eliminating manual setup steps.

Security by Default: Proper RBAC configurations ensure Jenkins has exactly the permissions it needs, no more and no less.

Scalable Agent Model: Build agents are ephemeral pods that scale based on demand, just like the containerized agents we used years ago, but with Kubernetes orchestration.

The beauty of this approach is that it combines the lessons learned from our Docker-based Jenkins setup with modern cloud-native practices. Agents are still containerized and ephemeral, but now they’re managed by Kubernetes rather than the Docker plugin. Storage is persistent but managed by Kubernetes PVCs. And everything is defined as code using a real programming language rather than YAML.

This setup gives you a production-ready Jenkins instance that can handle serious workloads while being completely reproducible and maintainable. But more importantly, it sets the foundation for the kind of advanced CI/CD practices that modern teams need. These practices go far beyond what we could achieve with our old VM-based approach.

Build your infrastructure pipeline

You can define your Jenkins deployment and the Pulumi programs it runs as code, version every change, and check updates with pulumi preview before they ship. Start a project to wire infrastructure as code into your pipeline.

Get started

Configuring Our First Pipeline to Deploy a Pulumi Program

Now comes the fun part: putting our Kubernetes-based Jenkins setup to work with a real-world scenario. What better way to test our new infrastructure than by creating a pipeline that deploys infrastructure itself? We’re going to configure Jenkins to automatically deploy Pulumi programs using Kubernetes agents, creating a complete infrastructure-as-code CI/CD workflow.

This scenario hits close to home for any modern DevOps team. Gone are the days when infrastructure changes required manual server configurations or clicking through cloud provider consoles. Today’s teams need automated, reliable ways to manage infrastructure changes through the same rigorous processes they use for application code, including peer reviews, automated testing, and controlled deployments.

The Power of Kubernetes Agents

One of the most compelling features of our Kubernetes-based Jenkins setup is its ability to spawn dedicated agent pods for specific workloads. Unlike traditional Jenkins agents that run on persistent VMs or containers, Kubernetes agents are truly ephemeral. They exist only for the duration of a single build, then disappear without a trace.

This approach solves several problems that plagued our old Docker-based setup:

Resource Efficiency: Agents consume cluster resources only when actively building, not sitting idle waiting for work.

Perfect Isolation: Each pipeline run gets its own dedicated environment with no risk of interference from previous builds.

Unlimited Scalability: Need to run 50 concurrent builds? Kubernetes will happily schedule 50 agent pods across your cluster.

Tool-Specific Environments: Different pipelines can use completely different toolchains without conflicts.

For Pulumi deployments, this means we can create specialized agent pods that include exactly the tools we need: the Pulumi CLI, cloud provider SDKs, and any additional dependencies our infrastructure code requires.

A Complete Pulumi Deployment Pipeline

Now let’s create a Jenkinsfile that demonstrates the full power of our setup. This pipeline will:

Check out a Pulumi program from source control
Install dependencies and validate the Pulumi code
Run a preview to show planned changes
Deploy the infrastructure changes (with manual approval for production)
Clean up temporary resources

pipeline {
 agent {
 kubernetes {
 label 'pulumi-deployment'
 yaml """
apiVersion: v1
kind: Pod
spec:
 serviceAccountName: jenkins
 containers:
 - name: pulumi
 image: pulumi/pulumi:latest
 command:
 - sleep
 args:
 - 99d
 env:
 - name: PULUMI_SKIP_UPDATE_CHECK
 value: "true"
 resources:
 requests:
 memory: "1Gi"
 cpu: "500m"
 limits:
 memory: "4Gi"
 cpu: "2000m"
"""
 }
 }

 parameters {
 string(name: 'GIT_URL', defaultValue: 'https://github.com/pulumi/examples', description: 'Git repository URL') // Official Pulumi examples
 string(name: 'PULUMI_FOLDER', defaultValue: 'aws-ts-s3-folder', description: 'Folder containing Pulumi program')
 }

 environment {
 PULUMI_ACCESS_TOKEN = credentials('pulumi-access-token')
 }

 stages {
 stage('Checkout') {
 steps {
 git url: "${params.GIT_URL}",
 branch: "master"
 }
 }

 stage('Install Dependencies') {
 steps {
 container('pulumi') {
 sh '''
 # Navigate to the Pulumi program directory
 cd ${PULUMI_FOLDER}/

 # Install Python dependencies if using Python
 if [ -f "requirements.txt" ]; then
 pip install -r requirements.txt
 fi

 # Install Node.js dependencies if using TypeScript/JavaScript
 if [ -f "package.json" ]; then
 npm install
 fi

 # Install Go dependencies if using Go
 if [ -f "go.mod" ]; then
 go mod download
 fi

 # Login to Pulumi service
 pulumi login

 echo "environment:\n- pulumi-ultimate-gitops/dev" > Pulumi.dev.yaml
 cat Pulumi.dev.yaml
 '''
 }
 }
 }

 stage('Preview Changes') {
 steps {
 container('pulumi') {
 sh '''
 cd ${PULUMI_FOLDER}/
 pulumi stack select dev

 # Generate a preview of changes
 pulumi preview --diff --cwd . > preview.txt

 # Display the preview
 cat preview.txt
 '''
 }

 // Archive the preview for later reference
 archiveArtifacts artifacts: "${params.PULUMI_FOLDER}/preview.txt", allowEmptyArchive: true
 }
 }

 stage('Approval') {
 steps {
 script {
 def preview = readFile "${params.PULUMI_FOLDER}/preview.txt"
 def userInput = input(
 id: 'deploy-approval',
 message: 'Deploy these infrastructure changes?',
 parameters: [
 text(name: 'PREVIEW', defaultValue: preview, description: 'Preview of changes')
 ],
 submitterParameter: 'APPROVER'
 )
 env.APPROVER = userInput
 }
 }
 }

 stage('Deploy Infrastructure') {
 steps {
 container('pulumi') {
 sh '''
 cd ${PULUMI_FOLDER}/
 pulumi stack select dev
 pulumi up --yes --cwd .
 '''
 }
 }
 }
 }

 post {
 always {
 sh "rm -f ${params.PULUMI_FOLDER}/preview.txt"
 }

 success {
 script {
 if (env.APPROVER) {
 echo "Infrastructure deployment completed successfully for stack ${env.PULUMI_STACK}. Approved by: ${env.APPROVER}"
 }
 }
 }

 failure {
 echo "Infrastructure deployment failed for stack ${env.PULUMI_STACK}. Check the build logs for details."
 }
 }
}

Open the Jenkins UI, create a new pipeline job, and paste this Jenkinsfile into the pipeline configuration.

Now we can run this pipeline to deploy a Pulumi program.

Show the preview of changes to be applied:

And wait for manual approval before proceeding with the deployment:

Key Features of This Pipeline

This Jenkinsfile demonstrates several advanced concepts that make modern CI/CD pipelines robust and production-ready:

Kubernetes-Native Agent Pods: The pipeline uses a dedicated Kubernetes pod with a Pulumi-specific container image, providing an isolated environment with all necessary tools pre-installed. The pod exists only for the duration of the build, ensuring perfect isolation between pipeline runs.

Parameterized Deployments: The pipeline accepts parameters for Git repository URL and Pulumi folder path, making it reusable across different Pulumi projects and repositories. This flexibility allows teams to standardize their deployment process while supporting diverse infrastructure codebases.

Multi-Language Support: The dependency installation stage automatically detects and installs dependencies for Python (requirements.txt), Node.js/TypeScript (package.json), and Go (go.mod) projects, making the pipeline adaptable to any Pulumi program regardless of the chosen language.

Manual Approval Gates: All deployments require explicit manual approval, with the complete preview of infrastructure changes displayed to the approver. This ensures that infrastructure modifications are always reviewed by a human before being applied to live environments.

Stack Configuration Management: The pipeline dynamically creates and configures Pulumi stack files, demonstrating how to manage environment-specific configurations programmatically within the CI/CD process.

Comprehensive Artifact Management: Preview outputs are automatically archived for audit trails and debugging, providing a historical record of what changes were planned and approved for each deployment.

You might also like:

Wrap Up

After this deep dive into Jenkins with Pulumi in 2025, I’m reminded of why Jenkins became the workhorse of CI/CD in the first place. It’s reliable, it’s battle-tested, and most importantly, it adapts. Just as we successfully containerized our build agents years ago, Jenkins has seamlessly evolved to thrive in the Kubernetes era.

The transformation from those early days of XML configurations and GUI-heavy setup processes to today’s infrastructure-as-code approach is remarkable. What we’ve built here (a fully automated Jenkins deployment with Kubernetes agents and Pulumi integration) represents the kind of setup that would have been a months-long project back in the day. Now, it’s defined in a single Python file and deployable in minutes.

But here’s the honest truth: Would I choose Jenkins for a greenfield project today? Probably not.

The modern DevOps landscape offers compelling alternatives that were purpose-built for cloud-native environments. GitHub Actions provides seamless integration with source control and requires zero infrastructure management. GitLab CI/CD offers similar convenience with its integrated approach. Cloud-native solutions like Tekton or Argo Workflows were designed from the ground up for Kubernetes, without the historical baggage that Jenkins carries.

These newer platforms offer several advantages over Jenkins:

Zero infrastructure overhead: No need to manage controllers, agents, or storage
Native cloud integration: Built-in support for modern authentication, secrets management, and scaling
Simpler configuration: Pipelines without the complexity of Jenkins’ plugin ecosystem
Better resource efficiency: No persistent controllers consuming resources when idle

However, Jenkins still has its place. If you’re working with legacy systems, have extensive existing Jenkins infrastructure, or need the ultimate flexibility that comes from its massive plugin ecosystem, Jenkins remains a solid choice. The setup we’ve demonstrated proves that Jenkins can absolutely hold its own in modern environments.

The beauty of what we’ve accomplished here isn’t just about Jenkins. It’s about the power of infrastructure-as-code. Whether you choose Jenkins, GitHub Actions, or any other CI/CD platform, the principles remain the same: define your infrastructure declaratively, version control everything, and automate the deployment process.

Jenkins may be the old workhorse, but as we’ve seen, this workhorse still has plenty of life left in it.

Next Steps

This post demonstrated how to modernize Jenkins deployment using Pulumi and Kubernetes. By leveraging infrastructure-as-code principles, we created a completely reproducible CI/CD platform that bridges traditional practices with cloud-native approaches.

If you want to explore more modern CI/CD approaches or dive deeper into infrastructure automation with Pulumi, check out our documentation and try building your own infrastructure-as-code solutions. For more insights on DevOps best practices, explore our guide on IaC best practices.

Try Pulumi for Free

Introducing the Pulumi Puluminaries 2.0 Program

Engin Diri — Mon, 10 Feb 2025 12:15:00 -0800

We are excited to announce the Pulumi Puluminaries 2.0 Program. This is a fresh and revitalized way to celebrate and support Pulumi’s most passionate community members. Pulumi Puluminaries are individuals who demonstrate leadership in the Pulumi ecosystem by sharing best practices, creating valuable content, and helping fellow practitioners succeed.

Before we dive into what is new, we want to recognize and applaud the incredible achievements of our existing Pulumi Puluminaries. You can check out the great folks currently making a difference in our community on the Pulumi Puluminaries page. Their hard work and dedication have laid a strong foundation for what is next.

What is the Pulumi Puluminaries 2.0 Program?

The Pulumi Puluminaries 2.0 Program is designed to recognize and empower developers, architects, and technologists who are committed to driving innovation with Pulumi. This program offers exclusive opportunities to collaborate with Pulumi engineers, gain early access to new features, and share insights with the community.

Similar to how other ambassador programs bring together dedicated advocates around a specific focus, the Pulumi Puluminaries 2.0 Program unites a community of individuals who champion Pulumi’s approach to cloud engineering. Pulumi Puluminaries lead conversations, help new users adopt Pulumi effectively, and showcase best practices and use cases that highlight the power of Infrastructure as Code. They are also key participants in shaping Pulumi’s future by providing feedback to the product team and taking part in special initiatives.

What does it mean to be a Pulumi Puluminary?

Pulumi Puluminaries serve as champions of cloud engineering. They help users navigate the complexities of building, deploying, and managing applications with Pulumi. They also promote the benefits of Infrastructure as Code in modern organizations by creating thought leadership content, hosting or co-hosting events, and offering guidance for those just getting started.

Key Responsibilities for Pulumi Puluminaries:

As a Pulumi Puluminary, you will be expected to engage with the community at least once a quarter. Here are some ways you can contribute:

Community Engagement
- Actively share Pulumi-related articles, tutorials, and project stories.
- Present at conferences, meetups, or webinars on topics that highlight Pulumi’s value.
- Mentor new users and help them get started with Pulumi.
- Participate in Pulumi’s community forums, Slack channels, and social media discussions.
Thought Leadership
- Write blog posts (like Simen A. W. Olsen did in his latest post Your Perfect Infrastructure May Not Be So Perfect or Tyler Mulligan) or create videos showcasing innovative ways to use Pulumi.
- Contribute to discussions around best practices and patterns for cloud engineering with Pulumi.
Product Feedback
- Offer beta-testing support and provide product feedback directly to the Pulumi team.
- Participate in early-access programs and help guide product roadmap decisions.

Benefits of Joining the Program

Early Access: Get sneak peeks at new Pulumi features and releases before they are publicly announced.
Direct Collaboration: Collaborate one-on-one with Pulumi engineers and product managers, influencing future capabilities.
Recognition and Swag: Receive official Pulumi Puluminaries 2.0 digital badges, exclusive swag, and spotlight opportunities on Pulumi’s blog and at events.
Networking and Growth: Build relationships with peers in the Pulumi ecosystem, deepen your technical expertise, and grow your professional network.

How to Get Involved

We are looking for Pulumi enthusiasts who have shown dedication and passion for empowering others in the Pulumi community. Whether you have built cutting-edge projects, delivered insightful conference talks, or helped users overcome technical hurdles, we would love to hear your story.

Whether you want to apply for yourself or nominate someone else, we’d love to hear from you!

Apply or Nominate Now

Run Open-Source LLMs on AWS EC2 with Ollama and Pulumi

Engin Diri — Mon, 27 Jan 2025 00:00:00 +0000

TL;DR. Want to self-host an open-source LLM on AWS? Use a g4dn.xlarge ($0.526/hr on-demand, 16 GB GPU memory) for 7B/8B models, a g5.xlarge ($1.006/hr, 24 GB) for 13B–14B models, a g5.2xlarge ($1.212/hr, 24 GB) for 32B models, or a g6e.2xlarge ($2.242/hr, 48 GB) for 70B models. Deploy with the Pulumi program below and Ollama will run any model from its library: DeepSeek-R1, Llama 3, Qwen, or Mistral, with a one-line change.

This guide walks through that deployment end-to-end: a single Pulumi program that provisions a GPU-enabled EC2 instance, installs Ollama and Open WebUI via cloud-init, and exposes both a chat UI and an OpenAI-compatible API. The model is configurable, so you can swap DeepSeek-R1 for Llama 3.1, Qwen 2.5, or Mistral without touching the infrastructure code.

Why run open-source LLMs on AWS EC2?

Self-hosting an open-source LLM on AWS gives you three things hosted APIs can’t: data stays inside your VPC, per-token costs collapse to a flat hourly rate at high volume, and you can fine-tune or quantize models freely under permissive licenses. Ollama handles all three concerns from a single binary: it downloads, manages, and serves models behind an OpenAI-compatible API on port 11434.

The original version of this post focused on DeepSeek-R1 because it landed in late January 2025 and reset expectations for what an open-weight reasoning model could do. DeepSeek-R1 is still an excellent default (MIT-licensed, strong on math and coding, with distilled 1.5B–70B variants) but the same infrastructure runs Meta’s Llama 3, Alibaba’s Qwen, and Mistral equally well. Picking a model is now a config change, not an infrastructure decision.

For reference, DeepSeek-R1 scores 79.8% on AIME 2024 (vs. 79.2% for OpenAI o1-1217), 97.3% on MATH-500 (vs. 96.4%), 96.3% on Codeforces (vs. 96.6%), and 49.2% on SWE-bench Verified (vs. 48.9%)—near-parity with closed frontier models on most reasoning benchmarks, with the same caveat that benchmark scores age fast.

You might also like:

Which models can I run, and which EC2 instance do I need?

The bottleneck for inference is GPU memory (VRAM): the model weights have to fit in it, plus a few GB of headroom for the KV cache. The table below maps the most common Ollama models to the smallest AWS GPU instance that comfortably runs them at 4-bit (Q4) quantization, which is what ollama pull <model> gives you by default.

Model family	Sizes	Approx. VRAM (Q4)	Smallest EC2 instance	On-demand price (us-east-1)
DeepSeek-R1 (distill)	1.5B / 7B / 8B	1–6 GB	`g4dn.xlarge` (T4, 16 GB)	$0.526/hr
Llama 3.1 / Llama 3.2	8B	~5 GB	`g4dn.xlarge` (T4, 16 GB)	$0.526/hr
Qwen 2.5	7B	~5 GB	`g4dn.xlarge` (T4, 16 GB)	$0.526/hr
Mistral 7B / Mistral Nemo	7B / 12B	5–8 GB	`g4dn.xlarge` (T4, 16 GB)	$0.526/hr
DeepSeek-R1 (distill)	14B	~10 GB	`g5.xlarge` (A10G, 24 GB)	$1.006/hr
Llama 3.3 / DeepSeek-R1	32B / 32B distill	~20 GB	`g5.2xlarge` (A10G, 24 GB)	$1.212/hr
Llama 3.1 / DeepSeek-R1	70B	~42 GB	`g6e.2xlarge` (L40S, 48 GB)	$2.242/hr
DeepSeek-R1 (full)	671B (MoE)	400 GB+	`p5.48xlarge` or multi-node	$98.32/hr

For most workloads—internal tools, RAG backends, code assistants—a g4dn.xlarge running an 8B model is the right starting point. Move up only if quality is the bottleneck.

How much does this cost vs. hosted APIs?

A g4dn.xlarge running 24/7 costs ~$378/month on-demand, or ~$237/month with a 1-year reserved instance. Whether that’s cheaper than a hosted API depends entirely on your token volume.

Compare against hosted pricing as of April 2026 (input + output blended, rough numbers):

Provider	Model	Approx. blended price
OpenAI	GPT-4o-mini	~$0.30 per 1M tokens
OpenAI	GPT-4o	~$5.00 per 1M tokens
Anthropic	Claude Sonnet 4	~$6.00 per 1M tokens
DeepSeek (hosted)	DeepSeek-V3	~$0.50 per 1M tokens

A g4dn.xlarge running Llama 3.1 8B sustains roughly 40–60 tokens/sec under single-user load, or about 100–155M tokens/month at 100% utilization. At that ceiling the effective rate is ~$2.40/M tokens—cheaper than GPT-4o or Claude, more expensive than GPT-4o-mini or DeepSeek’s own hosted API.

The takeaway: self-hosting wins on data residency, latency, and predictable cost at high utilization. Hosted APIs win below ~10M tokens/month or when you need frontier-class quality. Run the math against your actual token volume before committing.

How do I deploy Ollama on AWS EC2 with Pulumi?

Prerequisites

Before we start, make sure you have the following:

An AWS account
Pulumi CLI installed
AWS CLI installed and configured
A working understanding of Ollama

What is Ollama?

Ollama is an open-source runtime that downloads, manages, and serves large language models from a single binary. It runs on macOS, Linux, and Windows, supports GPU acceleration through CUDA and Metal, and exposes both a native HTTP API and an OpenAI-compatible API on port 11434. Most clients written for the OpenAI SDK work against an Ollama endpoint with only a base-URL change.

It supports the major open-weight families—DeepSeek-R1, Llama 3, Qwen, Mistral, Gemma, Phi—plus quantized and distilled variants for each. You pull a model by name and tag (ollama pull llama3.1:8b) and run it (ollama run llama3.1:8b); Ollama handles the rest.

Architecture

Create a new Pulumi project

First, scaffold a new Pulumi project. Run the following from an empty directory:

# Replace <language> with typescript, python, go, csharp, or yaml
pulumi new aws-<language>

Pick the language you are most comfortable with. The template installs the AWS provider and creates a working sample. You can delete the sample code—we’ll replace it with the snippets below.

Step 1: Create an instance role with S3 access

The EC2 instance needs to download NVIDIA drivers from a public AWS-managed S3 bucket. Create an IAM role with S3 read access and attach it to an instance profile:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as fs from "fs";

const role = new aws.iam.Role("deepSeekRole", {
 name: "deepseek-role",
 assumeRolePolicy: JSON.stringify({
 Version: "2012-10-17",
 Statement: [
 {
 Action: "sts:AssumeRole",
 Effect: "Allow",
 Principal: {
 Service: "ec2.amazonaws.com",
 },
 },
 ],
 }),
});

new aws.iam.RolePolicyAttachment("deepSeekS3Policy", {
 policyArn: "arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess",
 role: role.name,
});

const instanceProfile = new aws.iam.InstanceProfile("deepSeekProfile", {
 name: "deepseek-profile",
 role: role.name,
});

import pulumi
import pulumi_aws as aws
import json
import os

# IAM Role for EC2 instances
role = aws.iam.Role(
 "deepSeekRole",
 name="deepseek-role",
 assume_role_policy=json.dumps(
 {
 "Version": "2012-10-17",
 "Statement": [
 {
 "Action": "sts:AssumeRole",
 "Effect": "Allow",
 "Principal": {
 "Service": "ec2.amazonaws.com",
 },
 }
 ],
 }
 ),
)

# Attach S3 read-only policy to the IAM Role
iam_policy_attachment = aws.iam.RolePolicyAttachment(
 "deepSeekS3Policy",
 policy_arn="arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess",
 role=role.name,
)

# Instance Profile containing the IAM Role
instance_profile = aws.iam.InstanceProfile(
 "deepSeekProfile", name="deepseek-profile", role=role.name
)

package main

import (
 "encoding/json"
 "os"

 "github.com/pulumi/pulumi-aws/sdk/v6/go/aws/ec2"
 "github.com/pulumi/pulumi-aws/sdk/v6/go/aws/iam"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 rolePolicy, err := json.Marshal(map[string]interface{}{
 "Version": "2012-10-17",
 "Statement": []map[string]interface{}{
 {
 "Action": "sts:AssumeRole",
 "Effect": "Allow",
 "Principal": map[string]interface{}{"Service": "ec2.amazonaws.com"},
 },
 },
 })
 if err != nil {
 return err
 }

 role, err := iam.NewRole(ctx, "deepSeekRole", &iam.RoleArgs{
 Name: pulumi.String("deepseek-role"),
 AssumeRolePolicy: pulumi.String(rolePolicy),
 })
 if err != nil {
 return err
 }

 // Attach S3 read-only policy to the IAM Role
 _, err = iam.NewRolePolicyAttachment(ctx, "deepSeekS3Policy", &iam.RolePolicyAttachmentArgs{
 PolicyArn: pulumi.String("arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess"),
 Role: role.Name,
 })
 if err != nil {
 return err
 }

 // Instance Profile containing the IAM Role
 instanceProfile, err := iam.NewInstanceProfile(ctx, "deepSeekProfile", &iam.InstanceProfileArgs{
 Name: pulumi.String("deepseek-profile"),
 Role: role.Name,
 })
 if err != nil {
 return err
 }
 return nil
 })
}

using Pulumi;
using Pulumi.Aws.Ec2;
using Pulumi.Aws.Iam;
using System.Collections.Generic;
using System.IO;
using Pulumi.Aws.Ec2.Inputs;
using System.Text.Json;

return await Deployment.RunAsync(() =>
{
 var rolePolicy = JsonSerializer.Serialize(new Dictionary<string, object>
 {
 {
 ["Version"] = "2012-10-17",
 ["Statement"] = new[]
 {
 new Dictionary<string, object>
 {
 ["Action"] = "sts:AssumeRole",
 ["Effect"] = "Allow",
 ["Principal"] = new Dictionary<string, string>
 {
 ["Service"] = "ec2.amazonaws.com"
 }
 }
 }
 });

 var role = new Role("deepSeekRole", new RoleArgs
 {
 Name = "deepseek-role",
 AssumeRolePolicy = rolePolicy
 });

 var rolePolicyAttachment = new RolePolicyAttachment("deepSeekS3Policy", new RolePolicyAttachmentArgs
 {
 PolicyArn = "arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess",
 Role = role.Name
 });

 var instanceProfile = new InstanceProfile("deepSeekProfile", new InstanceProfileArgs
 {
 Name = "deepseek-profile",
 Role = role.Name
 });

 var vpc = new Vpc("deepSeekVpc", new VpcArgs
 {
 CidrBlock = "10.0.0.0/16",
 EnableDnsHostnames = true,
 EnableDnsSupport = true
 });

 var subnet = new Subnet("deepSeekSubnet", new SubnetArgs
 {
 VpcId = vpc.Id,
 CidrBlock = "10.0.48.0/20",
 AvailabilityZone = "eu-central-1a",
 MapPublicIpOnLaunch = true
 });

 var internetGateway = new InternetGateway("deepSeekInternetGateway", new InternetGatewayArgs
 {
 VpcId = vpc.Id
 });

 var routeTable = new RouteTable("deepSeekRouteTable", new RouteTableArgs

name: deepseek-ollama-yaml
description: DeepSeek Ollama AWS example
runtime: yaml

variables:
 publicKey:
 fn::readFile: ./deepseek.rsa
 userData:
 fn::readFile: ./cloud-init.yaml
 amiFilter: "amzn2-ami-hvm-*-x86_64-gp2"
 amiOwner: "137112412989"
 amiId:
 fn::invoke:
 function: aws:ec2:getAmi
 arguments:
 filters:
 - name: name
 values: ["${amiFilter}"]
 owners: ["${amiOwner}"]
 mostRecent: true
 return: id

resources:
 deepSeekRole:
 type: aws:iam:Role
 properties:
 name: deepseek-role
 assumeRolePolicy: |
 {
 "Version": "2012-10-17",
 "Statement": [
 {
 "Action": "sts:AssumeRole",
 "Effect": "Allow",
 "Principal": {
 "Service": "ec2.amazonaws.com"
 }
 }
 ]
 }

 deepSeekS3Policy:
 type: aws:iam:RolePolicyAttachment
 properties:
 role: ${deepSeekRole.name}
 policyArn: arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess

 deepSeekProfile:
 type: aws:iam:InstanceProfile
 properties:
 name: deepseek-profile
 role: ${deepSeekRole.name}

Step 2: Create the network

Next, create the VPC, subnet, internet gateway, and route table. The security group opens ports 22 (SSH), 3000 (Open WebUI), and 11434 (Ollama API):

const vpc = new aws.ec2.Vpc("deepSeekVpc", {
 cidrBlock: "10.0.0.0/16",
 enableDnsHostnames: true,
 enableDnsSupport: true,
});

const subnet = new aws.ec2.Subnet("deepSeekSubnet", {
 vpcId: vpc.id,
 cidrBlock: "10.0.48.0/20",
 availabilityZone: pulumi.interpolate`${aws.getAvailabilityZones().then(it => it.names[0])}`,
 mapPublicIpOnLaunch: true,
});

const internetGateway = new aws.ec2.InternetGateway("deepSeekInternetGateway", {
 vpcId: vpc.id,
});

const routeTable = new aws.ec2.RouteTable("deepSeekRouteTable", {
 vpcId: vpc.id,
 routes: [
 {
 cidrBlock: "0.0.0.0/0",
 gatewayId: internetGateway.id,
 },
 ],
});

const routeTableAssociation = new aws.ec2.RouteTableAssociation("deepSeekRouteTableAssociation", {
 subnetId: subnet.id,
 routeTableId: routeTable.id,
});

const securityGroup = new aws.ec2.SecurityGroup("deepSeekSecurityGroup", {
 vpcId: vpc.id,
 egress: [
 {
 fromPort: 0,
 toPort: 0,
 protocol: "-1",
 cidrBlocks: ["0.0.0.0/0"],
 },
 ],
 ingress: [
 {
 fromPort: 22,
 toPort: 22,
 protocol: "tcp",
 cidrBlocks: ["0.0.0.0/0"],
 },
 {
 fromPort: 3000,
 toPort: 3000,
 protocol: "tcp",
 cidrBlocks: ["0.0.0.0/0"],
 },
 {
 fromPort: 11434,
 toPort: 11434,
 protocol: "tcp",
 cidrBlocks: ["0.0.0.0/0"],
 },
 ],
});


# Create a VPC
vpc = aws.ec2.Vpc(
 "deepSeekVpc",
 cidr_block="10.0.0.0/16",
 enable_dns_hostnames=True,
 enable_dns_support=True,
)

# Create a subnet
subnet = aws.ec2.Subnet(
 "deepSeekSubnet",
 vpc_id=vpc.id,
 cidr_block="10.0.48.0/20",
 availability_zone="eu-central-1a",
 map_public_ip_on_launch=True,
)

# Create an internet gateway
internet_gateway = aws.ec2.InternetGateway("deepSeekInternetGateway", vpc_id=vpc.id)

# Create a route table and route table association
route_table = aws.ec2.RouteTable(
 "deepSeekRouteTable",
 vpc_id=vpc.id,
 routes=[
 aws.ec2.RouteTableRouteArgs(
 cidr_block="0.0.0.0/0", gateway_id=internet_gateway.id
 )
 ],
)

route_table_association = aws.ec2.RouteTableAssociation(
 "deepSeekRouteTableAssociation", subnet_id=subnet.id, route_table_id=route_table.id
)

# Create a security group
security_group = aws.ec2.SecurityGroup(
 "deepSeekSecurityGroup",
 vpc_id=vpc.id,
 egress=[
 {
 "from_port": 0,
 "to_port": 0,
 "protocol": "-1",
 "cidr_blocks": ["0.0.0.0/0"],
 }
 ],
 ingress=[
 {
 "from_port": 22,
 "to_port": 22,
 "protocol": "tcp",
 "cidr_blocks": ["0.0.0.0/0"],
 },
 {
 "from_port": 3000,
 "to_port": 3000,
 "protocol": "tcp",
 "cidr_blocks": ["0.0.0.0/0"],
 },
 {
 "from_port": 11434,
 "to_port": 11434,
 "protocol": "tcp",
 "cidr_blocks": ["0.0.0.0/0"],
 },
 ],
)

package main

import (
 "encoding/json"
 "os"

 "github.com/pulumi/pulumi-aws/sdk/v6/go/aws/ec2"
 "github.com/pulumi/pulumi-aws/sdk/v6/go/aws/iam"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // omitted for brevity
 }

 // Create a VPC
 vpc, err := ec2.NewVpc(ctx, "deepSeekVpc", &ec2.VpcArgs{
 CidrBlock: pulumi.String("10.0.0.0/16"),
 EnableDnsHostnames: pulumi.Bool(true),
 EnableDnsSupport: pulumi.Bool(true),
 })
 if err != nil {
 return err
 }

 // Create a subnet
 subnet, err := ec2.NewSubnet(ctx, "deepSeekSubnet", &ec2.SubnetArgs{
 VpcId: vpc.ID(),
 CidrBlock: pulumi.String("10.0.48.0/20"),
 AvailabilityZone: pulumi.String("eu-central-1a"),
 MapPublicIpOnLaunch: pulumi.Bool(true),
 })
 if err != nil {
 return err
 }

 // Create an internet gateway
 internetGateway, err := ec2.NewInternetGateway(ctx, "deepSeekInternetGateway", &ec2.InternetGatewayArgs{
 VpcId: vpc.ID(),
 })
 if err != nil {
 return err
 }

 // Create a route table and route table association
 routeTable, err := ec2.NewRouteTable(ctx, "deepSeekRouteTable", &ec2.RouteTableArgs{
 VpcId: vpc.ID(),
 Routes: ec2.RouteTableRouteArray{
 &ec2.RouteTableRouteArgs{
 CidrBlock: pulumi.String("0.0.0.0/0"),
 GatewayId: internetGateway.ID(),
 },
 },
 })
 if err != nil {
 return err
 }

 _, err = ec2.NewRouteTableAssociation(ctx, "deepSeekRouteTableAssociation", &ec2.RouteTableAssociationArgs{
 SubnetId: subnet.ID(),
 RouteTableId: routeTable.ID(),
 })
 if err != nil {
 return err
 }

 // Create a security group
 securityGroup, err := ec2.NewSecurityGroup(ctx, "deepSeekSecurityGroup", &ec2.SecurityGroupArgs{
 VpcId: vpc.ID(),
 Egress: ec2.SecurityGroupEgressArray{
 &ec2.SecurityGroupEgressArgs{
 FromPort: pulumi.Int(0),
 ToPort: pulumi.Int(0),
 Protocol: pulumi.String("-1"),
 CidrBlocks: pulumi.StringArray{pulumi.String("0.0.0.0/0")},
 },
 },
 Ingress: ec2.SecurityGroupIngressArray{
 &ec2.SecurityGroupIngressArgs{
 FromPort: pulumi.Int(22),
 ToPort: pulumi.Int(22),
 Protocol: pulumi.String("tcp"),
 CidrBlocks: pulumi.StringArray{pulumi.String("0.0.0.0/0")},
 },
 &ec2.SecurityGroupIngressArgs{
 FromPort: pulumi.Int(3000),
 ToPort: pulumi.Int(3000),
 Protocol: pulumi.String("tcp"),
 CidrBlocks: pulumi.StringArray{pulumi.String("0.0.0.0/0")},
 },
 &ec2.SecurityGroupIngressArgs{
 FromPort: pulumi.Int(11434),
 ToPort: pulumi.Int(11434),
 Protocol: pulumi.String("tcp"),
 CidrBlocks: pulumi.StringArray{pulumi.String("0.0.0.0/0")},
 },
 },
 })
 if err != nil {
 return err
 }

 return nil
 })
}

using Pulumi;
using Pulumi.Aws.Ec2;
using Pulumi.Aws.Iam;
using System.Collections.Generic;
using System.IO;
using Pulumi.Aws.Ec2.Inputs;
using System.Text.Json;

return await Deployment.RunAsync(() =>
{
 var rolePolicy = JsonSerializer.Serialize(new Dictionary<string, object>
 {
 // omitted for brevity
 {
 VpcId = vpc.Id,
 Routes =
 {
 new RouteTableRouteArgs
 {
 CidrBlock = "0.0.0.0/0",
 GatewayId = internetGateway.Id
 }
 }
 });

 var routeTableAssociation = new RouteTableAssociation("deepSeekRouteTableAssociation", new RouteTableAssociationArgs
 {
 SubnetId = subnet.Id,
 RouteTableId = routeTable.Id
 });

 var securityGroup = new SecurityGroup("deepSeekSecurityGroup", new SecurityGroupArgs
 {
 VpcId = vpc.Id,
 Egress =
 {
 new SecurityGroupEgressArgs
 {
 FromPort = 0,
 ToPort = 0,
 Protocol = "-1",
 CidrBlocks = { "0.0.0.0/0" }
 }
 },
 Ingress =
 {
 new SecurityGroupIngressArgs
 {
 FromPort = 22,
 ToPort = 22,
 Protocol = "tcp",
 CidrBlocks = { "0.0.0.0/0" }
 },
 new SecurityGroupIngressArgs
 {
 FromPort = 3000,
 ToPort = 3000,
 Protocol = "tcp",
 CidrBlocks = { "0.0.0.0/0" }
 },
 new SecurityGroupIngressArgs
 {
 FromPort = 11434,
 ToPort = 11434,
 Protocol = "tcp",
 CidrBlocks = { "0.0.0.0/0" }
 }
 }
 });

 var publicKey = File.ReadAllText("deepseek.rsa");
 var keyPair = new KeyPair("deepSeekKey", new KeyPairArgs
 {
 PublicKey = publicKey
 });

 var amazonLinux = Pulumi.Aws.Ec2.GetAmi.Invoke(new()
 {
 MostRecent = true,
 Filters = new[]
 {
 new GetAmiFilterInputArgs
 {
 Name = "name",
 Values = new[] { "amzn2-ami-hvm-*-x86_64-gp2" }
 },
 new GetAmiFilterInputArgs
 {
 Name = "architecture",


 deepSeekVpc:
 type: aws:ec2:Vpc
 properties:
 cidrBlock: 10.0.0.0/16
 enableDnsHostnames: true
 enableDnsSupport: true

 deepSeekSubnet:
 type: aws:ec2:Subnet
 properties:
 vpcId: ${deepSeekVpc.id}
 cidrBlock: 10.0.48.0/20
 availabilityZone: eu-central-1a
 mapPublicIpOnLaunch: true

 deepSeekInternetGateway:
 type: aws:ec2:InternetGateway
 properties:
 vpcId: ${deepSeekVpc.id}

 deepSeekRouteTable:
 type: aws:ec2:RouteTable
 properties:
 vpcId: ${deepSeekVpc.id}
 routes:
 - cidrBlock: 0.0.0.0/0
 gatewayId: ${deepSeekInternetGateway.id}

 deepSeekRouteTableAssociation:
 type: aws:ec2:RouteTableAssociation
 properties:
 subnetId: ${deepSeekSubnet.id}
 routeTableId: ${deepSeekRouteTable.id}

 deepSeekSecurityGroup:
 type: aws:ec2:SecurityGroup
 properties:
 vpcId: ${deepSeekVpc.id}
 ingress:
 - fromPort: 22
 toPort: 22
 protocol: tcp
 cidrBlocks:
 - 0.0.0.0/0
 - fromPort: 3000
 toPort: 3000
 protocol: tcp
 cidrBlocks:
 - 0.0.0.0/0
 - fromPort: 11434
 toPort: 11434
 protocol: tcp
 cidrBlocks:
 - 0.0.0.0/0
 egress:
 - fromPort: 0
 toPort: 0
 protocol: -1
 cidrBlocks:
 - 0.0.0.0/0

Step 3: Launch the GPU EC2 instance

Now create the EC2 instance itself. The example uses Amazon Linux because the NVIDIA driver install path is well-trodden, plus an SSH key pair you generate locally.

The default instance type is g4dn.xlarge—the cheapest option that fits any 7B/8B model. Bump it up if you picked a larger model from the table above: g5.xlarge for 13B–14B, g5.2xlarge for 32B, g6e.2xlarge for 70B. AWS publishes full specs for the G4, G5, and G6 families.

Generate the key pair if you don’t already have one:

openssl genrsa -out deepseek.pem 2048
openssl rsa -in deepseek.pem -pubout > deepseek.pub
ssh-keygen -f mykey.pub -i -mPKCS8 > deepseek.pem

const keyPair = new aws.ec2.KeyPair("deepSeekKey", {
 publicKey: pulumi.output(fs.readFileSync("deepseek.rsa", "utf-8")),
});

const deepSeekAmi = aws.ec2
 .getAmi({
 filters: [
 {
 name: "name",
 values: ["amzn2-ami-hvm-2.0.*-x86_64-gp2"],
 },
 {
 name: "architecture",
 values: ["x86_64"],
 },
 ],
 owners: ["137112412989"], // Amazon
 mostRecent: true,
 })
 .then(ami => ami.id);

const deepSeekInstance = new aws.ec2.Instance("deepSeekInstance", {
 ami: deepSeekAmi,
 instanceType: "g4dn.xlarge",
 keyName: keyPair.keyName,
 rootBlockDevice: {
 volumeSize: 100,
 volumeType: "gp3",
 },
 subnetId: subnet.id,
 vpcSecurityGroupIds: [securityGroup.id],
 iamInstanceProfile: instanceProfile.name,
 userData: fs.readFileSync("cloud-init.yaml", "utf-8"),
 tags: {
 Name: "deepSeek-server",
 },
});

export const amiId = deepSeekAmi;
export const instanceId = deepSeekInstance.id;
export const instancePublicDns = deepSeekInstance.publicIp;


# Key pair for SSH access
public_key = open("deepseek.rsa", "r").read()
key_pair = aws.ec2.KeyPair("deepSeekKey", public_key=public_key)


# Get the latest Amazon Linux 2 AMI
ami = aws.ec2.get_ami(
 filters=[
 {"name": "name", "values": ["amzn2-ami-hvm-2.0.*-x86_64-gp2"]},
 {"name": "architecture", "values": ["x86_64"]},
 ],
 owners=["137112412989"], # Amazon
 most_recent=True,
).id

# Create an EC2 instance
user_data = open("cloud-init.yaml", "r").read()
instance = aws.ec2.Instance(
 "deepSeekInstance",
 ami=ami,
 instance_type="g4dn.xlarge",
 key_name=key_pair.key_name,
 root_block_device=aws.ec2.InstanceRootBlockDeviceArgs(
 volume_size=100, volume_type="gp3"
 ),
 subnet_id=subnet.id,
 vpc_security_group_ids=[security_group.id],
 iam_instance_profile=instance_profile.name,
 user_data=user_data,
 tags={"Name": "deepSeek-server"},
)

pulumi.export("amiId", ami)
pulumi.export("instanceId", instance.id)
pulumi.export("instancePublicDns", instance.public_ip)

package main

import (
 "encoding/json"
 "os"

 "github.com/pulumi/pulumi-aws/sdk/v6/go/aws/ec2"
 "github.com/pulumi/pulumi-aws/sdk/v6/go/aws/iam"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // omitted for brevity

 // Key pair for SSH access
 publicKey, err := os.ReadFile("deepseek.rsa")
 if err != nil {
 return err
 }

 keyPair, err := ec2.NewKeyPair(ctx, "deepSeekKey", &ec2.KeyPairArgs{
 PublicKey: pulumi.String(string(publicKey)),
 })
 if err != nil {
 return err
 }

 // Get the latest Amazon Linux 2 AMI
 mostRecent := true
 ami, err := ec2.LookupAmi(ctx, &ec2.LookupAmiArgs{
 Filters: []ec2.GetAmiFilter{
 {
 Name: "name",
 Values: []string{"amzn2-ami-hvm-2.0.*-x86_64-gp2"},
 },
 {
 Name: "architecture",
 Values: []string{"x86_64"},
 },
 },
 Owners: []string{"137112412989"},
 MostRecent: &mostRecent,
 })
 if err != nil {
 return err
 }

 // Create an EC2 instance
 userData, err := os.ReadFile("cloud-init.yaml")
 if err != nil {
 return err
 }

 instance, err := ec2.NewInstance(ctx, "deepSeekInstance", &ec2.InstanceArgs{
 Ami: pulumi.String(ami.Id),
 InstanceType: pulumi.String("g4dn.xlarge"),
 KeyName: keyPair.KeyName,
 RootBlockDevice: &ec2.InstanceRootBlockDeviceArgs{
 VolumeSize: pulumi.Int(100),
 VolumeType: pulumi.String("gp3"),
 },
 SubnetId: subnet.ID(),
 VpcSecurityGroupIds: pulumi.StringArray{securityGroup.ID()},
 IamInstanceProfile: instanceProfile.Name,
 UserData: pulumi.String(string(userData)),
 Tags: pulumi.StringMap{
 "Name": pulumi.String("deepSeek-server"),
 },
 })
 if err != nil {
 return err
 }

 ctx.Export("amiId", pulumi.String(ami.Id))
 ctx.Export("instanceId", instance.ID())
 ctx.Export("instancePublicDns", instance.PublicIp)

 return nil
 })
}

using Pulumi;
using Pulumi.Aws.Ec2;
using Pulumi.Aws.Iam;
using System.Collections.Generic;
using System.IO;
using Pulumi.Aws.Ec2.Inputs;
using System.Text.Json;

return await Deployment.RunAsync(() =>
{
 var rolePolicy = JsonSerializer.Serialize(new Dictionary<string, object>
 {
 // omitted for brevity
 Name = "architecture",
 Values = new[] { "x86_64" }
 }
 },
 Owners = new[] { "137112412989" }
 });

 var userData = File.ReadAllText("cloud-init.yaml");
 var instance = new Instance("deepSeekInstance", new InstanceArgs
 {
 Ami = amazonLinux.Apply(r => r.Id),
 InstanceType = "g4dn.xlarge",
 KeyName = keyPair.KeyName,
 RootBlockDevice = new InstanceRootBlockDeviceArgs
 {
 VolumeSize = 100,
 VolumeType = "gp3"
 },
 SubnetId = subnet.Id,
 VpcSecurityGroupIds = { securityGroup.Id },
 IamInstanceProfile = instanceProfile.Name,
 UserData = userData,
 Tags = { { "Name", "deepSeek-server" } }
 });

 return new Dictionary<string, object?>
 {
 ["amiId"] = amazonLinux.Apply(r => r.Id),
 ["instanceId"] = instance.Id,
 ["instancePublicDns"] = instance.PublicIp
 };
});


 deepSeekKey:
 type: aws:ec2:KeyPair
 properties:
 publicKey: ${publicKey}

 deepSeekInstance:
 type: aws:ec2:Instance
 properties:
 ami: ${amiId}
 instanceType: "g4dn.xlarge"
 keyName: ${deepSeekKey.keyName}
 rootBlockDevice:
 volumeSize: 100
 volumeType: gp3
 subnetId: ${deepSeekSubnet.id}
 vpcSecurityGroupIds:
 - ${deepSeekSecurityGroup.id}
 iamInstanceProfile: ${deepSeekProfile.name}
 userData: ${userData}
 tags:
 Name: deepSeek-server

outputs:
 AmiId: ${amiId}
 InstanceId: ${deepSeekInstance.id}
 InstancePublicDns: ${deepSeekInstance.publicIp}

Step 4: Install Ollama via cloud-init

The EC2 instance is a blank box until cloud-init runs. The user-data script below installs the NVIDIA GRID drivers, Docker, and the NVIDIA Container Toolkit, then starts the Ollama and Open WebUI containers. To switch models, edit the ollama run line—the rest is identical regardless of which model you want.

#cloud-config
users:
- default

package_update: true

packages:
- apt-transport-https
- ca-certificates
- curl
- openjdk-17-jre-headless
- gcc

runcmd:
- yum install -y gcc kernel-devel-$(uname -r)
- aws s3 cp --recursive s3://ec2-linux-nvidia-drivers/latest/ .
- chmod +x NVIDIA-Linux-x86_64*.run
- /bin/sh ./NVIDIA-Linux-x86_64*.run --tmpdir . --silent
- touch /etc/modprobe.d/nvidia.conf
- echo "options nvidia NVreg_EnableGpuFirmware=0" | sudo tee --append /etc/modprobe.d/nvidia.conf
- yum install -y docker
- usermod -a -G docker ec2-user
- systemctl enable docker.service
- systemctl start docker.service
- curl -s -L https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo | sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo
- yum install -y nvidia-container-toolkit
- nvidia-ctk runtime configure --runtime=docker
- systemctl restart docker
- docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama --restart always ollama/ollama
- sleep 120
- docker exec ollama ollama run deepseek-r1:7b
- docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main

You might also like:

Step 5: Deploy the infrastructure

Make sure your AWS credentials are configured:

aws configure

Pulumi supports several other authentication methods for the AWS provider. Once credentials are in place, deploy the infrastructure:

pulumi up

Pulumi previews the changes; type yes to confirm.

pulumi up
Please choose a stack, or create a new one: <create a new stack>
Please enter your desired stack name.
To create a stack in an organization, use the format <org-name>/<stack-name> (e.g. `acmecorp/dev`): dev
Please enter your desired stack name.
Created stack 'dev'
Previewing update (dev)
View in Browser (Ctrl+O): https://app.pulumi.com/dirien/deepseek-ollama-typescript/dev/previews/1dbb18ea-ba31-4d5b-9510-5dce19eb8ee8
Type Name Plan
+ pulumi:pulumi:Stack deepseek-ollama-typescript-dev create
+ ├─ aws:ec2:KeyPair deepSeekKey create
+ ├─ aws:ec2:Vpc deepSeekVpc create
+ ├─ aws:iam:Role deepSeekRole create
+ ├─ aws:iam:InstanceProfile deepSeekProfile create
+ ├─ aws:ec2:SecurityGroup deepSeekSecurityGroup create
+ ├─ aws:ec2:RouteTable deepSeekRouteTable create
+ ├─ aws:ec2:InternetGateway deepSeekInternetGateway create
+ ├─ aws:ec2:Subnet deepSeekSubnet create
+ ├─ aws:iam:RolePolicyAttachment deepSeekS3Policy create
+ ├─ aws:ec2:RouteTableAssociation deepSeekRouteTableAssociation create
+ └─ aws:ec2:Instance deepSeekInstance create
Outputs:
amiId : "ami-085131ff43045c877"
instanceId : output<string>
instancePublicDns: output<string>
Resources:
+ 12 to create
Do you want to perform this update? yes
Updating (dev)
View in Browser (Ctrl+O): https://app.pulumi.com/dirien/deepseek-ollama-typescript/dev/updates/1
Type Name Status
+ pulumi:pulumi:Stack deepseek-ollama-typescript-dev created (40s)
+ ├─ aws:ec2:KeyPair deepSeekKey created (0.47s)
+ ├─ aws:iam:Role deepSeekRole created (1s)
+ ├─ aws:ec2:Vpc deepSeekVpc created (12s)
+ ├─ aws:iam:InstanceProfile deepSeekProfile created (6s)
+ ├─ aws:iam:RolePolicyAttachment deepSeekS3Policy created (0.90s)
+ ├─ aws:ec2:InternetGateway deepSeekInternetGateway created (0.69s)
+ ├─ aws:ec2:Subnet deepSeekSubnet created (11s)
+ ├─ aws:ec2:SecurityGroup deepSeekSecurityGroup created (2s)
+ ├─ aws:ec2:RouteTable deepSeekRouteTable created (1s)
+ ├─ aws:ec2:RouteTableAssociation deepSeekRouteTableAssociation created (0.92s)
+ └─ aws:ec2:Instance deepSeekInstance created (12s)
Outputs:
amiId : "ami-085131ff43045c877"
instanceId : "i-0ae7495781ace3e81"
instancePublicDns: "18.159.211.136"
Resources:
+ 12 created
Duration: 42s

The infrastructure provisions in under a minute, but the cloud-init script needs another 5–10 minutes to install drivers, pull container images, and download the model weights. SSH in to watch the progress, or just wait and load the Web UI when it’s ready.

ssh -i deepseek.pem ec2-user@<instance-public-ip>

Check the container status:

sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c8714335e205 ghcr.io/open-webui/open-webui:main "bash start.sh" 6 minutes ago Up 6 minutes (healthy) 0.0.0.0:3000->8080/tcp, :::3000->8080/tcp open-webui
bf4bb3b7ede1 ollama/ollama "/bin/ollama serve" 8 minutes ago Up 7 minutes 0.0.0.0:11434->11434/tcp, :::11434->11434/tcp ollama
[ec2-user@ip-10-0-58-122 ~]$

Step 6: Access the Web UI or API

Once the containers are healthy, open http://<instance-public-ip>:3000 in your browser for Open WebUI:

Open WebUI is not secured by default. Restrict the security group to your IP, put it behind an authenticated proxy, or terminate TLS at an ALB before exposing it to the internet.

For programmatic access, Ollama exposes an OpenAI-compatible API on port 11434. Most clients written for the OpenAI SDK only need a base-URL change:

from openai import OpenAI

client = OpenAI(
 base_url="http://<instance-public-ip>:11434/v1",
 api_key="ollama", # required by the SDK, ignored by Ollama
)

response = client.chat.completions.create(
 model="llama3.1:8b",
 messages=[{"role": "user", "content": "Why is the sky blue?"}],
)
print(response.choices[0].message.content)

Deploy GPU infrastructure with Pulumi

Provision the GPU EC2 instance, network, and IAM role for your self-hosted LLM in one program, then switch models with a single config change.

Get started

How do I switch models?

Ollama hosts every major open-weight family in its model library. Pulling a different model is two commands inside the EC2 instance—or a one-line edit to the cloud-init script if you want it provisioned automatically:

# DeepSeek-R1 distill (default in this guide)
docker exec ollama ollama run deepseek-r1:7b

# Llama 3.1 (Meta, 8B)
docker exec ollama ollama run llama3.1:8b

# Qwen 2.5 (Alibaba, 7B)
docker exec ollama ollama run qwen2.5:7b

# Mistral (7B)
docker exec ollama ollama run mistral:7b

# Larger reasoning model (needs g5.2xlarge or larger)
docker exec ollama ollama run deepseek-r1:32b

Tags follow a <size> or <size>-<quantization> pattern—8b, 8b-instruct-q4_K_M, 8b-instruct-q8_0. Q4 is the default and the right starting point; bump to Q8 only if you have spare VRAM and notice quality issues with Q4. Browse the full tag list for any model on its Ollama library page.

Cleaning up

When you’re done, tear everything down:

pulumi destroy

What are the next steps?

You now have a reproducible, IaC-managed deployment of any open-source LLM on AWS. The infrastructure is fixed; the model is a parameter. From here, the natural extensions are wiring this up to a real application, adding RAG over your own data, or moving the deployment behind an authenticated load balancer.

If you want to go further with AI on Pulumi, here are some related guides:

Deploy LangServe Apps with Pulumi on AWS (RAG & Chatbot) — Build a retrieval-augmented chatbot that could front-end this Ollama instance.
Deploy AI Models on Amazon SageMaker using Pulumi Python IaC — A SageMaker alternative when you’d rather not manage the EC2 host yourself.
Build an AI Slack Bot on AWS Using Embedchain & Pulumi — Wire an LLM into Slack as an internal assistant.
What is Infrastructure as Code? — Background on the IaC approach used throughout this guide.

Try Pulumi for Free

Changelog

2026-04-30 — Broadened scope from DeepSeek-only to any Ollama-supported model (Llama, Qwen, Mistral). Added TL;DR, instance-type recommendation table, cost-vs-hosted-API comparison, and HowTo structured data. Restructured headings as user questions. Verified Ollama and cloud-init commands against current versions.
2025-03-10 — Minor edits and corrections.
2025-01-27 — Original post: Run DeepSeek-R1 on AWS EC2 Using Ollama.

Kubernetes Best Practices I Wish I Had Known Before

Engin Diri — Mon, 20 Jan 2025 00:00:00 +0000

Kubernetes best practices are the production-tested rules that keep clusters reliable, secure, and affordable: set resource requests and limits on every container, isolate workloads with namespaces and NetworkPolicies, enforce least-privilege RBAC, automate health checks, ship via GitOps, validate every change with policy-as-code, generate an SBOM for every image, and manage Kubernetes itself with infrastructure as code instead of hand-rolled YAML. The 20 practices below cover what production teams actually do in 2026 — not what tutorials suggest.

TL;DR: 20 Kubernetes best practices for 2026

Set resource requests and limits on every container.
Use namespaces plus ResourceQuota and LimitRange for isolation.
Run one container per Pod unless you genuinely need a sidecar.
Manage manifests with Helm, Kustomize, or Pulumi — never raw copies.
Use the Gateway API (ingress-nginx is retired) for north-south traffic.
Configure liveness, readiness, and startup probes for every workload.
Enforce RBAC with the principle of least privilege from day one.
Apply Pod Security Admission at the restricted level by default.
Lock down east-west traffic with NetworkPolicy (default-deny).
Store secrets in an external secret manager (Pulumi ESC, Vault, AWS Secrets Manager) — not as plain Kubernetes Secrets.
Autoscale with HPA, VPA, and Cluster Autoscaler / Karpenter.
Monitor with Prometheus, Grafana, and OpenTelemetry; alert before users do.
Deploy with GitOps (Argo CD or Flux) — Git is the only source of truth.
Gate every change with policy-as-code (Pulumi CrossGuard, OPA/Gatekeeper, Kyverno).
Generate and verify an SBOM for every image; sign with Sigstore/cosign.
Manage cost with FinOps practices: right-size, schedule, and chargeback by namespace.
Keep clusters and add-ons patched on a documented cadence.
Use labels and annotations consistently for ownership, cost, and SLO tracking.
Separate dev, staging, and prod clusters — or virtualize with vCluster.
Treat clusters as cattle, not pets: rebuild from code, never edit live.

Last updated April 30, 2026 — added GitOps, policy-as-code, SBOM/supply-chain, autoscaling, NetworkPolicy, and FinOps sections; added an anti-pattern table; restructured headings around the questions teams actually ask.

Going to KubeCon Europe 2026? Tame Kubernetes complexity with code.

Pulumi demos, platform engineering talks, AI-powered Neo in action, and yes, free plushies. Booth 784.

The ‘Kubernetes is easy’-iceberg meme is a classic example of how Kubernetes can be deceivingly complex

What are Kubernetes best practices?

Kubernetes best practices are the operational patterns that keep workloads available, secure, observable, and cost-efficient on a Kubernetes cluster. They span four areas: workload configuration (resources, probes, images), security (RBAC, NetworkPolicy, Pod Security, secrets, supply chain), delivery (GitOps, policy-as-code, IaC), and platform operations (autoscaling, monitoring, upgrades, FinOps). The rest of this post explains each one with a copy-pasteable rule, a definition of the underlying Kubernetes object, and a common anti-pattern to avoid.

Common Kubernetes anti-patterns vs. the correct approach

Common mistake	What goes wrong	Correct approach
No resource requests or limits	Noisy-neighbor evictions, OOMKills, unschedulable Pods	Set requests for every container; use VPA recommendations to tune
Everything in the `default` namespace	No isolation, no quota, no RBAC boundary	One namespace per team or environment + ResourceQuota + RBAC
`cluster-admin` for service accounts	Full-cluster blast radius on compromise	Per-namespace Role + RoleBinding (least privilege)
Plain Kubernetes `Secret` objects	Base64 ≠ encryption; secrets leak via etcd, kubectl, audit logs	External Secrets Operator + Pulumi ESC, Vault, or AWS Secrets Manager
Open east-west traffic	One compromised Pod can reach every service	Default-deny NetworkPolicy; allow only required flows
`kubectl apply -f` from a laptop	No history, no review, no rollback	GitOps with Argo CD or Flux; Git is the source of truth
Manual `latest` image tags	Non-reproducible deploys, supply-chain risk	Pinned digests (`@sha256:...`) with cosign verification
Ingress-nginx in 2026	Project retired March 2026 — no security fixes	Migrate to the Gateway API (Envoy Gateway, Istio, Kgateway)
Hand-rolled YAML in dozens of repos	Drift, copy-paste bugs, no testing	Pulumi, Helm, or Kustomize with reviews
One shared `prod` cluster for everything	Blast radius spans every team	Separate clusters or vCluster virtual clusters

1. How do you set Kubernetes resource requests and limits?

What are resource requests and limits? A resource request is the minimum CPU and memory the kube-scheduler guarantees a container; a resource limit is the maximum the kubelet allows it to consume before throttling CPU or OOMKilling memory. Pods without requests are scheduled blind, so a noisy neighbor can evict your workload at any time.

The rule: every container in production sets a request for both CPU and memory, and a memory limit. CPU limits are optional and frequently counter-productive — they cause throttling spikes that look like outages.

Start at 100–200m CPU and 128–512Mi memory for a typical web service.
Tune from real data using Vertical Pod Autoscaler with updateMode: Off (recommendation-only) or Prometheus histograms.
Use a per-namespace LimitRange so teams cannot ship a container with no requests at all.

2. How should you structure Kubernetes namespaces?

What is a Kubernetes namespace? A namespace is a logical partition of cluster resources. It scopes object names, RBAC bindings, NetworkPolicies, and quotas — but does not provide hard isolation by itself.

The rule: never deploy production workloads to default. Map namespaces to ownership boundaries — usually one per team, environment, or tenant — and pair every namespace with three things:

A ResourceQuota to cap aggregate CPU, memory, and object counts.
A LimitRange to enforce per-container defaults.
A RoleBinding that grants only the permissions that team needs.

This is the smallest unit that gives you predictable cost, blast-radius containment, and an RBAC perimeter.

3. Should you run multiple containers in one Pod?

What is a Pod? A Pod is the smallest deployable unit in Kubernetes — one or more containers that share a network namespace, IPC, and storage volumes, and that are scheduled and scaled as a single unit.

The rule: one container per Pod by default. Add a second container only when it must share the Pod’s network or volumes — for example, a service-mesh sidecar (Istio/Linkerd), a logging shipper, or an init container that prepares state before the main process starts. Multi-container Pods couple lifecycles and scaling, so reach for them deliberately, not by default.

You might also like:

4. Should you use Helm, Kustomize, or Pulumi for Kubernetes manifests?

The rule: never copy-paste raw YAML across services. Pick one of these:

Helm — Go-templated charts; the de-facto distribution format for off-the-shelf software (Prometheus, cert-manager, Argo CD).
Kustomize — overlay-based, no templating; ships in kubectl. Best for “same app, different environment” diffs.
Pulumi — real programming languages (TypeScript, Python, Go, Java, .NET) for Kubernetes plus the cloud resources around it (EKS, GKE, AKS, DNS, IAM). Type checking, tests, and the Pulumi Kubernetes Operator for GitOps-style reconciliation.

If you are already managing cloud infrastructure with code, extending the same Pulumi program to Kubernetes is the smallest cognitive jump — the cluster, its IAM, its DNS, and its workloads live in one stack and one review.

5. How should you handle ingress and networking on Kubernetes in 2026?

What is the Gateway API? The Gateway API is the successor to the legacy Ingress resource. It splits responsibility between infrastructure (GatewayClass, Gateway) and application (HTTPRoute, GRPCRoute, TLSRoute) teams, and it is the SIG-Network direction for north-south traffic.

The rule: new clusters should standardize on the Gateway API. The widely deployed ingress-nginx controller retired in March 2026, and remaining deployments should migrate to Envoy Gateway, Istio, Linkerd Gateway, or Kgateway.

Terminate TLS at the gateway and automate certificates with cert-manager.
Use path-based and host-based routing instead of one gateway per service.
Protect every gateway with a WAF (managed cloud WAF or Coraza on Envoy).

ingress-nginx is retiring in March 2026.

Here is a guide on How to Move to the Gateway API: post ingress-nginx Retirement.

6. What’s the difference between liveness, readiness, and startup probes?

Probes tell the kubelet whether to restart a container, route traffic to it, or wait for it. Configure them per workload:

Liveness probe — restarts a hung container. Use sparingly; a bad liveness probe is a self-inflicted DoS.
Readiness probe — gates traffic from Services and Gateway routes. Use for every network-facing workload.
Startup probe — gives slow-booting apps (JVM, ML models) time to come up before liveness checks begin. Use whenever cold-start exceeds 10 seconds.

See the upstream guide for HTTP, TCP, and exec probe syntax. Tune failureThreshold, periodSeconds, and timeoutSeconds based on real latency, not defaults.

7. How do you secure a Kubernetes cluster?

Cluster security is layered: identity, workload posture, network, and secrets. Get all four right.

What is RBAC in Kubernetes?

RBAC (Role-Based Access Control) is the Kubernetes authorization model that maps subjects (users, groups, ServiceAccounts) to verbs (get, list, create, delete) on resources, scoped to a namespace (Role) or the whole cluster (ClusterRole).

Grant least privilege — never bind humans or workloads to cluster-admin.
Prefer per-namespace Role + RoleBinding over ClusterRoleBinding.
Audit with kubectl auth can-i --as=... and tools like rbac-lookup.

What is Pod Security Admission?

Pod Security Admission is the built-in admission controller that enforces the three Pod Security Standards (privileged, baseline, restricted) at the namespace level.

Label every namespace pod-security.kubernetes.io/enforce=restricted by default.
Drop NET_RAW and all capabilities; run as non-root with a read-only root filesystem.
Use seccompProfile: RuntimeDefault and a tight securityContext for every container.

What is a NetworkPolicy?

A NetworkPolicy is a Kubernetes object that defines allowed ingress and egress traffic for selected Pods at the IP/port level. Without one, every Pod can reach every other Pod by default.

Apply a default-deny ingress and egress policy in every namespace.
Allow only the flows your service actually needs (DNS, the database, the upstream API).
For richer L7 controls (mTLS, JWT auth), layer a service mesh on top.

How should you manage Kubernetes secrets?

Built-in Secret objects are base64-encoded, not encrypted. Treat them as a transport, not as storage.

Store the source of truth in Pulumi ESC, HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, or Azure Key Vault.
Project secrets into Pods with the External Secrets Operator or the Secret Store CSI Driver.
Enable etcd encryption-at-rest with a KMS provider so leaked etcd snapshots stay opaque.
Rotate automatically; never commit secrets to Git, even encrypted.

8. How do you set up Kubernetes observability?

Observability rests on the three pillars: metrics, logs, and traces. The 2026 default stack is OpenTelemetry-first.

Metrics — Prometheus (or a Prometheus-compatible TSDB like Mimir, Thanos, VictoriaMetrics) plus Grafana dashboards.
Logs — Fluent Bit or the OpenTelemetry Collector shipping structured JSON to Loki, Elastic/OpenSearch, or a managed service. Never depend on kubectl logs for incidents — Pods are ephemeral.
Traces — OpenTelemetry SDKs in your apps emitting to Tempo, Jaeger, or a vendor backend. Correlate trace IDs with logs.
Alerts — page on SLO burn rate, not on every crashed Pod. Burn-rate alerting catches real user impact without firing on every restart.

9. How do you automate Kubernetes deployments with GitOps?

What is GitOps? GitOps is a deployment model where the desired cluster state lives in a Git repository and a controller continuously reconciles the live cluster to match. Changes happen through pull requests, not kubectl apply.

The rule: every cluster has a controller — Argo CD, Flux, or the Pulumi Kubernetes Operator — that owns reconciliation. Humans never kubectl apply to production.

Use app-of-apps (Argo) or Kustomization trees (Flux) to bootstrap whole clusters from a single root.
Keep manifests, Helm values, and Pulumi stack references in Git, with environment promotion via PR.
Combine GitOps with Pulumi Deployments so cloud infrastructure (VPCs, EKS clusters, IAM, DNS) and the workloads on top promote through the same review pipeline. See improving GitOps with the Pulumi Operator for a worked example.
Configure automated rollback on health-check failure, and drift detection alerts when someone edits live state.

10. How often should you upgrade Kubernetes?

Kubernetes ships a minor release roughly every four months and supports each one for about 14 months. Treat upgrades as routine maintenance, not a project.

Stay within two minor versions of upstream — older versions miss CVE patches.
Test upgrades in a non-prod cluster, run a conformance test, then promote.
Back up etcd before control-plane changes; managed services (EKS, GKE, AKS) handle most of this for you.
Upgrade add-ons (CNI, CSI, Gateway controller, cert-manager, metrics-server) on the same cadence — pin versions in code so the upgrade is auditable.
Watch the Kubernetes deprecation guide and run pluto or kubent to find deprecated APIs before they break.

11. How should you label and annotate Kubernetes resources?

Labels and annotations are the metadata layer that everything — Services, NetworkPolicies, monitoring, cost tools, GitOps — keys off of. Consistency matters more than cleverness.

Adopt the recommended labels: app.kubernetes.io/name, app.kubernetes.io/instance, app.kubernetes.io/version, app.kubernetes.io/component, app.kubernetes.io/part-of, app.kubernetes.io/managed-by.
Add organization-specific labels for owner, cost-center, environment, and SLO tier — your FinOps and on-call processes will need them.
Reserve annotations for non-identifying metadata: build SHA, change-ticket, last-deployed timestamp, ingress controller hints.

Manage Kubernetes with code

Provision your clusters, their cloud resources, and the workloads on them from one Pulumi program, and use stacks to promote changes from dev to production.

Get started

12. How should you separate Kubernetes environments?

The rule: production is its own cluster. Dev and staging can share — sometimes. Mixing prod with anything else couples blast radius, upgrade cadence, and quotas.

Separate clusters per environment is the safe default. With managed control planes (EKS, GKE, AKS) the cost is small.
Virtual clusters with vCluster give each team a Kubernetes API of their own without the operational cost of a real cluster.
Namespace-only segregation can work for dev/staging if you enforce strict NetworkPolicy, RBAC, and ResourceQuota boundaries.
Provision all of them from the same Pulumi program with different stacks — dev, staging, prod — so promotion is a config change, not a copy.

13. How do you optimize Kubernetes container images?

Start from a distroless (gcr.io/distroless) or chainguard/static base. Alpine is fine if you accept musl.
Build with multi-stage Dockerfiles so build tools never ship in the final image.
Pin the base image by digest (@sha256:...), not by tag — latest and 1.21 move under you.
Scan every image with Trivy, Grype, or your registry’s built-in scanner. Block builds on Critical/High CVEs.
Smaller images mean faster pulls, faster autoscaling, and a smaller attack surface.

14. What’s a reliable Kubernetes logging strategy?

Centralize to Loki, Elastic/OpenSearch, or a managed service via Fluent Bit or the OpenTelemetry Collector running as a DaemonSet.
Structure logs as JSON with consistent field names (trace_id, span_id, service, level).
Retain by tier — 7 days hot, 30–90 days warm, archive to object storage for compliance.
Redact secrets and PII at the collector, not after the fact.

You might also like:

15. How should you autoscale workloads on Kubernetes?

What is the Horizontal Pod Autoscaler? The HPA scales the replica count of a Deployment or StatefulSet up and down based on CPU, memory, or custom/external metrics.

What is the Vertical Pod Autoscaler? The VPA adjusts a Pod’s CPU and memory requests over time based on observed usage. Run it with updateMode: Off (recommendation-only) in production and apply the recommendations through your IaC tool.

What are Cluster Autoscaler and Karpenter? Cluster Autoscaler adds and removes nodes to match Pod demand. Karpenter is a faster, group-less node provisioner now widely used on EKS and gaining traction on other clouds.

The rule: every production workload has an HPA. Every cluster has a node-level autoscaler. Set sensible min/max replicas, define a PodDisruptionBudget, and gate scale-down behavior so traffic spikes don’t pile up at restart time. Pair with KEDA for event-driven scaling (queues, Kafka lag, scheduled scale).

16. How do you enforce policy-as-code on Kubernetes?

What is policy-as-code? Policy-as-code expresses governance rules — security baselines, tagging, region restrictions, cost guardrails — as code that runs in CI and at admission time, blocking non-compliant changes before they reach the cluster.

The rule: every change goes through at least one policy engine.

Pre-deploy — validate Pulumi or Helm output with Pulumi CrossGuard or Conftest in CI.
Admission-time — install OPA Gatekeeper or Kyverno and require, for example, signed images, a team label, and a non-root securityContext on every Pod.
Continuous — scan running clusters with Pulumi Insights or kubescape for drift from policy.

See the benefits of policy-as-code and enforcing policy-as-code on discovered resources for end-to-end examples.

17. How do you secure the Kubernetes software supply chain?

What is an SBOM? A Software Bill of Materials is a machine-readable inventory of every component in a container image — packages, versions, licenses, hashes — typically in SPDX or CycloneDX format.

The rule: every image you run in production has a verified SBOM and a verified signature.

Generate SBOMs at build time with Syft or docker sbom.
Sign images and SBOMs with Sigstore/cosign keylessly via GitHub OIDC.
Verify signatures at admission with Kyverno or Connaisseur.
Generate provenance attestations (SLSA) so you can prove which build pipeline produced a given image.
Pin base images by digest and rebuild on a cadence so CVE patches actually land.

This is what regulators and customers ask for in 2026 — Executive Order 14028, the EU Cyber Resilience Act, and most enterprise procurement checklists all require it.

18. How do you manage Kubernetes cost (FinOps)?

The rule: Kubernetes cost is a labeling and right-sizing problem before it is a discount problem.

Right-size with VPA recommendations, Goldilocks, or your APM’s recommendation engine. Most workloads request more CPU than they actually use; right-sizing is usually the largest single saving.
Schedule non-prod clusters off overnight and on weekends.
Spot/preemptible nodes for fault-tolerant workloads; use Karpenter’s consolidation to compact bin-packing.
Show back / charge back by namespace and label using OpenCost or Kubecost. Without per-team attribution, no team owns cost.
Provision the underlying cluster with IaC — see Pulumi for Kubernetes — so node types, autoscaling limits, and reserved capacity are reviewable artifacts. Pulumi’s hidden-cost analysis walks through where the money actually goes.

19. Why should you treat Kubernetes clusters as cattle, not pets?

The old “cattle, not pets” adage applies more strictly to Kubernetes than to VMs. Manual edits to a live cluster will be reconciled away, drift between environments, or vanish on the next Pod restart.

Fix problems in code — YAML, Helm chart, or Pulumi program — and let the controller redeploy.
If you can’t redeploy a cluster from Git in under an hour, you have a pet.
Use ephemeral preview environments for PRs; use blue/green or progressive delivery (Argo Rollouts, Flagger) for production cutovers.

20. Why use Pulumi to manage Kubernetes?

Native YAML scales until your team doesn’t. Once you have more than a handful of services, dependencies between cloud resources and Kubernetes objects, or more than one cluster, infrastructure as code in a real programming language wins on every axis.

Real languages — TypeScript, Python, Go, Java, .NET — for type-safe, testable Kubernetes infrastructure. Loops, conditionals, and unit tests instead of templating.
One stack, full topology — manage the cloud (EKS/GKE/AKS, VPC, IAM, DNS) and the workloads in it together. See easily create and manage AWS EKS clusters with Pulumi.
Reusable components — abstract platform patterns into Pulumi packages other teams import instead of copy-pasting.
GitOps reconciliation — the Pulumi Kubernetes Operator reconciles a cluster to a Pulumi stack on every Git push.
Policy-as-code built in — CrossGuard blocks non-compliant changes before pulumi up.
Secrets done right — Pulumi ESC federates secrets and configuration across stacks, environments, and Kubernetes clusters.

For a deeper comparison of hand-written YAML, Terraform, and Pulumi for Kubernetes, see YAML, Terraform, Pulumi: what’s the smart choice for deployment automation with Kubernetes, and the beyond YAML write-up on where Kubernetes automation is heading in 2026.

See how Pulumi helps you manage Kubernetes Services and Deployments with code instead of YAML—bringing structure, reuse, and type safety to your configs.

By adopting Pulumi, you can avoid the complexity of juggling endless YAML files and gain a more streamlined, maintainable workflow for your Kubernetes infrastructure.

Final thoughts

Kubernetes rewards discipline. The 20 practices above — resource hygiene, namespaced isolation, RBAC, NetworkPolicy, Pod Security, external secrets, probes, the Gateway API, observability, GitOps, policy-as-code, signed images and SBOMs, autoscaling, FinOps, multi-cluster strategy, and IaC — are the dial settings that separate clusters that page their owners every week from clusters that don’t.

Pick the three weakest spots in your environment and fix those first. Then promote the fixes through code, not through kubectl.

Want to learn how to put these practices into action? Meet us at KubeCon Europe 2026 (Booth 784) or register for our upcoming Zero to Production in Kubernetes workshop.

Try Pulumi for Free

Master Kubernetes Secrets with Pulumi ESC + Secrets Store CSI Driver

Engin Diri — Wed, 18 Dec 2024 09:58:01 +0100

Welcome to the second blog post of the Pulumi ESC and Kubernetes secrets series. If you haven’t had the chance to read the first blog post, go ahead and read it here.

In the previous blog post, we have learned how to manage secrets with Pulumi ESC and the External Secrets Operator. While the External Secrets Operator is a great tool to manage secrets in a cloud-native way, it still creates Kubernetes secrets in the cluster. Depending on your security requirements, you might want to avoid the use of Kubernetes secrets in your cluster at all. This is the point where you hit the limits of the External Secrets Operator.

But don’t worry, we also have a solution for this problem. In this blog post, we will introduce the Secrets Store CSI Driver.

TL;DR?

Jump straight to the comparison of the External Secrets Operator and the Secrets Store CSI Driver here.

Refresher: Why Should You Avoid Using Kubernetes Secrets?

Before we dive into the Secrets Store CSI Driver, let’s quickly recap why you might want to avoid using native Kubernetes secrets. I wrote in depth about this topic in the previous blog post, but here is a quick summary:

By default, etcd stores them in a base64-encoded format, which is not secure.
Developers create them either manually by using kubectl commands or by using a manifest file, which makes them hard to manage at scale.
Hard to manage and synchronise secrets across different environments and clusters.
There is no default way to rotate secrets automatically.

Point 1 is the most critical one, as there is even a warning in the Kubernetes documentation:

Kubernetes Secrets. Credit: kubernetes.io/docs/concepts/configuration/secret/

What is the Secrets Store CSI Driver?

The first thing you may notice is that in the name of the Secret Store CSI Driver, there is a reference to the Container Storage Interface (CSI) standard.

What is CSI and why it is used in the Secret Store CSI Driver*?*

How to write a Container Storage Interface (CSI) plugin. Credit: arslan.io/2018/06/21/how-to-write-a-container-storage-interface-csi-plugin/

Container Storage Interface (CSI)

The Container Storage Interface (CSI) is a standard to unify the interface between container orchestrators (like Kubernetes) and different storage vendors (like NetApp, Ceph, etc.). This helps to guarantee that implementing a CSI for a storage vendor is going to work with all orchestrators that support CSI.

Before CSI, we needed to create volume plugins for every orchestrator and storage vendor. This meant that developers coupled volume plugin development with the Kubernetes version and made it dependent on it. Bugs in volume plugins could break the Kubernetes components, instead of the volume plugin. Besides, volume plugins had full privileges on the Kubernetes components like the kubelet.

Combining CSI, Kubernetes and Secrets = Secret Store CSI Driver

The Secret Store CSI Driver is a CSI driver that allows you to mount many secrets, certificates, and keys from external secret stores into Kubernetes pods as volumes. After attaching the volume, the system mounts the secrets into the container file system.

The benefits of using the Secret Store CSI Driver are that you manage the lifecycle of the secrets outside of Kubernetes while still providing a Kubernetes-native experience of using the secrets in your pods.

The Architecture of the Secret Store CSI Driver

The Secret Store CSI Driver is a daemonset that lets all kublets communicate with each other and uses gRPC to talk to a provider. A SecretProviderClass custom resource specifies the definition of the external Secret Store. Then the system mounts a volume in the pod as tmpfs, and it injects the secret into the volume. When you delete the pod, the system cleans up the volume and removes the secret from the tmpfs volume.

How to Use the Secrets Store CSI Driver with Pulumi ESC

Before we start, make sure you have the following prerequisites:

A Pulumi Cloud account. If you don’t have one, you can create one for free.
A Kubernetes cluster (I will be using a local KinD cluster, but you can use any Kubernetes cluster)
Pulumi CLI installed. You can use the pulumi env commands to manage secrets and configurations.
kubectl CLI installed for some debugging

Step 1: Deploy the Secrets Store CSI Driver and Pulumi ESC CSI Provider

We will use Pulumi with the pulumi-kubernetes provider to deploy the Secrets Store CSI Driver and the Pulumi ESC CSI Provider to our Kubernetes cluster.

And here we also use the integration of Pulumi ESC with Pulumi IaC to supply the Pulumi access token in a secure way.

First, we need to create a new Pulumi project:

# Choose your favorite Pulumi supported language
pulumi new kubernetes-<your-programming-language> --name

Before we dig into the code, let’s head to the Pulumi Cloud Console and create a new Pulumi ESC project with the name esc-secrets-store-csi-driver-demo.

And create the environment dev:

In the editor add the following yaml into the Environment definition:

values:
 pulumiConfig:
 pulumi-pat:
 fn::secret: <your-pulumi-pat>

Don’t know how to create a PAT? Check out the official Pulumi documentation.

If you prefer to use the Pulumi CLI, you can create the environment by running:

pulumi env init <your-org>/esc-secrets-store-csi-driver-demo/dev

And set the configuration by running the env edit command and copy the above YAML into the editor:

pulumi env edit <your-org>/esc-secrets-store-csi-driver-demo/dev

Now, we need to link the Pulumi ESC project to the Pulumi IaC project. To do this, we need to add the following to your Pulumi.dev.yaml:

environment:
- esc-secrets-store-csi-driver-demo/dev

import * as pulumi from "@pulumi/pulumi";
import * as k8s from "@pulumi/kubernetes";

const secretsStoreCSIDriver = new k8s.helm.v4.Chart("secrets-store-csi-driver", {
 chart: "secrets-store-csi-driver",
 namespace: "kube-system",
 repositoryOpts: {
 repo: "https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts",
 },
 values: {
 nodeSelector: {
 "kubernetes.io/os": "linux",
 },
 },
});

const secretsStoreCSIPulumiESCProvider = new k8s.helm.v4.Chart(
 "secrets-store-csi-pulumi-esc-provider",
 {
 chart: "oci://ghcr.io/pulumi/helm-charts/pulumi-esc-csi-provider",
 namespace: "kube-system",
 values: {
 nodeSelector: {
 "kubernetes.io/os": "linux",
 },
 },
 },
 { dependsOn: secretsStoreCSIDriver },
);

const config = new pulumi.Config();

const mySecret = new k8s.core.v1.Secret(
 "my-secret",
 {
 metadata: {
 namespace: "default",
 name: "pulumi-access-token",
 },
 stringData: {
 "pulumi-access-token": config.require("pulumi-pat"),
 },
 type: "Opaque",
 },
 { dependsOn: secretsStoreCSIPulumiESCProvider },
);

import pulumi
import pulumi_kubernetes as k8s

secrets_store_csi_driver = k8s.helm.v4.Chart(
 "secrets-store-csi-driver",
 k8s.helm.v4.ChartArgs(
 chart="secrets-store-csi-driver",
 namespace="kube-system",
 repository_opts=k8s.helm.v4.RepositoryOptsArgs(
 repo="https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts",
 ),
 values={
 "nodeSelector": {
 "kubernetes.io/os": "linux",
 },
 },
 ),
)

secrets_store_csi_pulumi_esc_provider = k8s.helm.v4.Chart(
 "secrets-store-csi-pulumi-esc-provider",
 k8s.helm.v4.ChartArgs(
 chart="oci://ghcr.io/pulumi/helm-charts/pulumi-esc-csi-provider",
 namespace="kube-system",
 values={
 "nodeSelector": {
 "kubernetes.io/os": "linux",
 },
 },
 ),
 opts=pulumi.ResourceOptions(depends_on=[secrets_store_csi_driver]),
)

config = pulumi.Config()

my_secret = k8s.core.v1.Secret(
 "my-secret",
 metadata=k8s.meta.v1.ObjectMetaArgs(
 namespace="default", name="pulumi-access-token"
 ),
 string_data={
 "pulumi-access-token": config.require("pulumi-pat"),
 },
 type="Opaque",
 opts=pulumi.ResourceOptions(depends_on=[secrets_store_csi_pulumi_esc_provider]),
)

package main

import (
 k8s "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes"
 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/apiextensions"
 appv1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/apps/v1"
 corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/core/v1"
 chartv4 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/helm/v4"
 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/meta/v1"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi/config"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 secretsStoreCsiDriver, err := chartv4.NewChart(ctx, "secrets-store-csi-driver", &chartv4.ChartArgs{
 Chart: pulumi.String("secrets-store-csi-driver"),
 Namespace: pulumi.String("kube-system"),
 RepositoryOpts: chartv4.RepositoryOptsArgs{
 Repo: pulumi.String("https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts"),
 },
 Values: pulumi.Map{
 "nodeSelector": pulumi.Map{
 "kubernetes.io/os": pulumi.String("linux"),
 },
 },
 })
 if err != nil {
 return err
 }

 secretsStoreCsiPulumiEscProvider, err := chartv4.NewChart(ctx, "secrets-store-csi-pulumi-esc-provider", &chartv4.ChartArgs{
 Chart: pulumi.String("oci://ghcr.io/pulumi/helm-charts/pulumi-esc-csi-provider"),
 Namespace: pulumi.String("kube-system"),
 Values: pulumi.Map{
 "nodeSelector": pulumi.Map{
 "kubernetes.io/os": pulumi.String("linux"),
 },
 },
 }, pulumi.DependsOn([]pulumi.Resource{secretsStoreCsiDriver}))
 if err != nil {
 return err
 }

 pulumiPAT := config.Get(ctx, "pulumi-pat")

 mySecret, err := corev1.NewSecret(ctx, "my-secret", &corev1.SecretArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Namespace: pulumi.String("default"),
 Name: pulumi.String("pulumi-access-token"),
 },
 StringData: pulumi.StringMap{
 "pulumi-access-token": pulumi.String(pulumiPAT),
 },
 Type: pulumi.String("Opaque"),
 }, pulumi.DependsOn([]pulumi.Resource{secretsStoreCsiPulumiEscProvider}))
 if err != nil {
 return err
 }


 return nil
 })
}

using Pulumi;
using Pulumi.Kubernetes.Core.V1;
using Pulumi.Kubernetes.Types.Inputs.Core.V1;
using Pulumi.Kubernetes.Helm.V3;
using Pulumi.Kubernetes.Helm;
using Pulumi.Kubernetes.Types.Inputs.Meta.V1;
using Pulumi.Kubernetes.ApiExtensions;
using System.Collections.Generic;



return await Deployment.RunAsync(() =>
{
 var secretsStoreCsiDriver = new Release("secrets-store-csi-driver", new()
 {
 Chart = "secrets-store-csi-driver",
 Namespace = "kube-system",
 RepositoryOpts = new Pulumi.Kubernetes.Types.Inputs.Helm.V3.RepositoryOptsArgs
 {
 Repo = "https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts",
 },
 Values = new Dictionary<string, object>
 {
 { "nodeSelector", new Dictionary<string, object>
 {
 { "kubernetes.io/os", "linux" },
 } },
 },
 });

 var secretsStoreCsiPulumiEscProvider = new Release("secrets-store-csi-pulumi-esc-provider", new()
 {
 Chart = "oci://ghcr.io/pulumi/helm-charts/pulumi-esc-csi-provider",
 Namespace = "kube-system",
 Values = new Dictionary<string, object>
 {
 { "nodeSelector", new Dictionary<string, object>
 {
 { "kubernetes.io/os", "linux" },
 } },
 },
 }, new CustomResourceOptions { DependsOn = { secretsStoreCsiDriver } });

 var config = new Config();
 var pulumiPAT = config.Require("pulumi-pat");

 var mySecret = new Secret("my-secret", new SecretArgs
 {
 Metadata = new ObjectMetaArgs
 {
 Namespace = "default",
 Name = "pulumi-access-token",
 },
 StringData =
 {
 { "pulumi-access-token", pulumiPAT }
 },
 Type = "Opaque",
 }, new CustomResourceOptions { DependsOn = { secretsStoreCsiPulumiEscProvider } });

});


class SecretProviderClassArgs : CustomResourceArgs
{
 public SecretProviderClassArgs(): base("secrets-store.csi.x-k8s.io/v1", "SecretProviderClass")
 {
 }

 [Input("spec")]
 public Input<SecretProviderClassSpecArgs>? Spec { get; set; }
}


class SecretProviderClassSpecArgs : ResourceArgs
{
 [Input("provider")]
 public Input<string>? Provider { get; set; }

 [Input("parameters")]
 public Input<InputMap<object>>? Parameters { get; set; }
}

class SecretProviderParametersArgs : ResourceArgs
{
 [Input("apiUrl")]
 public Input<string>? ApiUrl { get; set; }

 [Input("organization")]
 public Input<string>? Organization { get; set; }

 [Input("project")]
 public Input<string>? Project { get; set; }

 [Input("environment")]
 public Input<string>? Environment { get; set; }

 [Input("authSecretName")]
 public Input<string>? AuthSecretName { get; set; }

 [Input("authSecretNamespace")]
 public Input<string>? AuthSecretNamespace { get; set; }

 [Input("secrets")]
 public Input<string>? Secrets { get; set; }
}

Deploy the stack by running:

pulumi up

And you should see that the secret was created in the Kubernetes cluster and the ESO instance was deployed successfully.

kubectl get secret pulumi-access-token -o jsonpath='{.data.PULUMI_ACCESS_TOKEN}' | base64 -d

Step 2: Create a SecretProviderClass

Now, we will create a secret in the Pulumi ESC project and synchronize it into the Kubernetes cluster by creating an SecretProviderClass:

Create a new ESC environment called csi-secrets-store-app in the esc-secrets-store-csi-driver-demo project:

values:
 app:
 hello: world
 hello-secret:
 fn::secret: world

If you prefer to use the Pulumi CLI, you can create the environment by running:

pulumi env init <your-org>/esc-secrets-store-csi-driver-demo/csi-secrets-store-app

And set the configuration by running the env edit command and copy the above YAML into the editor:

pulumi env edit <your-org>/esc-secrets-store-csi-driver-demo/csi-secrets-store-app

Either way, you should see following environment configuration in the Pulumi Cloud Console:

We can now create the SecretProviderClass in the Kubernetes cluster:

const secretProviderClass = new k8s.apiextensions.CustomResource(
 "example-provider-pulumi-esc",
 {
 apiVersion: "secrets-store.csi.x-k8s.io/v1",
 kind: "SecretProviderClass",
 metadata: {
 name: "example-provider-pulumi-esc",
 namespace: "default",
 },
 spec: {
 provider: "pulumi",
 parameters: {
 apiUrl: "https://api.pulumi.com/api/esc",
 organization: "dirien",
 project: "esc-secrets-store-csi-driver-demo",
 environment: "csi-secrets-store-app",
 authSecretName: mySecret.metadata.name,
 authSecretNamespace: mySecret.metadata.namespace,
 secrets: `- secretPath: "/"
 fileName: "hello"
 secretKey: "app.hello"
`,
 },
 },
 },
 { dependsOn: secretsStoreCSIPulumiESCProvider },
);

secret_provider_class = k8s.apiextensions.CustomResource(
 "example-provider-pulumi-esc",
 api_version="secrets-store.csi.x-k8s.io/v1",
 kind="SecretProviderClass",
 metadata=k8s.meta.v1.ObjectMetaArgs(
 name="example-provider-pulumi-esc", namespace="default"
 ),
 spec={
 "provider": "pulumi",
 "parameters": {
 "apiUrl": "https://api.pulumi.com/api/esc",
 "organization": "dirien",
 "project": "esc-secrets-store-csi-driver-demo",
 "environment": "csi-secrets-store-app",
 "authSecretName": my_secret.metadata["name"],
 "authSecretNamespace": my_secret.metadata["namespace"],
 "secrets": '- secretPath: "/"\n fileName: "hello"\n secretKey: "app.hello"\n',
 },
 },
 opts=pulumi.ResourceOptions(depends_on=[secrets_store_csi_pulumi_esc_provider]),
)

 secretProviderClass, err := apiextensions.NewCustomResource(ctx, "example-provider-pulumi-esc", &apiextensions.CustomResourceArgs{
 ApiVersion: pulumi.String("secrets-store.csi.x-k8s.io/v1"),
 Kind: pulumi.String("SecretProviderClass"),
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("example-provider-pulumi-esc"),
 Namespace: pulumi.String("default"),
 },
 OtherFields: k8s.UntypedArgs{
 "provider": pulumi.String("pulumi"),
 "parameters": pulumi.Map{
 "apiUrl": pulumi.String("https://api.pulumi.com/api/esc"),
 "organization": pulumi.String("dirien"),
 "project": pulumi.String("esc-secrets-store-csi-driver-demo"),
 "environment": pulumi.String("csi-secrets-store-app"),
 "authSecretName": mySecret.Metadata.Name().Elem(),
 "authSecretNamespace": mySecret.Metadata.Namespace().Elem(),
 "secrets": pulumi.String(`- secretPath: "/"
 fileName: "hello"
 secretKey: "app.hello"
`),
 },
 },
 }, pulumi.DependsOn([]pulumi.Resource{secretsStoreCsiPulumiEscProvider}))
 if err != nil {
 return err
 }

 var secretProviderClass = new Pulumi.Kubernetes.ApiExtensions.CustomResource("example-provider-pulumi-esc", new SecretProviderClassArgs
 {
 Metadata = new ObjectMetaArgs
 {
 Name = "example-provider-pulumi-esc",
 Namespace = "default",
 },
 Spec = new SecretProviderClassSpecArgs
 {
 Provider = "pulumi",
 Parameters = new InputMap<object>
 {
 { "apiUrl", "https://api.pulumi.com/api/esc" },
 { "organization", "dirien" },
 { "project", "esc-secrets-store-csi-driver-demo" },
 { "environment", "csi-secrets-store-app" },
 { "authSecretName", mySecret.Metadata.Apply(metadata => metadata.Name) },
 { "authSecretNamespace", mySecret.Metadata.Apply(metadata => metadata.Namespace) },
 { "secrets", "- secretPath: \"/\"\n fileName: \"hello\"\n secretKey: \"app.hello\"\n" }
 },
 },
 }, new CustomResourceOptions { DependsOn = { secretsStoreCsiPulumiEscProvider } });

We can check that the secret was successfully synchronized by running:

kubectl get secretproviderclasses example-provider-pulumi-esc

You might also like:

Step 3: Deploy an Application and Mount the Secret

Now, we can deploy an application that references the secret from the Kubernetes cluster. I am going to use busybox that reads mounted file in.

const deployment = new k8s.apps.v1.Deployment("example-provider-pulumi-esc", {
 metadata: {
 name: "example-provider-pulumi-esc",
 namespace: "default",
 labels: {
 app: "example-provider-pulumi-esc",
 },
 },
 spec: {
 replicas: 1,
 selector: {
 matchLabels: {
 app: "example-provider-pulumi-esc",
 },
 },
 template: {
 metadata: {
 labels: {
 app: "example-provider-pulumi-esc",
 },
 },
 spec: {
 containers: [
 {
 name: "client",
 image: "busybox:latest",
 command: ["sh", "-c"],
 args: [
 `set -eux
 ls /run/secrets
 find /run/secrets/ -mindepth 1 -maxdepth 1 -not -name '.*' | xargs -t -I {} sh -c 'echo "$(cat "{}")"'
 tail -f /dev/null`,
 ],
 volumeMounts: [
 {
 name: "data",
 mountPath: "/run/secrets",
 },
 ],
 },
 ],
 volumes: [
 {
 name: "data",
 csi: {
 driver: "secrets-store.csi.k8s.io",
 readOnly: true,
 volumeAttributes: {
 secretProviderClass: secretProviderClass.metadata.name,
 },
 },
 },
 ],
 },
 },
 },
});

export const deploymentName = deployment.metadata.name;

deployment = k8s.apps.v1.Deployment(
 "example-provider-pulumi-esc",
 metadata=k8s.meta.v1.ObjectMetaArgs(
 namespace="default",
 name="example-provider-pulumi-esc",
 labels={
 "app": "example-provider-pulumi-esc",
 },
 ),
 spec=k8s.apps.v1.DeploymentSpecArgs(
 replicas=1,
 selector=k8s.meta.v1.LabelSelectorArgs(
 match_labels={
 "app": "example-provider-pulumi-esc",
 },
 ),
 template=k8s.core.v1.PodTemplateSpecArgs(
 metadata=k8s.meta.v1.ObjectMetaArgs(
 labels={
 "app": "example-provider-pulumi-esc",
 },
 ),
 spec=k8s.core.v1.PodSpecArgs(
 containers=[
 k8s.core.v1.ContainerArgs(
 name="client",
 image="busybox:latest",
 command=["sh", "-c"],
 args=[
 "set -eux\nls /run/secrets\nfind /run/secrets/ -mindepth 1 -maxdepth 1 -not -name '.*' | xargs -t -I {} sh -c 'echo \"$(cat \"{}\")\"'\ntail -f /dev/null",
 ],
 volume_mounts=[
 k8s.core.v1.VolumeMountArgs(
 name="data",
 mount_path="/run/secrets",
 ),
 ],
 ),
 ],
 volumes=[
 k8s.core.v1.VolumeArgs(
 name="data",
 csi=k8s.core.v1.CSIVolumeSourceArgs(
 driver="secrets-store.csi.k8s.io",
 read_only=True,
 volume_attributes={
 "secretProviderClass": secret_provider_class.metadata[
 "name"
 ],
 },
 ),
 ),
 ],
 ),
 ),
 ),
)

pulumi.export("deploymentName", deployment.metadata["name"])

 deployment, err := appv1.NewDeployment(ctx, "example-provider-pulumi-esc", &appv1.DeploymentArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("example-provider-pulumi-esc"),
 Namespace: pulumi.String("default"),
 Labels: pulumi.StringMap{
 "app": pulumi.String("example-provider-pulumi-esc"),
 },
 },
 Spec: &appv1.DeploymentSpecArgs{
 Replicas: pulumi.Int(1),
 Selector: &metav1.LabelSelectorArgs{
 MatchLabels: pulumi.StringMap{
 "app": pulumi.String("example-provider-pulumi-esc"),
 },
 },
 Template: &corev1.PodTemplateSpecArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Labels: pulumi.StringMap{
 "app": pulumi.String("example-provider-pulumi-esc"),
 },
 },
 Spec: &corev1.PodSpecArgs{
 Containers: corev1.ContainerArray{
 &corev1.ContainerArgs{
 Name: pulumi.String("client"),
 Image: pulumi.String("busybox:latest"),
 Command: pulumi.StringArray{
 pulumi.String("sh"),
 pulumi.String("-c"),
 },
 Args: pulumi.StringArray{
 pulumi.String(`set -eux
 ls /run/secrets
 find /run/secrets/ -mindepth 1 -maxdepth 1 -not -name '.*' | xargs -t -I {} sh -c 'echo "$(cat "{}")"'
 tail -f /dev/null`),
 },
 VolumeMounts: &corev1.VolumeMountArray{
 &corev1.VolumeMountArgs{
 Name: pulumi.String("data"),
 MountPath: pulumi.String("/run/secrets"),
 },
 },
 },
 },
 Volumes: &corev1.VolumeArray{
 &corev1.VolumeArgs{
 Name: pulumi.String("data"),
 Csi: corev1.CSIVolumeSourceArgs{
 Driver: pulumi.String("secrets-store.csi.k8s.io"),
 ReadOnly: pulumi.Bool(true),
 VolumeAttributes: pulumi.StringMap{
 "secretProviderClass": secretProviderClass.Metadata.Name().Elem(),
 },
 },
 },
 },
 },
 },
 },
 }, pulumi.DependsOn([]pulumi.Resource{secretProviderClass}))
 if err != nil {
 return err
 }

 var deployment = new Pulumi.Kubernetes.Apps.V1.Deployment("example-provider-pulumi-esc", new Pulumi.Kubernetes.Types.Inputs.Apps.V1.DeploymentArgs
 {
 Metadata = new ObjectMetaArgs
 {
 Name = "example-provider-pulumi-esc",
 Namespace = "default",
 Labels =
 {
 { "app", "example-provider-pulumi-esc" },
 },
 },
 Spec = new Pulumi.Kubernetes.Types.Inputs.Apps.V1.DeploymentSpecArgs
 {
 Replicas = 1,
 Selector = new Pulumi.Kubernetes.Types.Inputs.Meta.V1.LabelSelectorArgs
 {
 MatchLabels =
 {
 { "app", "example-provider-pulumi-esc" },
 },
 },
 Template = new Pulumi.Kubernetes.Types.Inputs.Core.V1.PodTemplateSpecArgs
 {
 Metadata = new ObjectMetaArgs
 {
 Labels =
 {
 { "app", "example-provider-pulumi-esc" },
 },
 },
 Spec = new Pulumi.Kubernetes.Types.Inputs.Core.V1.PodSpecArgs
 {
 Containers =
 {
 new Pulumi.Kubernetes.Types.Inputs.Core.V1.ContainerArgs
 {
 Name = "client",
 Image = "busybox:latest",
 Command =
 {
 "sh",
 "-c",
 },
 Args =
 {
 "set -eux\nls /run/secrets\nfind /run/secrets/ -mindepth 1 -maxdepth 1 -not -name '.*' | xargs -t -I {} sh -c 'echo \"$(cat \"{}\")\"'\ntail -f /dev/null",
 },
 VolumeMounts =
 {
 new Pulumi.Kubernetes.Types.Inputs.Core.V1.VolumeMountArgs
 {
 Name = "data",
 MountPath = "/run/secrets",
 },
 },
 },
 },
 Volumes =
 {
 new Pulumi.Kubernetes.Types.Inputs.Core.V1.VolumeArgs
 {
 Name = "data",
 Csi = new Pulumi.Kubernetes.Types.Inputs.Core.V1.CSIVolumeSourceArgs
 {
 Driver = "secrets-store.csi.k8s.io",
 ReadOnly = true,
 VolumeAttributes =
 {
 { "secretProviderClass", secretProviderClass.Metadata.Apply(metadata => metadata.Name) },
 },
 },
 },
 },
 },
 },
 },
 }, new CustomResourceOptions { DependsOn = { secretProviderClass } });

After deploying the stack, you can get the logs of the busybox pod to see that the secret was successfully mounted:

NAME=$(kubectl get pods -o name | grep example-provider-pulumi-esc | cut -d'/' -f2)
kubectl logs $NAME

You should see the following output:

+ ls /run/secrets
hello
+ find /run/secrets/ -mindepth 1 -maxdepth 1 -not -name '.*'
+ xargs -t -I '{}' sh -c 'echo "$(cat "{}")"'
sh -c echo "$(cat "/run/secrets/hello")"
world
+ tail -f /dev/null

Or you can exec into the pod and check the content of the mounted secret:

kubectl exec -it $NAME -- cat /run/secrets/hello

You should see the following output:

world

Step 4: Clean Up

After you are done with the demo, you can clean up the resources by running:

pulumi destroy

Conclusion

The Secret Store CSI Driver is a great option for managing secrets in a cloud-native way when the use of Kubernetes secrets is not an option due to enhanced security requirements.

Below is a quick comparison of the External Secrets Operator and the Secrets Store CSI Driver to help you decide which one is the best fit for your use case. As always, the best way to find out is to try both solutions and see which one fits your requirements best.

The good part is, whatever you choose, you can use Pulumi ESC to manage your secrets and avoid any secret sprawl in your organization as both solutions are supported by Pulumi ESC.

Comparison of the External Secrets Operator and the Secrets Store CSI Driver

Feature/Aspect	External Secrets Operator	CSI Secrets Driver
Primary Use Case	Synchronizing external secrets into Kubernetes as native secrets.	Mounting secrets directly to pods as files or environment variables.
Integration with External Secret Stores	Supports multiple external secret stores like AWS Secrets Manager, Google Secret Manager, Vault, Azure Key Vault, etc.	Primarily supports integrations defined by the CSI specification (e.g., Vault, AWS, Azure Key Vault, Pulumi ESC).
Mechanism for Secrets Delivery	Secrets are pulled and stored as Kubernetes `Secret` resources.	Secrets are mounted directly into pods as files or injected as environment variables.
Kubernetes Resource Requirements	Requires a custom Kubernetes resource (`ExternalSecret`) and the External Secrets Operator.	Relies on the Kubernetes CSI (Container Storage Interface) standard and requires a specific CSI driver.
Secret Rotation	Supports automatic rotation by polling the external secret store at defined intervals.	Supports rotation, but the application needs to handle reloading mounted secrets.
Security Considerations	Secrets are stored temporarily in Kubernetes as native secrets, which may pose a risk if not properly managed.	Secrets are mounted directly and not stored as native Kubernetes secrets, reducing exposure but relying on file permissions.
Customization and Features	Highly customizable with fine-grained mappings and transformation options.	Limited customization, focused on mounting secrets as-is.
Application Compatibility	Applications access secrets as Kubernetes secrets (e.g., via env variables or volumes).	Applications access secrets directly via mounted files or injected env variables.
Performance Considerations	Potential slight delay due to syncing secrets to Kubernetes secrets.	Faster access as secrets are directly mounted.
Community and Ecosystem	Supported by a growing community; integrates well with diverse secret management solutions.	Part of the Kubernetes ecosystem; supports CSI-compliant secret storage solutions.
Best Use Cases	Ideal for scenarios where secrets need to be accessed as Kubernetes native secrets.	Best for applications requiring direct access to secrets without intermediate storage in Kubernetes secrets.

You can find detailed information about all current available Pulumi ESC Kubernetes integrations in the Pulumi ESC documentation.

Let us know what you think about the Secret Store CSI Driver and how you are managing secrets in your Kubernetes clusters. We would love to hear your feedback and experiences.

Try Pulumi for Free

Why Every Platform Engineer Should Care About Kubernetes Operators

Engin Diri — Tue, 10 Dec 2024 09:31:38 +0100

In one of my recent talks, I mentioned that the foundation of a successful Kubernetes-powered platform is the use of Kubernetes Operators, as they are a great way to automate operational tasks and the lifecycle of complex applications and services on Kubernetes.

In case you missed the talk, here is the link to the recording:

After the talk, I received feedback from the audience that they would like to have more information about Kubernetes operators and some of the advanced operators I would recommend to use in their Kubernetes-powered platforms. And who am I to reject such a great opportunity to share my knowledge with the community?

As you guess, the base for a Kubernetes Operator is, well, Kubernetes. So, if you are not familiar with Kubernetes, let me give you a brief introduction. Kubernetes, or K8s in short, is by far the most powerful open-source container orchestration platform out there - especially for its capability to maintain a desired state. This means you, as the platform admin, can define how you want your cluster and the workload on top of it to look, and Kubernetes (aka the control plane) executes the necessary steps to ensure the desired state is achieved using a feedback loop.

And how does Kubernetes achieve this complex operational task management? Kubernetes leverages the concept of controllers and operators. Both ensure that the cluster resources are in the desired state. But why is there a distinction between controllers and operators inside the Kubernetes ecosystem? There is also great confusion when folks talk about controllers and operators, as the differences are in the details and operators are a subcategory of controllers.

Kubernetes Controllers vs. Operators

K8s controllers are control loops that watch your resources and reconcile the current state with the desired state in a continuous loop. Think about how a thermostat works in your house. The thermostat watches the current temperature and, if it is below the desired temperature, it turns on the heating. If it is above the desired temperature, it turns off the heating. This is how a controller works in Kubernetes - it watches the resources and conditions defined in the desired state and executes the necessary steps to make sure the current state matches the desired state.

Here are some typical Kubernetes controller key functions as part of the maintain, observe, and enforce loop of the control loop:

Function	Description
Rollouts and Rollbacks	Deployment controller takes care of rolling out new versions of your application and rolling back to the previous version in case of failure.
Cluster maintenance	Node controller watches the nodes in the cluster and makes sure that the nodes are in the desired state.
Event handling	Event controller watches for events and takes action based on the events.
Schedule	Controller to run jobs and cron jobs at a specific time.
Resource enforcement	Resource controller watches the resource usage and enforces resource limits in a namespace.
State enforcement	Different controllers watch the state of the resources and enforce the desired state.

As you can see, controllers automate the routine operational tasks, enforce policies, handle failures and take actions to continuously maintain the desired state of the cluster.

Some of the typical use cases for controllers are:

Limit resource usage - Prevent the overload of the cluster by enforcing resource limits with controllers like ResourceQuota.
Auto-scaling - Automatically scales the number of pods based on the different metrics with HorizontalPodAutoscaler.
Task scheduling - Run jobs and cron jobs at a specific time with Job and CronJob controllers.
Run stateful workloads - Deploy and manage stateful workloads like databases with StatefulSet controller, by ensuring that the pods are created in a specific order and have a stable network identity.
Scaling deployments - Deploy and manage stateless workloads like web applications with the Deployment controller, by ensuring that the desired number of pods is running and handling the rolling updates and rollbacks.

Controllers, as you see, reduce the manual effort for the platform engineering teams and gives the cluster self-managing capabilities.

With all the above points, the benefits of controllers are clear - Let me list some of the main ones:

Automation - Controllers automate the routine operational tasks and enforce policies.
Availability - Controllers ensure that the resources are always available, addressing failures and taking actions to maintain the desired state.
Efficiency - Controllers provide an efficient way to manage the resources and workloads in the cluster with quota management and auto-scaling.
Reliability - Controllers help to make workload management more reliable with features like replica management, pod creation and deletion, and scheduling.
Flexibility - Controllers provide a flexible way to handle a variety of workloads and resources in the cluster, such as DaemonSet, StatefulSet, jobs, cron jobs, deployments, and services.
Observability - Controllers provide a view into the cluster resources and the state of the resources via the Kubernetes API.

With Controllers explained, let’s move on to the next level of automation in Kubernetes and talk about Kubernetes Operators.

Kubernetes Operators

As mentioned before, Kubernetes Operators are a subcategory of controllers that use API extensions (Custom Resource) to complete the automation task. Operators are a set of independent controllers, with each controller responsible for its own task. And while controllers can share similar functions with a controller, it is only focused on one domain and only uses Custom Resources to manage the domain.

Keep in mind that controllers work without the need for custom resources or a link to a specific domain.

Operators are custom-built controllers that focus on the deployment, management, and operation of a specific application or service. Some key functions of operators include:

Function	Description
Security	Operators integrate security best practices and policies, such as encryption, access control, and secret management, to protect the application. You can customize these measures and adjust them to the specific needs of the application.
Workload Lifecycle management	Operators manages the full lifecycle of complex stateful applications through deployment, upgrades, and scaling. It even applies logic to automate operations like backup and restore.
Error Handling	Operators detects and handles errors, such as application crashes, network failures, and resource constraints.
Migration	Operators help to migrate the application to a new version, a new cluster, or a new cloud provider.
Disaster Recovery	Operators help to recover the application from a disaster, such as data loss, network failure, or hardware failure.

Operators are a great way to encode the operational knowledge of the platform engineering team into automated controllers. This way, the platform engineering team can focus on the strategic tasks and let the operators handle the boring routine operational tasks. This reduces the manual effort and increases the reliability of the platform.

Some of the typical use cases for operators are:

Database management - Deploy and manage databases such as MySQL, PostgreSQL, and MongoDB.
Storage management - Deploy and manage storage such as Ceph, GlusterFS, or NFS.
Logging and monitoring - Simplify the deployment and management of logging and monitoring solutions like Prometheus, Grafana, or ELK stack.
CI/CD - Automate the deployment and management of CI/CD pipelines like Jenkins, GitLab, or Tekton. - Backup and restore - Automate the backup and restore of applications and data.
Messaging and event streaming - Simplify the deployment and management of messaging and event streaming solutions like Kafka, RabbitMQ, or NATS.
Machine learning - Automate the deployment and management of machine learning workloads such as TensorFlow, PyTorch, or Kubeflow.

With the applications-centric approach provided by operators, the platform engineering team can now enable fully automated operations while enhancing the reliability, efficiency, and observability of the platform and ensuring adherence to best practices and compliance requirements.

And similar to controllers, operators also have distinct benefits:

Simplification - Operators simplify the deployment and management of complex applications and services by providing a declarative way (via Custom Resources).
Higher Productivity - Operators increase the productivity of the platform engineering team by automating the routine operational tasks and allowing users to focus on delivering business value.
Extensibility - Extending the K8s API with Custom Resource Definitions (CRDs) allows administrators to support new applications and services without changing the core Kubernetes code.
Consistency - Operators ensure that they deploy and run the applications and services consistently across different environments, such as on-premises, cloud, or hybrid.
Modularity - Operators are modular and focused on a specific domain.
Legacy Transformation - Operators help to transform legacy applications into cloud-native applications by automating the deployment and management of complex applications.

Here is a visual representation of a Kubernetes Operator in action, which I found very nice and helpful:

Kubernetes Operators: what are they? Some examples Credit: SparkFabrik

Comparison of Kubernetes Controllers and Operators

Here is a quick summary of the differences between Kubernetes controllers and operators:

Feature	Kubernetes Controllers	Kubernetes Operators
Definition	Core control loops within Kubernetes that reconcile the current state with the desired state of resources.	Custom-built controllers with domain-specific logic, typically extending Kubernetes API with Custom Resource Definitions (CRDs).
Scope	General-purpose automation for cluster-wide resource management and workload operations.	Focused on application or service-specific lifecycle management and operations.
Custom Resources (CRDs)	Not mandatory. Most controllers work with built-in Kubernetes resources like Pods, Deployments, and StatefulSets.	Mandatory. Operators depend on CRDs to define and manage application-specific resources.
Domain-Specific Logic	Limited. Primarily focuses on generic Kubernetes operations like scheduling, scaling, and resource enforcement.	Extensive. Includes application-specific tasks such as backups, upgrades, migrations, and disaster recovery.
Automation	Automates operational tasks like resource scheduling, scaling, and maintaining cluster states.	Automates both operational and domain-specific tasks, often encoding deep operational knowledge of specific applications.
Complexity	Relatively low complexity as they focus on Kubernetes-native workflows.	Higher complexity due to domain-specific implementations and integrations with external systems.
Observability	Provides insight into cluster resource states via Kubernetes API and built-in events.	Enhances observability with application-specific metrics, logs, and health checks tailored to the managed application.
Use Cases	- Pod scaling (e.g., HPA) - Resource enforcement (e.g., ResourceQuota) - Rolling updates and rollbacks - Scheduling jobs (e.g., CronJob).	- Database management (e.g., MySQL Operator) - CI/CD pipelines (e.g., Tekton) - Backup and restore - Monitoring solutions (e.g., Prometheus Operator) - Stateful applications (e.g., Kafka Operator).
Deployment	Part of Kubernetes core components or extensions like kube-controller-manager.	External tools developed by vendors, community, or in-house, deployed using Custom Resources.
Key Benefits	- Cluster-wide automation - Ensures resource availability - Efficient and reliable workload management.	- Simplifies application lifecycle management - Increases productivity with domain-specific automation - Consistent deployments across environments.
Flexibility	Flexible for general Kubernetes operations but limited to built-in resources.	Highly flexible and extensible for custom applications and domain-specific requirements.
Target Audience	Platform administrators and Kubernetes users looking to automate Kubernetes-native workflows.	Developers and platform engineers needing advanced automation and lifecycle management for specific applications or services.
Maintenance	Managed and updated by Kubernetes core or community projects.	Requires domain-specific knowledge and regular updates to accommodate application changes and best practices.

With this detailed explanation of Kubernetes controllers and operators, I would like to share some advanced Kubernetes operators that I recommend for use in your Kubernetes-powered platform.

Advanced Kubernetes Operators

As we have seen, Kubernetes Operators are the go-to way to automate operational tasks and lifecycle management. I should mention that not all operators are created equal. Some operators are more advanced and provide more features than others. That is why, for example, OperatorHub.io categorises operators into five capability levels:

Basic Install - Installs the application on the cluster.
Seamless Upgrades - Manages Application upgrades.
Full Lifecycle Management - Manages the full lifecycle of the application.
Deep Insights - Provides monitoring and metrics.
Auto Pilot - Automatically optimizes and tunes the application.

The more advanced the operator, the more features it provides and the more automation it offers. At its most advanced level, the Auto Pilot, the operator automatically optimizes and tunes the application based on the workload and environment.

Here is a list of advanced Kubernetes operators that I recommend using in your Kubernetes-powered platform:

CloudNativePG Operator

CloudNativePG Operator Credit: cloudnative-pg.io

CloudNativePG Operator is a PostgreSQL operator that covers the full lifecycle of PostgreSQL databases on Kubernetes. It simplifies the deployment, management, and scaling of PostgreSQL databases by providing a declarative way to define and manage PostgreSQL clusters using custom resources.

Some of the key features of CloudNativePG Operator are:

Automated Backups - Schedule and manage backups of PostgreSQL databases.
High Availability - Deploy and manage high-availability PostgreSQL clusters.
Scaling - Scale the number of PostgreSQL instances based on the workload.

Installation

Either use the Helm chart or the Kubernetes manifests to deploy the CloudNativePG Operator:

helm repo add cloudnative-pg https://cloudnative-pg.io/charts/
helm repo update
helm upgrade --install cloudnativepg cloudnative-pg/cloudnative-pg --namespace cnpg-system --create-namespace

Or use Pulumi to deploy the CloudNativePG Operator:

import * as k8s from "@pulumi/kubernetes";
import * as k8shelm from "@pulumi/kubernetes/helm/v3";

const ns = new k8s.core.v1.Namespace("cnpg-system", {
 metadata: {
 name: "cnpg-system",
 },
});

new k8shelm.Release("cloudnativepg", {
 chart: "oci://ghcr.io/cloudnative-pg/charts/cloudnative-pg",
 namespace: ns.metadata.name,
 createNamespace: true,
});

import pulumi_kubernetes as k8s
import pulumi_kubernetes.helm.v3 as k8shelm
import pulumi_kubernetes.meta.v1 as meta

ns = k8s.core.v1.Namespace(
 "cnpg-system",
 metadata=meta.ObjectMetaArgs(
 name="cnpg-system",
 ),
)

release = k8shelm.Release(
 "cloudnativepg",
 chart="oci://ghcr.io/cloudnative-pg/charts/cloudnative-pg",
 namespace=ns.metadata["name"],
 create_namespace=True,
)

package main

import (
 k8s "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes"
 apiv1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/apiextensions"
 corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/core/v1"
 helmv3 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/helm/v3"
 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/meta/v1"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {

 ns, err := corev1.NewNamespace(ctx, "cnpg-system", &corev1.NamespaceArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("cnpg-system"),
 },
 })
 if err != nil {
 return err
 }

 _, err = helmv3.NewRelease(ctx, "cloudnativepg", &helmv3.ReleaseArgs{
 Chart: pulumi.String("oci://ghcr.io/cloudnative-pg/charts/cloudnative-pg"),
 Namespace: ns.Metadata.Name().Elem(),
 CreateNamespace: pulumi.Bool(true),
 })
 if err != nil {
 return err
 }
 return nil
 })
}

using System.Threading.Tasks;
using Pulumi;
using Pulumi.Kubernetes.Core.V1;
using Pulumi.Kubernetes.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Core.V1;
using Pulumi.Kubernetes.Types.Inputs.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Meta.V1;
using System.Collections.Generic;
using Pulumi.Kubernetes.ApiExtensions;

class MyStack : Stack
{
 public MyStack()
 {
 var ns = new Namespace("cnpg-system", new NamespaceArgs
 {
 Metadata = new ObjectMetaArgs
 {
 Name = "cnpg-system"
 }
 });

 var release = new Release("cloudnativepg", new ReleaseArgs
 {
 Chart = "oci://ghcr.io/cloudnative-pg/charts/cloudnative-pg",
 Namespace = ns.Metadata.Apply(m => m.Name),
 CreateNamespace = true
 });

}

class Program
{
 static Task<int> Main() => Deployment.RunAsync<MyStack>();
}

Usage

Here is a simple example of a PostgreSQL cluster with backup configured on S3 API-compatible storage:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
 name: cluster-example-with-backup
spec:
 instances: 3
 primaryUpdateStrategy: unsupervised

 # Persistent storage configuration
 storage:
 storageClass: standard
 size: 1Gi

 # Backup properties
 # This assumes a local minio setup
 backup:
 barmanObjectStore:
 destinationPath: s3://backups/
 endpointURL: http://minio:9000
 s3Credentials:
 accessKeyId:
 name: minio
 key: ACCESS_KEY_ID
 secretAccessKey:
 name: minio
 key: ACCESS_SECRET_KEY
 wal:
 compression: gzip

Or use Pulumi:

new k8s.apiextensions.CustomResource("cluster-example-with-backup", {
 apiVersion: "postgresql.cnpg.io/v1",
 kind: "Cluster",
 metadata: {
 name: "cluster-example-with-backup",
 },
 spec: {
 instances: 3,
 primaryUpdateStrategy: "unsupervised",

 // Persistent storage configuration
 storage: {
 storageClass: "standard",
 size: "1Gi",
 },

 // Backup properties
 backup: {
 barmanObjectStore: {
 destinationPath: "s3://backups/",
 endpointURL: "http://minio:9000",
 s3Credentials: {
 accessKeyId: {
 name: "minio",
 key: "ACCESS_KEY_ID",
 },
 secretAccessKey: {
 name: "minio",
 key: "ACCESS_SECRET_KEY",
 },
 },
 wal: {
 compression: "gzip",
 },
 },
 },
 },
});

from pulumi_kubernetes import apiextensions

postgresql_cluster = apiextensions.CustomResource(
 "cluster-example-with-backup",
 api_version="postgresql.cnpg.io/v1",
 kind="Cluster",
 metadata=meta.ObjectMetaArgs(
 name="cluster-example-with-backup",
 ),
 spec={
 "instances": 3,
 "primaryUpdateStrategy": "unsupervised",
 # Persistent storage configuration
 "storage": {
 "storageClass": "standard",
 "size": "1Gi",
 },
 # Backup properties
 "backup": {
 "barmanObjectStore": {
 "destinationPath": "s3://backups/",
 "endpointURL": "http://minio:9000",
 "s3Credentials": {
 "accessKeyId": {
 "name": "minio",
 "key": "ACCESS_KEY_ID",
 },
 "secretAccessKey": {
 "name": "minio",
 "key": "ACCESS_SECRET_KEY",
 },
 },
 "wal": {
 "compression": "gzip",
 },
 },
 },
 },
)

package main

import (
 k8s "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes"
 apiv1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/apiextensions"
 corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/core/v1"
 helmv3 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/helm/v3"
 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/meta/v1"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {

 _, err = apiv1.NewCustomResource(ctx, "cluster-example-with-backup", &apiv1.CustomResourceArgs{
 ApiVersion: pulumi.String("postgresql.cnpg.io/v1"),
 Kind: pulumi.String("Cluster"),
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("cluster-example-with-backup"),
 },
 OtherFields: k8s.UntypedArgs{
 "instances": pulumi.Int(3),
 "primaryUpdateStrategy": pulumi.String("unsupervised"),
 "storage": pulumi.Map{
 "storageClass": pulumi.String("standard"),
 "size": pulumi.String("1Gi"),
 },
 "backup": pulumi.Map{
 "barmanObjectStore": pulumi.Map{
 "destinationPath": pulumi.String("s3://backups/"),
 "endpointURL": pulumi.String("http://minio:9000"),
 "s3Credentials": pulumi.Map{
 "accessKeyId": pulumi.Map{
 "name": pulumi.String("minio"),
 "key": pulumi.String("ACCESS_KEY_ID"),
 },
 "secretAccessKey": pulumi.Map{
 "name": pulumi.String("minio"),
 "key": pulumi.String("ACCESS_SECRET_KEY"),
 },
 },
 "wal": pulumi.Map{
 "compression": pulumi.String("gzip"),
 },
 },
 },
 },
 })
 if err != nil {
 return err
 }

 return nil
 })
}

using System.Threading.Tasks;
using Pulumi;
using Pulumi.Kubernetes.Core.V1;
using Pulumi.Kubernetes.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Core.V1;
using Pulumi.Kubernetes.Types.Inputs.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Meta.V1;
using System.Collections.Generic;
using Pulumi.Kubernetes.ApiExtensions;

class PostGreSQLClusterArgs : CustomResourceArgs
{
 [Input("postgresql.cnpg.io/v1")]
 public Input<string> ApiVersion { get; set; }
 [Input("Cluster")]
 public Input<string> Kind { get; set; }
 public ObjectMetaArgs Metadata { get; set; }
 public Dictionary<string, object> Others { get; set; }
 [Input("spec")]
 public InputMap<object> Spec { get; set; }

 public PostGreSQLClusterArgs(string apiVersion, string kind) : base(apiVersion, kind)
 {
 }
}
class MyStack : Stack
{
 public MyStack()
 {
 var postgresqlCluster = new Pulumi.Kubernetes.ApiExtensions.CustomResource("cluster-example-with-backup", new PostGreSQLClusterArgs("postgresql.cnpg.io/v1", "Cluster")
 {
 ApiVersion = "postgresql.cnpg.io/v1",
 Kind = "Cluster",
 Metadata = new ObjectMetaArgs
 {
 Name = "cluster-example-with-backup",
 },
 Spec = new InputMap<object>
 {
 { "instances", 3 },
 { "primaryUpdateStrategy", "unsupervised" },
 { "storage", new InputMap<object>
 {
 { "storageClass", "standard" },
 { "size", "1Gi" },
 }
 },
 { "backup", new InputMap<object>
 {
 { "barmanObjectStore", new InputMap<object>
 {
 { "destinationPath", "s3://backups/" },
 { "endpointURL", "http://minio:9000" },
 { "s3Credentials", new InputMap<object>
 {
 { "accessKeyId", new InputMap<object>
 {
 { "name", "minio" },
 { "key", "ACCESS_KEY_ID" },
 }
 },
 { "secretAccessKey", new InputMap<object>
 {
 { "name", "minio" },
 { "key", "ACCESS_SECRET_KEY" },
 }
 },
 }
 },
 { "wal", new InputMap<object>
 {
 { "compression", "gzip" },
 }
 },
 }
 },
 }
 },
 }
 });

}

class Program
{
 static Task<int> Main() => Deployment.RunAsync<MyStack>();
}

Summary

CloudNativePG Operator is a great way to deploy and manage PostgreSQL databases on Kubernetes. It simplifies a wide range of tasks, including backups, high availability, and scaling, making it a must-have for PostgreSQL users.

You might also like:

Flux Operator

Flux Operator Credit: fluxcd.control-plane.io/operator

The Flux Operator is a full autopilot solution that provides an alternative to the traditional Flux Bootstrap procedure, removing the operational burden of managing Flux across multiple clusters by automating the installation, configuration, and upgrade of Flux fully.

Some of the key features of Flux Operator are:

GitOps Deployment - Automate the deployment of applications and services using GitOps.
Lifecycle Management - Manage the full lifecycle of Flux, including upgrades and rollbacks.
Deep Insights - Provide deep insights into the state of Flux and the applications deployed with Flux.

Installation

Use the Helm chart to deploy the Flux Operator:

helm upgrade -i flux-operator oci://ghcr.io/controlplaneio-fluxcd/charts/flux-operator --namespace flux-system --create-namespace

Or use Pulumi to deploy the Flux Operator:

import * as k8s from "@pulumi/kubernetes";
import * as k8shelm from "@pulumi/kubernetes/helm/v3";

const namespace = new k8s.core.v1.Namespace("flux-system", {
 metadata: {
 name: "flux-system",
 },
});

new k8s.helm.v3.Release("flux-operator", {
 chart: "oci://ghcr.io/controlplaneio-fluxcd/charts/flux-operator",
 namespace: namespace.metadata.name,
 createNamespace: true,
});

import pulumi_kubernetes as k8s
import pulumi_kubernetes.helm.v3 as k8shelm
import pulumi_kubernetes.meta.v1 as meta

flux_namespace = k8s.core.v1.Namespace(
 "flux-system",
 metadata=meta.ObjectMetaArgs(
 name="flux-system",
 ),
)
flux_operator = k8shelm.Release(
 "flux-operator",
 chart="oci://ghcr.io/controlplaneio-fluxcd/charts/flux-operator",
 namespace=flux_namespace.metadata["name"],
 create_namespace=True,
)

package main

import (
 k8s "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes"
 apiv1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/apiextensions"
 corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/core/v1"
 helmv3 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/helm/v3"
 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/meta/v1"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {

 namespace, err := corev1.NewNamespace(ctx, "flux-system", &corev1.NamespaceArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("flux-system"),
 },
 })
 if err != nil {
 return err
 }

 _, err = helmv3.NewRelease(ctx, "flux-operator", &helmv3.ReleaseArgs{
 Chart: pulumi.String("oci://ghcr.io/controlplaneio-fluxcd/charts/flux-operator"),
 Namespace: namespace.Metadata.Name().Elem(),
 CreateNamespace: pulumi.Bool(true),
 })
 if err != nil {
 return err
 }
 return nil
 })
}

using System.Threading.Tasks;
using Pulumi;
using Pulumi.Kubernetes.Core.V1;
using Pulumi.Kubernetes.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Core.V1;
using Pulumi.Kubernetes.Types.Inputs.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Meta.V1;
using System.Collections.Generic;
using Pulumi.Kubernetes.ApiExtensions;

 var namespaceFluxSystem = new Namespace("flux-system", new NamespaceArgs
 {
 Metadata = new ObjectMetaArgs
 {
 Name = "flux-system",
 },
 });

 var fluxOperator = new Release("flux-operator", new ReleaseArgs
 {
 Chart = "oci://ghcr.io/controlplaneio-fluxcd/charts/flux-operator",
 Namespace = namespaceFluxSystem.Metadata.Apply(metadata => metadata.Name),
 CreateNamespace = true,
 });
}

class Program
{
 static Task<int> Main() => Deployment.RunAsync<MyStack>();
}

Usage

Here is an example of a FluxInstance Custom Resource that defines a Flux instance:

apiVersion: fluxcd.controlplane.io/v1
kind: FluxInstance
metadata:
 name: flux
 namespace: flux-system
 annotations:
 fluxcd.controlplane.io/reconcileEvery: "1h"
 fluxcd.controlplane.io/reconcileTimeout: "5m"
spec:
 distribution:
 version: "2.x"
 registry: "ghcr.io/fluxcd"
 artifact: "oci://ghcr.io/controlplaneio-fluxcd/flux-operator-manifests"
 components:
 - source-controller
 - kustomize-controller
 - helm-controller
 - notification-controller
 - image-reflector-controller
 - image-automation-controller
 cluster:
 type: kubernetes
 multitenant: false
 networkPolicy: true
 domain: "cluster.local"
 kustomize:
 patches:
 - target:
 kind: Deployment
 name: "(kustomize-controller|helm-controller)"
 patch: |
 - op: add
 path: /spec/template/spec/containers/0/args/-
 value: --concurrent=10
 - op: add
 path: /spec/template/spec/containers/0/args/-
 value: --requeue-dependency=5s

Or use Pulumi:

new k8s.apiextensions.CustomResource("flux", {
 apiVersion: "fluxcd.controlplane.io/v1",
 kind: "FluxInstance",
 metadata: {
 name: "flux",
 namespace: "flux-system",
 annotations: {
 "fluxcd.controlplane.io/reconcileEvery": "1h",
 "fluxcd.controlplane.io/reconcileTimeout": "5m",
 },
 },
 spec: {
 distribution: {
 version: "2.x",
 registry: "ghcr.io/fluxcd",
 artifact: "oci://ghcr.io/controlplaneio-fluxcd/flux-operator-manifests",
 },
 components: ["source-controller", "kustomize-controller", "helm-controller", "notification-controller", "image-reflector-controller", "image-automation-controller"],
 cluster: {
 type: "kubernetes",
 multitenant: false,
 networkPolicy: true,
 domain: "cluster.local",
 },
 kustomize: {
 patches: [
 {
 target: {
 kind: "Deployment",
 name: "(kustomize-controller|helm-controller)",
 },
 patch: `
 - op: add
 path: /spec/template/spec/containers/0/args/-
 value: --concurrent=10
 - op: add
 path: /spec/template/spec/containers/0/args/-
 value: --requeue-dependency=5s
 `,
 },
 ],
 },
 },
});

from pulumi_kubernetes import apiextensions

flux_instance = k8s.apiextensions.CustomResource(
 "flux",
 api_version="fluxcd.controlplane.io/v1",
 kind="FluxInstance",
 metadata={
 "name": "flux",
 "namespace": flux_namespace.metadata["name"],
 "annotations": {
 "fluxcd.controlplane.io/reconcileEvery": "1h",
 "fluxcd.controlplane.io/reconcileTimeout": "5m",
 },
 },
 spec={
 "distribution": {
 "version": "2.x",
 "registry": "ghcr.io/fluxcd",
 "artifact": "oci://ghcr.io/controlplaneio-fluxcd/flux-operator-manifests",
 },
 "components": [
 "source-controller",
 "kustomize-controller",
 "helm-controller",
 "notification-controller",
 "image-reflector-controller",
 "image-automation-controller",
 ],
 "cluster": {
 "type": "kubernetes",
 "multitenant": False,
 "networkPolicy": True,
 "domain": "cluster.local",
 },
 "kustomize": {
 "patches": [
 {
 "target": {
 "kind": "Deployment",
 "name": "(kustomize-controller|helm-controller)",
 },
 "patch": [
 {
 "op": "add",
 "path": "/spec/template/spec/containers/0/args/-",
 "value": "--concurrent=10",
 },
 {
 "op": "add",
 "path": "/spec/template/spec/containers/0/args/-",
 "value": "--requeue-dependency=5s",
 },
 ],
 }
 ]
 },
 },
)

package main

import (
 k8s "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes"
 apiv1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/apiextensions"
 corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/core/v1"
 helmv3 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/helm/v3"
 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/meta/v1"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {

 _, err = apiv1.NewCustomResource(ctx, "flux", &apiv1.CustomResourceArgs{
 ApiVersion: pulumi.String("fluxcd.controlplane.io/v1"),
 Kind: pulumi.String("FluxInstance"),
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("flux"),
 Namespace: pulumi.String("flux-system"),
 Annotations: pulumi.StringMap{
 "fluxcd.controlplane.io/reconcileEvery": pulumi.String("1h"),
 "fluxcd.controlplane.io/reconcileTimeout": pulumi.String("5m"),
 },
 },
 OtherFields: k8s.UntypedArgs{
 "distribution": pulumi.Map{
 "version": pulumi.String("2.x"),
 "registry": pulumi.String("ghcr.io/fluxcd"),
 "artifact": pulumi.String("oci://ghcr.io/controlplaneio-fluxcd/flux-operator-manifests"),
 },
 "components": pulumi.StringArray{
 pulumi.String("source-controller"),
 pulumi.String("kustomize-controller"),
 pulumi.String("helm-controller"),
 pulumi.String("notification-controller"),
 pulumi.String("image-reflector-controller"),
 pulumi.String("image-automation-controller"),
 },
 "cluster": pulumi.Map{
 "type": pulumi.String("kubernetes"),
 "multitenant": pulumi.Bool(false),
 "networkPolicy": pulumi.Bool(true),
 "domain": pulumi.String("cluster.local"),
 },
 "kustomize": pulumi.Map{
 "patches": pulumi.Array{
 pulumi.Map{
 "target": pulumi.Map{
 "kind": pulumi.String("Deployment"),
 "name": pulumi.String("(kustomize-controller|helm-controller)"),
 },
 "patch": pulumi.Array{
 pulumi.Map{
 "op": pulumi.String("add"),
 "path": pulumi.String("/spec/template/spec/containers/0/args/-"),
 "value": pulumi.String("--concurrent=10"),
 },
 pulumi.Map{
 "op": pulumi.String("add"),
 "path": pulumi.String("/spec/template/spec/containers/0/args/-"),
 "value": pulumi.String("--requeue-dependency=5s"),
 },
 },
 },
 },
 },
 },
 })
 if err != nil {
 return err
 }

 return nil
 })
}

using System.Threading.Tasks;
using Pulumi;
using Pulumi.Kubernetes.Core.V1;
using Pulumi.Kubernetes.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Core.V1;
using Pulumi.Kubernetes.Types.Inputs.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Meta.V1;
using System.Collections.Generic;
using Pulumi.Kubernetes.ApiExtensions;

class FluxInstanceArgs : CustomResourceArgs
{
 public string ApiVersion { get; set; }
 public string Kind { get; set; }
 public ObjectMetaArgs Metadata { get; set; }
 public Dictionary<string, object> Spec { get; set; }

 public FluxInstanceArgs(string apiVersion, string kind) : base(apiVersion, kind)
 {
 }
}
 var flux = new Pulumi.Kubernetes.ApiExtensions.CustomResource("flux", new FluxInstanceArgs("fluxcd.controlplane.io/v1", "FluxInstance")
 {
 ApiVersion = "fluxcd.controlplane.io/v1",
 Kind = "FluxInstance",
 Metadata = new ObjectMetaArgs
 {
 Name = "flux",
 Annotations =
 {
 { "fluxcd.controlplane.io/reconcileEvery", "1h" },
 { "fluxcd.controlplane.io/reconcileTimeout", "5m" },
 }
 },
 Spec = new Dictionary<string, object>
 {
 { "distribution", new Dictionary<string, object>
 {
 { "version", "2.x" },
 { "registry", "ghcr.io/fluxcd" },
 { "artifact", "oci://ghcr.io/controlplaneio-fluxcd/flux-operator-manifests" }
 }
 },
 { "components", new[]
 {
 "source-controller",
 "kustomize-controller",
 "helm-controller",
 "notification-controller",
 "image-reflector-controller",
 "image-automation-controller"
 }
 },
 { "cluster", new Dictionary<string, object>
 {
 { "type", "kubernetes" },
 { "multitenant", false },
 { "networkPolicy", true },
 { "domain", "cluster.local" }
 }
 },
 { "kustomize", new Dictionary<string, object>
 {
 { "patches", new[]
 {
 new Dictionary<string, object>
 {
 { "target", new Dictionary<string, object>
 {
 { "kind", "Deployment" },
 { "name", "(kustomize-controller|helm-controller)" }
 }
 },
 { "patch", new[]
 {
 new Dictionary<string, object>
 {
 { "op", "add" },
 { "path", "/spec/template/spec/containers/0/args/-" },
 { "value", "--concurrent=10" }
 },
 new Dictionary<string, object>
 {
 { "op", "add" },
 { "path", "/spec/template/spec/containers/0/args/-" },
 { "value", "--requeue-dependency=5s" }
 }
 }
 }
 }
 }
 }
 }
 }
 },
 });

}

class Program
{
 static Task<int> Main() => Deployment.RunAsync<MyStack>();
}

Summary

The Flux Operator is another excellent example of an advanced Kubernetes operator, providing full lifecycle management with numerous automation features.

Strimzi Operator

Strimzi Operator Credit: strimzi.io

Last but not least, the Strimzi Operator is a full lifecycle management operator for Apache Kafka on Kubernetes. As event-driven architectures are becoming increasingly popular, Apache Kafka has become the de-facto standard for building scalable and reliable event streaming platforms. The Strimzi Operator makes it easier to deploy, manage, and scale Apache Kafka clusters on Kubernetes.

Some of the key features of Strimzi Operator are:

Cluster Management - Deploy and manage Apache Kafka clusters.
Mirror Clusters - Deploy and manage mirror clusters for disaster recovery.
Streaming Capabilities - With Kafka Connect you can stream data between Kafka and other systems.

Installation

Use the Helm chart to deploy the Strimzi Operator:

helm repo add strimzi https://strimzi.io/charts/
helm repo update
helm install strimzi strimzi/strimzi-kafka-operator -n kafka --create-namespace

Or use Pulumi to deploy the Strimzi Operator:

import * as k8s from "@pulumi/kubernetes";
import * as k8shelm from "@pulumi/kubernetes/helm/v3";

const namespaceKafka = new k8s.core.v1.Namespace("kafka", {
 metadata: {
 name: "kafka",
 },
});

new k8s.helm.v3.Release("strimzi-kafka-operator", {
 chart: "strimzi-kafka-operator",
 namespace: namespaceKafka.metadata.name,
 repositoryOpts: {
 repo: "https://strimzi.io/charts/",
 },
 createNamespace: true,
});

import pulumi_kubernetes as k8s
import pulumi_kubernetes.helm.v3 as k8shelm
import pulumi_kubernetes.meta.v1 as meta

kafka_namespace = k8s.core.v1.Namespace(
 "kafka",
 metadata=meta.ObjectMetaArgs(
 name="kafka",
 ),
)

strimzi_kafka_operator = k8shelm.Release(
 "strimzi-kafka-operator",
 chart="strimzi-kafka-operator",
 namespace=kafka_namespace.metadata["name"],
 repository_opts={
 "repo": "https://strimzi.io/charts/",
 },
 create_namespace=True,
)

package main

import (
 k8s "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes"
 apiv1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/apiextensions"
 corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/core/v1"
 helmv3 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/helm/v3"
 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/meta/v1"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 namespaceKafka, err := corev1.NewNamespace(ctx, "kafka", &corev1.NamespaceArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("kafka"),
 },
 })
 if err != nil {
 return err
 }

 _, err = helmv3.NewRelease(ctx, "strimzi-kafka-operator", &helmv3.ReleaseArgs{
 Chart: pulumi.String("strimzi-kafka-operator"),
 Namespace: namespaceKafka.Metadata.Name(),
 RepositoryOpts: helmv3.RepositoryOptsArgs{
 Repo: pulumi.String("https://strimzi.io/charts/"),
 },
 CreateNamespace: pulumi.Bool(true),
 })
 if err != nil {
 return err
 }

 return nil
 })
}

using System.Threading.Tasks;
using Pulumi;
using Pulumi.Kubernetes.Core.V1;
using Pulumi.Kubernetes.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Core.V1;
using Pulumi.Kubernetes.Types.Inputs.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Meta.V1;
using System.Collections.Generic;
using Pulumi.Kubernetes.ApiExtensions;
class MyStack : Stack
{
 public MyStack()
 {
 var namespaceKafka = new Namespace("kafka", new NamespaceArgs
 {
 Metadata = new ObjectMetaArgs
 {
 Name = "kafka",
 }
 });

 var strimziKafkaOperator = new Release("strimzi-kafka-operator", new ReleaseArgs
 {
 Chart = "strimzi-kafka-operator",
 Namespace = namespaceKafka.Metadata.Apply(ns => ns.Name),
 RepositoryOpts = new RepositoryOptsArgs
 {
 Repo = "https://strimzi.io/charts/",
 },
 CreateNamespace = true,
 });

}

class Program
{
 static Task<int> Main() => Deployment.RunAsync<MyStack>();
}

Usage

Here is an example of a Kafka Custom Resource that defines a Kafka cluster:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
 name: example-kafka-node-pool
 labels:
 strimzi.io/cluster: example-kafka-cluster
spec:
 replicas: 1
 roles:
 - controller
 - broker
 storage:
 type: ephemeral
---
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
 name: example-kafka-cluster
 annotations:
 strimzi.io/kraft: enabled
 strimzi.io/node-pools: enabled
spec:
 kafka:
 replicas: 1
 version: 3.8.0
 storage:
 type: ephemeral
 metadataVersion: 3.8-IV0
 listeners:
 - name: plain
 port: 9092
 type: internal
 tls: false
 - name: tls
 port: 9093
 type: internal
 tls: true
 config:
 offsets.topic.replication.factor: 1
 transaction.state.log.replication.factor: 1
 transaction.state.log.min.isr: 1
 default.replication.factor: 1
 min.insync.replicas: 1
 entityOperator:
 topicOperator: {}
 userOperator: {}

This example defines a Kafka cluster with one replica and two listeners, one for plain and one for TLS and uses ephemeral storage.

To create a topic, you can use the following KafkaTopic Custom Resource:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
 name: example-kafka-cluster
 labels:
 strimzi.io/cluster:
spec:
 partitions: 5
 replicas: 1
 config:
 retention.ms: 7200000
 segment.bytes: 1073741824

Or use Pulumi:

new k8s.apiextensions.CustomResource("example-kafka-node-pool", {
 apiVersion: "kafka.strimzi.io/v1beta2",
 kind: "KafkaNodePool",
 metadata: {
 name: "example-kafka-node-pool",
 labels: {
 "strimzi.io/cluster": "example-kafka-cluster",
 },
 namespace: "kafka",
 },
 spec: {
 replicas: 1,
 roles: ["controller", "broker"],
 storage: {
 type: "ephemeral",
 },
 },
});

new k8s.apiextensions.CustomResource("example-kafka-cluster", {
 apiVersion: "kafka.strimzi.io/v1beta2",
 kind: "Kafka",
 metadata: {
 name: "example-kafka-cluster",
 annotations: {
 "strimzi.io/kraft": "enabled",
 "strimzi.io/node-pools": "enabled",
 },
 namespace: "kafka",
 },
 spec: {
 kafka: {
 replicas: 1,
 version: "3.8.0",
 storage: {
 type: "ephemeral",
 },
 metadataVersion: "3.8-IV0",
 listeners: [
 {
 name: "plain",
 port: 9092,
 type: "internal",
 tls: false,
 },
 {
 name: "tls",
 port: 9093,
 type: "internal",
 tls: true,
 },
 ],
 config: {
 "offsets.topic.replication.factor": 1,
 "transaction.state.log.replication.factor": 1,
 "transaction.state.log.min.isr": 1,
 "default.replication.factor": 1,
 "min.insync.replicas": 1,
 },
 },
 entityOperator: {
 topicOperator: {},
 userOperator: {},
 },
 },
});

new k8s.apiextensions.CustomResource("example-kafka-topic", {
 apiVersion: "kafka.strimzi.io/v1beta2",
 kind: "KafkaTopic",
 metadata: {
 name: "example-kafka-topic",
 labels: {
 "strimzi.io/cluster": "example-kafka-cluster",
 },
 namespace: "kafka",
 },
 spec: {
 partitions: 5,
 replicas: 1,
 config: {
 "retention.ms": 7200000,
 "segment.bytes": 1073741824,
 },
 },
});

from pulumi_kubernetes import apiextensions

kafka_node_pool = k8s.apiextensions.CustomResource(
 "example-kafka-node-pool",
 api_version="kafka.strimzi.io/v1beta2",
 kind="KafkaNodePool",
 metadata={
 "name": "example-kafka-node-pool",
 "labels": {
 "strimzi.io/cluster": "example-kafka-cluster",
 },
 "namespace": kafka_namespace.metadata["name"],
 },
 spec={
 "replicas": 1,
 "roles": ["controller", "broker"],
 "storage": {
 "type": "ephemeral",
 },
 },
)

kafka_cluster = k8s.apiextensions.CustomResource(
 "example-kafka-cluster",
 api_version="kafka.strimzi.io/v1beta2",
 kind="Kafka",
 metadata={
 "name": "example-kafka-cluster",
 "annotations": {
 "strimzi.io/kraft": "enabled",
 "strimzi.io/node-pools": "enabled",
 },
 "namespace": kafka_namespace.metadata["name"],
 },
 spec={
 "kafka": {
 "replicas": 1,
 "version": "3.8.0",
 "storage": {
 "type": "ephemeral",
 },
 "metadataVersion": "3.8-IV0",
 "listeners": [
 {
 "name": "plain",
 "port": 9092,
 "type": "internal",
 "tls": False,
 },
 {
 "name": "tls",
 "port": 9093,
 "type": "internal",
 "tls": True,
 },
 ],
 "config": {
 "offsets.topic.replication.factor": 1,
 "transaction.state.log.replication.factor": 1,
 "transaction.state.log.min.isr": 1,
 "default.replication.factor": 1,
 "min.insync.replicas": 1,
 },
 },
 "entityOperator": {
 "topicOperator": {},
 "userOperator": {},
 },
 },
)

kafka_topic = k8s.apiextensions.CustomResource(
 "example-kafka-topic",
 api_version="kafka.strimzi.io/v1beta2",
 kind="KafkaTopic",
 metadata={
 "name": "example-kafka-topic",
 "labels": {
 "strimzi.io/cluster": "example-kafka-cluster",
 },
 "namespace": kafka_namespace.metadata["name"],
 },
 spec={
 "partitions": 5,
 "replicas": 1,
 "config": {
 "retention.ms": 7200000,
 "segment.bytes": 1073741824,
 },
 },
)

package main

import (
 k8s "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes"
 apiv1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/apiextensions"
 corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/core/v1"
 helmv3 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/helm/v3"
 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/meta/v1"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {

 _, err = apiv1.NewCustomResource(ctx, "example-kafka-node-pool", &apiv1.CustomResourceArgs{
 ApiVersion: pulumi.String("kafka.strimzi.io/v1beta2"),
 Kind: pulumi.String("KafkaNodePool"),
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("example-kafka-node-pool"),
 Namespace: pulumi.String("kafka"),
 Labels: pulumi.StringMap{
 "strimzi.io/cluster": pulumi.String("example-kafka-cluster"),
 },
 },
 OtherFields: k8s.UntypedArgs{
 "replicas": pulumi.Int(1),
 "roles": pulumi.StringArray{
 pulumi.String("controller"),
 pulumi.String("broker"),
 },
 "storage": pulumi.Map{
 "type": pulumi.String("ephemeral"),
 },
 },
 })
 if err != nil {
 return err
 }

 _, err = apiv1.NewCustomResource(ctx, "example-kafka-cluster", &apiv1.CustomResourceArgs{
 ApiVersion: pulumi.String("kafka.strimzi.io/v1beta2"),
 Kind: pulumi.String("Kafka"),
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("example-kafka-cluster"),
 Namespace: pulumi.String("kafka"),
 Annotations: pulumi.StringMap{
 "strimzi.io/kraft": pulumi.String("enabled"),
 "strimzi.io/node-pools": pulumi.String("enabled"),
 },
 },
 OtherFields: k8s.UntypedArgs{
 "kafka": pulumi.Map{
 "replicas": pulumi.Int(1),
 "version": pulumi.String("3.8.0"),
 "storage": pulumi.Map{
 "type": pulumi.String("ephemeral"),
 },
 "metadataVersion": pulumi.String("3.8-IV0"),
 "listeners": pulumi.Array{
 pulumi.Map{
 "name": pulumi.String("plain"),
 "port": pulumi.Int(9092),
 "type": pulumi.String("internal"),
 "tls": pulumi.Bool(false),
 },
 pulumi.Map{
 "name": pulumi.String("tls"),
 "port": pulumi.Int(9093),
 "type": pulumi.String("internal"),
 "tls": pulumi.Bool(true),
 },
 },
 "config": pulumi.Map{
 "offsets.topic.replication.factor": pulumi.Int(1),
 "transaction.state.log.replication.factor": pulumi.Int(1),
 "transaction.state.log.min.isr": pulumi.Int(1),
 "default.replication.factor": pulumi.Int(1),
 "min.insync.replicas": pulumi.Int(1),
 },
 },
 "entityOperator": pulumi.Map{
 "topicOperator": pulumi.Map{},
 "userOperator": pulumi.Map{},
 },
 },
 })
 if err != nil {
 return err
 }

 _, err = apiv1.NewCustomResource(ctx, "example-kafka-topic", &apiv1.CustomResourceArgs{
 ApiVersion: pulumi.String("kafka.strimzi.io/v1beta2"),
 Kind: pulumi.String("KafkaTopic"),
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("example-kafka-topic"),
 Namespace: pulumi.String("kafka"),
 Labels: pulumi.StringMap{
 "strimzi.io/cluster": pulumi.String("example-kafka-cluster"),
 },
 },
 OtherFields: k8s.UntypedArgs{
 "partitions": pulumi.Int(5),
 "replicas": pulumi.Int(1),
 "config": pulumi.Map{
 "retention.ms": pulumi.Int(7200000),
 "segment.bytes": pulumi.Int(1073741824),
 },
 },
 })
 if err != nil {
 return err
 }

 return nil
 })
}

using System.Threading.Tasks;
using Pulumi;
using Pulumi.Kubernetes.Core.V1;
using Pulumi.Kubernetes.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Core.V1;
using Pulumi.Kubernetes.Types.Inputs.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Meta.V1;
using System.Collections.Generic;
using Pulumi.Kubernetes.ApiExtensions;

class KafkaNodePoolArgs : CustomResourceArgs
{
 public string ApiVersion { get; set; }
 public string Kind { get; set; }
 public ObjectMetaArgs Metadata { get; set; }
 public Dictionary<string, object> Spec { get; set; }

 public KafkaNodePoolArgs(string apiVersion, string kind) : base(apiVersion, kind)
 {
 }
}

class KafkaClusterArgs : CustomResourceArgs
{
 public string ApiVersion { get; set; }
 public string Kind { get; set; }
 public ObjectMetaArgs Metadata { get; set; }
 public Dictionary<string, object> Spec { get; set; }

 public KafkaClusterArgs(string apiVersion, string kind) : base(apiVersion, kind)
 {
 }
}

class KafkaTopicArgs : CustomResourceArgs
{
 public string ApiVersion { get; set; }
 public string Kind { get; set; }
 public ObjectMetaArgs Metadata { get; set; }
 public Dictionary<string, object> Spec { get; set; }

 public KafkaTopicArgs(string apiVersion, string kind) : base(apiVersion, kind)
 {
 }
}

class MyStack : Stack
{
 public MyStack()
 {
 var kafkaNodePool = new Pulumi.Kubernetes.ApiExtensions.CustomResource("exampleKafkaNodePool", new KafkaNodePoolArgs("kafka.strimzi.io/v1beta2", "KafkaNodePool")
 {
 ApiVersion = "kafka.strimzi.io/v1beta2",
 Kind = "KafkaNodePool",
 Metadata = new ObjectMetaArgs
 {
 Name = "example-kafka-node-pool",
 Labels =
 {
 { "strimzi.io/cluster", "example-kafka-cluster" }
 }
 },
 Spec = new Dictionary<string, object>
 {
 { "replicas", 1 },
 { "roles", new List<string> { "controller", "broker" } },
 { "storage", new Dictionary<string, object> { { "type", "ephemeral" } } }
 }
 });

 var kafkaCluster = new Pulumi.Kubernetes.ApiExtensions.CustomResource("exampleKafkaCluster", new KafkaClusterArgs("kafka.strimzi.io/v1beta2", "Kafka")
 {
 ApiVersion = "kafka.strimzi.io/v1beta2",
 Kind = "Kafka",
 Metadata = new ObjectMetaArgs
 {
 Name = "example-kafka-cluster",
 Annotations =
 {
 { "strimzi.io/kraft", "enabled" },
 { "strimzi.io/node-pools", "enabled" }
 }
 },
 Spec = new Dictionary<string, object>
 {
 { "kafka", new Dictionary<string, object> {
 { "replicas", 1 },
 { "version", "3.8.0" },
 { "storage", new Dictionary<string, object> { { "type", "ephemeral" } } },
 { "metadataVersion", "3.8-IV0" },
 { "listeners", new List<Dictionary<string, object>> {
 new Dictionary<string, object> {
 { "name", "plain" },
 { "port", 9092 },
 { "type", "internal" },
 { "tls", false }
 },
 new Dictionary<string, object> {
 { "name", "tls" },
 { "port", 9093 },
 { "type", "internal" },
 { "tls", true }
 }
 }},
 { "config", new Dictionary<string, object> {
 { "offsets.topic.replication.factor", 1 },
 { "transaction.state.log.replication.factor", 1 },
 { "transaction.state.log.min.isr", 1 },
 { "default.replication.factor", 1 },
 { "min.insync.replicas", 1 }
 }}
 }},
 { "entityOperator", new Dictionary<string, object> {
 { "topicOperator", new Dictionary<string, object>() },
 { "userOperator", new Dictionary<string, object>() }
 }}
 }
 });

 var kafkaTopic = new Pulumi.Kubernetes.ApiExtensions.CustomResource("exampleKafkaTopic", new KafkaTopicArgs("kafka.strimzi.io/v1beta2", "KafkaTopic")
 {
 ApiVersion = "kafka.strimzi.io/v1beta2",
 Kind = "KafkaTopic",
 Metadata = new ObjectMetaArgs
 {
 Name = "example-kafka-topic",
 Labels =
 {
 { "strimzi.io/cluster", "example-kafka-cluster" }
 }
 },
 Spec = new Dictionary<string, object>
 {
 { "partitions", 5 },
 { "replicas", 1 },
 { "config", new Dictionary<string, object> {
 { "retention.ms", 7200000 },
 { "segment.bytes", 1073741824 }
 }}
 }
 });
}

class Program
{
 static Task<int> Main() => Deployment.RunAsync<MyStack>();
}

Summary

The Strimzi Operator demonstrates the power of Kubernetes Operators by providing full lifecycle management for Apache Kafka on Kubernetes. It takes care of the heavy lifting of deploying and managing Kafka clusters, making the Strimzi Operator a valuable tool that should be part of every Kubernetes-powered platform, as it covers a wide range of use cases.

Conclusion

In this blog post, I highlighted the importance of Kubernetes Operators for platform engineers and how they can harness the advanced automation capabilities of operators to simplify deploying, managing, and scaling applications and services on Kubernetes.

I provided some examples of advanced Kubernetes operators that I recommend, but if you want to try using Kubernetes Operators in your Kubernetes-powered platform, here are some great guides to get you started with Pulumi and Kubernetes:

Get Started with Pulumi

Use Pulumi's open-source SDK to create, deploy, and manage infrastructure on any cloud.

YAML, Terraform, Pulumi: What’s the Smart Choice for Deployment Automation with Kubernetes?

Engin Diri — Mon, 18 Nov 2024 08:00:00 +0100

YAML and Kubernetes go together like peanut butter and jelly. While Kubernetes objects can be defined in JSON, YAML has emerged as the de facto standard.

It’s often the first tool developers encounter when diving into Kubernetes, and for good reason - its human-readable format makes it the preferred choice in most tutorials, documentation, and even production deployments.

But is it the best way to manage your Kubernetes resources? Let’s compare YAML, Terraform, and Pulumi and see which is best for managing your Kubernetes resources.

I also gave a talk before on No YAML! and what alternatives you have if you would only stick with the Go language:

Kubernetes Components and the Kubernetes API

Before we discuss the ways to manage Kubernetes objects, let’s look at the components of a typical Kubernetes cluster and their interactions.

Kubernetes Components. Credit: kubernetes.io

A Kubernetes cluster has two component categories: control plane and node. The control plane components manage the cluster. They consist of:

kube-apiserver - The API server is the front end for the Kubernetes control plane. It exposes the Kubernetes API.
etcd - A consistent, highly available key-value store. It is Kubernetes’ backing store for all API data.
kube-scheduler - Watches for newly created Pods with no assigned node and selects a node for them to run on.
kube-controller-manager - Runs a controller to implement the control loop.
cloud-controller-manager - Runs controllers that interact with the underlying cloud providers.

The node components run on every node in the cluster. They provide the runtime environment for the containers.

kubelet - An agent that runs on each node in the cluster. It ensures that pods are running.
kube-proxy - Maintains network rules on nodes.
Container Runtime - Is responsible for running containers.

The Kubernetes API is the central component in a Kubernetes cluster. It is the interface to the cluster and the only way to work with Kubernetes objects.

A Kubernetes object is a “record of intent”. It represents the desired state of the workload running on the cluster. The whole Kubernetes system will continuously work to ensure that the object exists.

YAML Manifests

If you are long enough exposed to Kubernetes, you will have seen a lot of YAML files. You can define many Kubernetes objects in a YAML file. These include Deployments, Services, ConfigMaps, Secrets, and more.

Let’s have a look at an example of a Deployment object in YAML for an Nginx web server:

Example

apiVersion: apps/v1
kind: Deployment
metadata:
 name: nginx-deployment
spec:
 replicas: 1
 selector:
 matchLabels:
 app: nginx
 template:
 metadata:
 labels:
 app: nginx
 spec:
 containers:
 - name: "nginx:latest"
 image: nginx

Pros and Cons of YAML

Pros

Simple: Kubernetes YAML files are easy to read and write.
The standard: YAML is the standard way to define Kubernetes objects. It is a first-class citizen in the kubectl command-line tool.

Cons

Multiple environments: If you have multiple environments, like development, staging, and production, you need to maintain multiple YAML files.
Too manual: You must create and update the YAML files by hand. Also, you must track changes across many files.
Not DRY: You cannot use the DRY principle (Don’t Repeat Yourself) in YAML files. If you have the same configuration in many files, you need to update all of them by hand.
No real abstraction: The advantage of a static configuration file is also its disadvantage. You cannot use dynamic values in the YAML files.
Need for tools from the ecosystem: You need to use tools like helm or kustomize to manage the complexity of many YAML files, environments, and templating.

To sum it up, YAML is a good choice for simple Kubernetes objects and small projects. But, if you have a complex project, and to be honest, most are, you need a better way to manage your Kubernetes resources.

Or to rephrase it: Friends don’t let friends manage Kubernetes resources with YAML.

Why should you avoid YAML? Credit: No YAML!

Terraform Kubernetes Provider

Terraform is an Infrastructure as Code tool. It lets you define and manage your infrastructure in a declarative way. It uses a high-level configuration language called HashiCorp Configuration Language (HCL). To manage Kubernetes objects with Terraform, use the Kubernetes provider. It allows you to define Kubernetes objects in Terraform HCL.

The Kubernetes provider allows you to define Kubernetes objects like kubernetes_deployment, kubernetes_service, kubernetes_config_map, kubernetes_secret, and more in code, leveraging the language features of Terraform. You can use, to a certain extent, loops, conditions, and variables in a way that is not possible with plain YAML files.

Let’s have a look at an example of a Deployment object in Terraform HCL for an Nginx web server:

Example

resource "kubernetes_deployment" "nginx" {
 metadata {
 name = "nginx-deployment"
 }

 spec {
 replicas = 1

 selector {
 match_labels = {
 app = "nginx"
 }
 }

 template {
 metadata {
 labels = {
 app = "nginx"
 }
 }

 spec {
 container {
 image = "nginx:latest"
 name = "nginx"
 }
 }
 }
 }
}

This is a very simple example of a Deployment object in Terraform HCL. Let’s add some complexity to the example by adding variables and loops:

Example with Variables and Loops

variable "namespace" {
 description = "The namespace to deploy the resources"
 type = string
 default = "default"
}

variable "nginx_image" {
 description = "The Nginx image to use"
 type = string
 default = "nginx:latest"
}

variable "replicas" {
 description = "The number of replicas to deploy"
 type = number
 default = 1
}

variable "number_of_deployments" {
 description = "The number of deployments to create"
 type = number
 default = 2
}

locals {
 nginx_image = var.nginx_image
 replicas = var.replicas
 namespace = var.namespace
 number_of_deployments = var.number_of_deployments
}

resource "kubernetes_deployment" "nginx" {
 for_each = {for i in range(local.number_of_deployments) : i => i}

 metadata {
 name = "nginx-deployment-${each.key}"
 namespace = local.namespace
 }

 spec {
 replicas = local.replicas
 selector {
 match_labels = {
 app = "nginx-${each.key}"
 }
 }

 template {
 metadata {
 labels = {
 app = "nginx-${each.key}"
 }
 }

 spec {
 container {
 image = local.nginx_image
 name = "nginx"
 }
 }
 }
 }
}

In this example, we can already see the advantage of Terraform over plain YAML files. We can use variables and loops to define multiple kubernetes_deployment objects in a more dynamic way.

Pros and Cons of Terraform

Pros

Dynamic: You can use variables, loops, and conditions in Terraform HCL. This lets you define your Kubernetes objects in a more dynamic way.
Abstraction: You can apply the DRY principle in Terraform HCL by defining and reusing modules across projects.
Environment management: By defining variables, you can manage multiple environments, like development, staging, and production. This is a more structured way to do it, without needing extra tools.

Cons

Complexity: Terraform is more complex than plain YAML files. You must learn Terraform’s HCL to define your Kubernetes objects.
DSL: Terraform is a Domain-Specific Language (DSL). You need to learn it to define your Kubernetes objects.
Testability: Testing Terraform code is harder. You need tools like Terratest and Kitchen-Terraform for it.
Helm Support: Terraform does not support Helm charts out of the box. You need to use the Helm provider to manage them with Terraform. This can add complexity to your Terraform code, and you need to keep track of the Helm provider dependency.
IDE Support: Terraform lacks the same IDE support as YAML files. You need plugins, like the Terraform Extension for Visual Studio Code, for syntax highlighting and autocompletion.

Terraform is a good choice for managing your Kubernetes resources if you have a complex project and you do not mind the complexity of the Terraform language. It gives you more flexibility and abstraction than plain YAML files, and you can manage multiple environments in a more structured way.

But is there a better way to manage your Kubernetes resources? Let’s have a look at Pulumi.

Pulumi and the Power of General Purpose Languages

Last but not least, let’s have a look at Pulumi. Pulumi is an Infrastructure as Code tool. It lets you define and manage your Kubernetes objects using general-purpose programming languages. Pulumi supports multiple languages like Python, Go, TypeScript, JavaScript, .NET, and JVM-based languages. This gives you all the benefits of a full programming language. It includes native IDE support, code completion, and debugging.

You need to install the Pulumi CLI. Then, create a new Pulumi project. Finally, add the Kubernetes library to it. It depends on the language you choose. You could use a package manager, like npm or pip.

Let’s have a look at an example of what a Deployment object would look like in all the supported languages.

import * as pulumi from "@pulumi/pulumi";
import * as k8s from "@pulumi/kubernetes";

const nginxDeployment = new k8s.apps.v1.Deployment("nginx-deployment", {
 metadata: {
 name: "nginx-deployment",
 },
 spec: {
 replicas: 1,
 selector: {
 matchLabels: {
 app: "nginx",
 },
 },
 template: {
 metadata: {
 labels: {
 app: "nginx",
 },
 },
 spec: {
 containers: [
 {
 name: "nginx",
 image: "nginx:latest",
 },
 ],
 },
 },
 },
});

import pulumi
import pulumi_kubernetes as k8s

nginx_deployment = k8s.apps.v1.Deployment(
 "nginx-deployment",
 metadata=k8s.meta.v1.ObjectMetaArgs(
 name="nginx-deployment",
 ),
 spec=k8s.apps.v1.DeploymentSpecArgs(
 replicas=1,
 selector=k8s.meta.v1.LabelSelectorArgs(
 match_labels={
 "app": "nginx",
 },
 ),
 template=k8s.core.v1.PodTemplateSpecArgs(
 metadata=k8s.meta.v1.ObjectMetaArgs(
 labels={
 "app": "nginx",
 },
 ),
 spec=k8s.core.v1.PodSpecArgs(
 containers=[
 k8s.core.v1.ContainerArgs(
 name="nginx",
 image="nginx:latest",
 ),
 ],
 ),
 ),
 ),
)

package main

import (
 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/apps/v1"
 corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/core/v1"
 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/meta/v1"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 _, err := v1.NewDeployment(ctx, "nginxDeployment", &v1.DeploymentArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("nginx-deployment"),
 },
 Spec: &v1.DeploymentSpecArgs{
 Replicas: pulumi.Int(1),
 Selector: &metav1.LabelSelectorArgs{
 MatchLabels: pulumi.StringMap{
 "app": pulumi.String("nginx"),
 },
 },
 Template: &corev1.PodTemplateSpecArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Labels: pulumi.StringMap{
 "app": pulumi.String("nginx"),
 },
 },
 Spec: &corev1.PodSpecArgs{
 Containers: corev1.ContainerArray{
 &corev1.ContainerArgs{
 Name: pulumi.String("nginx"),
 Image: pulumi.String("nginx:latest"),
 },
 },
 },
 },
 },
 })
 if err != nil {
 return err
 }
 return nil
 })
}

using Pulumi;
using Pulumi.Kubernetes.Types.Inputs.Core.V1;
using Pulumi.Kubernetes.Types.Inputs.Apps.V1;
using Pulumi.Kubernetes.Types.Inputs.Meta.V1;
using System.Threading.Tasks;
using Deployment = Pulumi.Deployment;

class MyStack : Stack
{
 public MyStack()
 {
 var nginxDeployment = new Pulumi.Kubernetes.Apps.V1.Deployment("nginxDeployment", new DeploymentArgs
 {
 Metadata = new ObjectMetaArgs
 {
 Name = "nginx-deployment",
 },
 Spec = new DeploymentSpecArgs
 {
 Replicas = 1,
 Selector = new LabelSelectorArgs
 {
 MatchLabels =
 {
 { "app", "nginx" }
 }
 },
 Template = new PodTemplateSpecArgs
 {
 Metadata = new ObjectMetaArgs
 {
 Labels =
 {
 { "app", "nginx" }
 }
 },
 Spec = new PodSpecArgs
 {
 Containers =
 {
 new ContainerArgs
 {
 Name = "nginx",
 Image = "nginx:latest",
 }
 }
 }
 }
 }
 });
 }
}

class Program
{
 static Task<int> Main() => Deployment.RunAsync<MyStack>();
}

package myproject;

import com.pulumi.Pulumi;
import com.pulumi.kubernetes.apps.v1.Deployment;
import com.pulumi.kubernetes.apps.v1.DeploymentArgs;
import com.pulumi.kubernetes.apps.v1.inputs.DeploymentSpecArgs;
import com.pulumi.kubernetes.core.v1.inputs.ContainerArgs;
import com.pulumi.kubernetes.core.v1.inputs.PodSpecArgs;
import com.pulumi.kubernetes.core.v1.inputs.PodTemplateSpecArgs;
import com.pulumi.kubernetes.meta.v1.inputs.LabelSelectorArgs;
import com.pulumi.kubernetes.meta.v1.inputs.ObjectMetaArgs;

import java.util.Map;

public class App {
 public static void main(String[] args) {
 Pulumi.run(ctx -> {
 var nginxDeployment = new Deployment("nginxDeployment", DeploymentArgs.builder()
 .metadata(ObjectMetaArgs.builder()
 .name("nginx-deployment")
 .build())
 .spec(DeploymentSpecArgs.builder()
 .replicas(1)
 .selector(LabelSelectorArgs.builder()
 .matchLabels(Map.of("app", "nginx"))
 .build())
 .template(PodTemplateSpecArgs.builder()
 .metadata(ObjectMetaArgs.builder()
 .labels(Map.of("app", "nginx"))
 .build())
 .spec(PodSpecArgs.builder()
 .containers(ContainerArgs.builder()
 .name("nginx")
 .image("nginx:latest")
 .build())
 .build())
 .build())
 .build())
 .build());
 });
 }

Now have a look into the more complex example with variables and loops in Pulumi. I am using TypeScript as an example, but you can of course use any of the Pulumi supported languages.

Example with Variables and Loops

import * as pulumi from "@pulumi/pulumi";
import * as k8s from "@pulumi/kubernetes";

let config = new pulumi.Config();

const namespace = config.get("namespace") || "namespace";
const nginxImage = config.get("nginxImage") || "nginx:latest";
const replicas = config.getNumber("replicas") || 1;
const numberOfDeployments = config.getNumber("numberOfDeployments") || 1;

for (let i = 0; i < numberOfDeployments; i++) {
 new k8s.apps.v1.Deployment(`nginxDeployment-${i}`, {
 metadata: {
 name: `nginx-deployment-${i}`,
 namespace: namespace,
 },
 spec: {
 replicas: replicas,
 selector: {
 matchLabels: {
 app: `nginx-${i}`,
 },
 },
 template: {
 metadata: {
 labels: {
 app: `nginx-${i}`,
 },
 },
 spec: {
 containers: [
 {
 name: "nginx",
 image: nginxImage,
 },
 ],
 },
 },
 },
 });
}

In this example, you can see how easy it is to define multiple Deployment objects with variables and loops syntax provided by TypeScript. This is a big advantage over plain YAML files and Terraform HCL. Blending the full power of a general-purpose programming language with the Kubernetes API is a natural process. Consider the possibilities of creating reusable functions, classes, and modules. You could share them across multiple projects and teams.

You might also like:

Pros and Cons of Pulumi

Pros

You can use general-purpose languages to define your Kubernetes objects. These include Python, Go, TypeScript, JavaScript, .NET, and JVM-based languages.

Full IDE support: You get full IDE support. It includes syntax highlighting, code completion, and debugging.
Testability: You can use the full power of your programming language to test your Pulumi code.
Abstraction: You can use the DRY principle in Pulumi. Define reusable functions, classes, and modules in your chosen programming language.
Environment management: You can manage multiple environments, like development, staging, and production, by defining config variables. This is a more structured way, without needing extra tools.
Kubernetes Helm Support: Pulumi supports Helm charts out of the box. You can manage Helm charts with Pulumi without needing extra tools or providers.
Programmatic execution of Pulumi: You can further blend Pulumi code into your projects using the Pulumi Automation API.

Cons

Runtime dependency: You must have the runtime of your chosen programming language installed on your machine, or use a containerised version of it in your CI/CD pipeline.
Follow best practices and compliance: Set up a code review process. It should ensure the Pulumi code follows the best practices of your chosen programming language and your organization’s additional best practices.
No possibility to convert one Pulumi language to another: If you start with one language, you can’t convert the Pulumi code to another language. You need to rewrite the Pulumi code in the new language.

YAML vs. Terraform vs. Pulumi - Results

Feature	YAML	Terraform	Pulumi
Readability	✅ Simple and human-readable	⚠️ Requires learning HCL	✅ Uses familiar programming languages
Environment Management	❌ Requires manual management of multiple files	✅ Structured management via variables	✅ Structured management via config variables
DRY Principle	❌ No reusability, requires manual updates	✅ Supports modules and reusable code	✅ Full support via functions, classes, modules
Dynamic Configuration	❌ Static configuration only	✅ Supports loops, conditions, variables	✅ Full programming language capabilities
IDE Support	✅ Good support	⚠️ Limited, requires plugins	✅ Full language-native IDE support
Testing	❌ Limited testing options	⚠️ Requires specialized tools (Terratest)	✅ Native language testing frameworks
Helm Support	✅ Native support	⚠️ Requires additional provider	✅ Built-in support
Learning Curve	✅ Low	⚠️ Medium (new DSL)	⚠️ Varies by language knowledge
Dependencies	✅ Minimal	⚠️ Terraform installation	❌ Language runtime required
Ecosystem Tools	❌ Requires additional tools (Helm, Kustomize)	✅ Integrated tooling	✅ Integrated tooling
Abstraction	❌ Limited	✅ Good	✅ Excellent

Legend:

✅ Good/Supported
⚠️ Partial/With limitations
❌ Poor/Not supported

Time to say Goodbye to YAML?

So, what’s the smart choice for deployment automation with Kubernetes? Is YAML still the best choice for Kubernetes? Or, is there a better way to manage your resources? I understand if you are a YAML fan. Despite complaints, it is still the most popular way to define Kubernetes objects. Many folks feel a sense of accomplishment when their YAML file works on the first try.

If you have a complex project, you will soon hit the limits of YAML. You want to manage your Kubernetes resources in a more structured, dynamic way while following best practices and compliance. Then, you keep adding more complexity to your YAML files. You do this by using tools like helm and kustomize to fix YAML’s flaws. Or, you adopt a hybrid approach with jsonnet, cuelang, or ytt. This makes it even more complex.

Giving up on YAML is not easy, but it does not mean that you have to give up on control. You can still have full control by using Terraform or Pulumi.

Conclusion

Now it’s time to make a decision. What’s the smart choice for deployment automation with Kubernetes? Is it YAML, Terraform, or Pulumi? For me, the smart choice is Pulumi. It gives you all the benefits of a general-purpose programming language. You gain full control, a more streamlined workflow, and the power to define all your infrastructure needs as code beyond Kubernetes. Especially on complex projects, use Pulumi. It’s better than dealing with many YAML files and the complexity of Terraform HCL.

Next time you start a new project, try Pulumi. See for yourself that it’s a smart choice. It’s an upgrade to your productivity and happiness.

But that’s my opinion. What’s your opinion? What’s your smart choice for deployment automation with Kubernetes?

If you want to try out Pulumi or play around to prove me wrong, you can start with:

Get Started with Pulumi

Use Pulumi's open-source SDK to create, deploy, and manage infrastructure on any cloud.

How Secrets Sprawl is Slowing You Down—And What to Do About It.

Engin Diri — Mon, 28 Oct 2024 12:40:10 +0200

Only a few things are certain in the lives of developers and DevOps engineers: taxes, yearly performance reviews, and secret sprawl. While the first two are inevitable, the last one is something that can be managed.

As we keep adding new cloud resources and releasing new applications, the number of secrets we need to manage keeps growing: passwords, API keys, certificates, and more. And as if this isn’t enough, we need to manage secrets across different systems and environments with different teams that need to access them; we end up with duplicates.

Let’s have a look at a typical environment and see what teams use: Everything container-related may use Docker secrets or Kubernetes secrets. There is also a high chance that they might use cloud-specific secrets managers like Azure Key Vault, AWS Secrets Manager, or Google Secret Manager. In addition to these, there is a high chance that they might use HashiCorp Vault or some enterprise-grade secrets manager like CyberArk.

There are still a lot of organizations that store their secrets (like the API key for a cloud service) in environment variables to configure their applications on virtual machines, containers, or serverless functions. And as if that weren’t risky enough, some developers still hardcode sensitive information in their application code. Or use plain unencrypted text files such as .env files and share them inside the team. To be fair, this is not something that is done on purpose, and most teams realize the risks of these practices. But the problem is that they have not found a better way to manage their secrets that won’t slow them down and impact their productivity.

As the IT infrastructure grows, it becomes increasingly challenging for organizations to secure, organize, and distribute their secrets. This is where secrets sprawl comes into play. Secrets sprawl is the uncontrolled and unmanaged distribution of secrets across different systems and environments.

Modern architectures are adding to the problem of secret sprawl as modern applications are more distributed and consist of smaller building blocks with a growing number of different technologies.

While teams are ensuring that each of their applications has everything it needs to run, it’s also easy for them to lose track of the secrets they are using. Everything can be fine until they need to rotate a secret or revoke a compromised one. Then they need to go through all the systems and environments to find and update the secret. Tracking down all the places where a secret is used can be a time-consuming and error-prone process with a high risk of missing some places.

So what are some of the pain points of secret sprawl, and how can we overcome them to regain our productivity?

Pain Points of Secrets Sprawl

With secrets sprawled all across our teams (development and operations) and infrastructure, we face several points:

It is challenging to keep track of all the secrets and where they are used. When we need, for example, to find a specific secret, we need to go through different secret managers, code, and, in the worst case, the local files of one of the developers.

Do we need to rotate a secret? Now we need to go through all the systems and environments to find and update every duplicated secret.
Security risk is another pain point to mention. When a secret is spread throughout the organization, it’s hard to manage access control and enforce specific access policies.

Security best practices also recommend that secrets should be rotated regularly. But with secrets sprawled all over the place, it makes it a lot of work to rotate them. Time we could spend on more important tasks like developing new features.
Secrets sprawl makes it extremely challenging to distribute secrets. Having a secret, for example, in a cloud provider’s secret manager is cool and great, but what if we need to use it in a Kubernetes cluster? What is the process for that?

Also, having secrets in multiple places means there are different owners for them. These secrets are now some kind of gatekeeping by the team that owns them. This makes it hard for other teams to access them when they need them. And even if they get access, they will then be mostly shared in plain text as the other team does not have access to the secret manager itself.

As we can see, secret sprawl is a real problem that can slow down our teams and impact our productivity. But it also affects how fast an organization can scale its development and operations teams.

With an “organic” way of managing secrets and no real transparency, it’s hard to keep up with the pace of the business, and secret sprawl becomes a major bottleneck. Additionally, that growth means a whole bunch of new secrets that need to be managed.

Rinse and repeat.

But behold, there is a solution to this problem. And it’s called centralized secrets management.

Centralized Secrets Management against Secrets Sprawl

The cure for secrets sprawl is centralized secrets management. When you centralize your secrets in one place, you know where they are and who has access to them.

This makes it easier to manage access control and enforce specific access policies. It also makes it easier to rotate secrets as you only need to do it in one place.

Have a look at an example scaling of an organization or team:

When the company is small, the team decides to use Kubernetes to deploy their applications, and they store their secrets in Kubernetes secrets. This first step is already a centralization of secrets as it tightens access to the secrets and makes it easier to audit them. Maintaining an encrypted secrets store and centralizing the secrets in one place is a good start and ensures the organization is following a consistent threat protection policy.

As the company grows, one DevOps engineer is responsible for multiple Kubernetes clusters. In these cases, individual users can’t enforce their own application-wide policies, meaning secrets are better managed and controlled.

But what if the company grows even more and the team decides to use multiple cloud providers? There are many reasons for this, like costs, different service offerings, and better disaster recovery when using multiple cloud providers.

Sticking to the built-in secrets managers of the cloud providers is not a good idea as it will lead to secrets sprawl.

You might also like:

Why Pulumi ESC is the right choice for centralized multi-cloud secrets management

Pulumi ESC is the right choice for centralized multi-cloud capable secrets management as it reduces secrets sprawl by centralizing secrets storage with a lot of integrations for a wide range of platforms.

Pulumi ESC delivers a seamless integration so you can still use your existing secrets managers like Azure Key Vault, AWS Secrets Manager, Google Secret Manager, or HashiCorp Vault. Secrets are automatically synced to Pulumi ESC, saving developers a lot of time so they can focus on their main tasks like developing and building new products and features.

Pulumi ESC is a completely managed service that again saves time and resources for the organization as they don’t need to worry about the infrastructure, maintenance, and support of their self-hosted secrets manager.

Wrapping Up

Every organization faces the risk of secrets sprawl as they grow and add more cloud resources and applications, which they may not even be aware of.

Multi-cloud scenarios make it even more challenging to manage secrets across different systems and environments.

As teams have to juggle between delivering and building new products and features, managing secrets in a secure and efficient way becomes a major bottleneck on productivity and a major security risk. A ticking time bomb that can explode at any time.

Secrets managers can help you reduce secrets sprawl by centralizing secrets storage and providing a seamless way to use, change, and distribute secrets across different systems and environments. Independent of the size of your organization, a startup or a large enterprise, it’s never too late to start managing your secrets in a secure and efficient way.

Pulumi ESC is on the way to becoming the best choice to solve the problem of secrets sprawl. Making both developers and security teams happy by providing a secure and efficient way to manage secrets across different systems and environments.

Start using Pulumi ESC and ESO today by creating an account on the Pulumi Cloud Console and begin managing your secrets in a secure and efficient way.

Create an Account

Pulumi ESC and External Secrets Operator: The Perfect Solution for Today's Cloud-Native Secret Management

Engin Diri — Fri, 04 Oct 2024 00:00:00 +0000

Managing secrets in a cloud-native environment can be challenging, but it is crucial for ensuring the security and integrity of any application or infrastructure. We encounter a lot of different types of secrets, from API keys, database passwords, and certificates to tokens and passwords. These secrets need to be stored securely and accessed by different services in a secure way without exposing any sensitive information to unauthorized users.

Here is where Pulumi ESC and External Secrets Operator come into play by providing a secure and efficient solution for cloud-native secret management.

Pulumi ESC (Environments, Secrets, and Configuration)

Pulumi ESC is our new secrets management & orchestration service for environments, secrets, and configurations. It can be used as part of your Pulumi IaC (Infrastructure as Code) projects using pulumiConfig to expose the stored values to your Pulumi stacks.

But it can also be used as a standalone service to manage secrets and configurations. Pulumi ESC provides support for applications through its dedicated SDKs built for different programming languages. It can also be managed from the command line with the pulumi env commands in the Pulumi CLI, which can inject secrets and configuration as a context for your CLI tools (e.g. terraform).

For more information on Pulumi ESC, you can check out the official documentation.

External Secrets Operator (ESO)

The Open Source project External Secrets Operator (ESO) is a Kubernetes operator that helps manage secrets in a highly secure and scalable way. ESO synchronizes secrets stored in external secret management systems like Pulumi ESC, HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, and more into Kubernetes secrets. This approach allows us to control secret sprawl and centralize the management of secrets in a single place while providing a secure and controlled way to access them.

Why Kubernetes secrets are not enough?

Kubernetes secrets are a great way to store sensitive information in a Kubernetes cluster. Kubernetes secrets are created and stored as a Kubernetes resource object and then used by pods or other resources inside the cluster.

Here is an example of a Kubernetes secret manifest file:

apiVersion: v1
kind: Secret
metadata:
 name: mysecret
type: Opaque
data:
 username: YWRtaW4= # admin
 password: YWRtaW4= # admin

or using kubectl:

kubectl create secret generic mysecret --from-literal=username=admin --from-literal=password=admin

Kubernetes secrets have several limitations. Here are the top ones:

They are stored by default in etcd in base64 encoded format, which is not secure.
They are created either manually by using kubectl commands or by using a manifest file, making them hard to manage at scale.
Hard to manage and synchronize secrets across different environments and clusters.
There is no default way to rotate secrets automatically.

ESO shines by tackling all the challenges listed and providing a way to manage secrets in a more secure and efficient manner. Benefits include:

Mitigate security risks as secrets are stored in a secure external secret management system like Pulumi ESC, which provides more robust security features like encryption, access control, and auditing.
Eliminate manual management of secrets by synchronizing them from external secret management systems into the Kubernetes cluster, reducing the need for manual intervention and minimizing the risk of human error.
ESO provides a unified interface, making it easy to manage secrets across clusters and multi-cloud environments.
ESO integrates with external secret management systems which ensure that access to secrets is tracked and an audit trail is maintained. This is crucial for security compliance and adhering to regulatory standards.
Automate secrets rotation when the external secret management system has support it.

External Secrets Operator architecture

The central part of the ESO architecture is the extension of the Kubernetes API with a new Custom Resource Definition (CRD), which defines where the secrets are stored and how they are synchronized into Kubernetes secrets. The ESO controller then retrieves the secrets from the external secret management system and creates or updates the Kubernetes secrets. Every time a secret is updated in the external secret management system, the ESO controller will synchronize the secret in the Kubernetes cluster and update the secret accordingly.

Core resources of ESO

(Cluster)SecretStore: Defines the external secret management system where the secrets are stored. It contains the necessary information to connect to the external secret management system, such as the endpoint, authentication credentials, and other configuration options. The SecretStore is namespaced while the ClusterSecretStore is cluster wide, which allows you cross-namespaced access to the secrets.
(Cluster)ExternalSecret: Defines the secret that needs to be synchronized into the Kubernetes cluster. It contains the reference to the SecretStore and the secret key and how the data should be transformed and saved as a Kubernetes secret.
PushSecret: With the PushSecret resource, you can push secrets from the Kubernetes cluster to the external secret management system. The Pulumi ESC provider for ESO supports this feature, which allows you to push secrets from the cluster into Pulumi ESC. This feature is useful when you create secrets through other means and want to synchronize them into Pulumi ESC to manage them centrally.

How to Use Pulumi ESC and ESO Together

To demonstrate how to use Pulumi ESC and ESO together, we will store some secrets and configuration values in Pulumi ESC, then synchronize them into a Kubernetes cluster using ESO.

To follow along, you’ll need the following prerequisites:

A Kubernetes cluster (I will be using a local KinD cluster, but you can use any Kubernetes cluster)
Pulumi CLI installed. You can use the pulumi env commands to manage secrets and configurations.
kubectl CLI installed for some debugging

Step 1: Install ESO and Link to Pulumi ESC

We will use Pulumi with the pulumi-kubernetes provider to deploy ESO into the Kubernetes cluster. The power of Pulumi ESC integration comes in two ways: First, to supply the Pulumi access token to ESO via Pulumi IaC, allowing the new ESO instance to access Pulumi ESC later. Second, to provide configuration and secrets from Pulumi ESC, via ESO, to the PodInfo application within the Kubernetes cluster.

pulumi new kubernetes-typescript --name eso-pulumi-esc

You should see the following output:

This command will walk you through creating a new Pulumi project.

Enter a value or leave blank to accept the (default), and press <ENTER>.
Press ^C at any time to quit.

Project description (A minimal Kubernetes TypeScript Pulumi program):
Created project 'eso-pulumi-esc'

Please enter your desired stack name.
To create a stack in an organization, use the format <org-name>/<stack-name> (e.g. `acmecorp/dev`).
Stack name (dev):
Created stack 'dev'

The package manager to use for installing dependencies npm
Installing dependencies...


added 310 packages, and audited 311 packages in 12s

44 packages are looking for funding
 run `npm fund` for details

found 0 vulnerabilities
Finished installing dependencies

Your new project is ready to go! ✨

To perform an initial deployment, run `pulumi up`

Head to Pulumi Cloud Console and create a new Pulumi ESC project with the name eso-esc-demo:

And the environment eso-esc-dev:

In the editor add the following yaml into the Environment definition:

values:
 pulumiConfig:
 pulumi-pat:
 fn::secret: <your-pulumi-pat>

Don’t know how to create a PAT? Check out the official Pulumi documentation.

If you prefer to use the Pulumi CLI, you can create the environment by running:

pulumi env init <your-org>/eso-esc-demo/eso-esc-dev

And set the configuration by running the env edit command and copy the above YAML into the editor:

pulumi env edit <your-org>/eso-esc-demo/eso-esc-dev

Now, we need to link the Pulumi ESC project to the Pulumi IaC project. To do this, we need to add the following to your Pulumi.dev.yaml:

environment:
- eso-esc-demo/eso-esc-dev

Now that all the wiring is done, we can deploy the ESO instance into the Kubernetes cluster. Open the index.ts file and add the following code:

import * as k8s from "@pulumi/kubernetes";
import * as pulumi from "@pulumi/pulumi";

const config = new pulumi.Config();

// Create a Kubernetes namespace
const ns = new k8s.core.v1.Namespace("external-secrets", {
 metadata: {
 name: "external-secrets",
 },
});

// Deploy a Helm release into the namespace
const externalSecrets = new k8s.helm.v3.Release("external-secrets", {
 chart: "external-secrets",
 version: "0.10.4", // Specify the version of the chart
 namespace: ns.metadata.name,
 repositoryOpts: {
 repo: "https://charts.external-secrets.io",
 },
});

// Deploy a secret into the namespace
const mySecret = new k8s.core.v1.Secret("my-secret", {
 metadata: {
 namespace: ns.metadata.name,
 name: "pulumi-access-token",
 },
 stringData: {
 "PULUMI_ACCESS_TOKEN": config.require("pulumi-pat"),
 },
 type: "Opaque",
});

// Export the namespace name
export const namespaceName = ns.metadata.name;

// Export the Helm release name
export const helmReleaseName = externalSecrets.name;

Deploy the stack by running:

pulumi up

And you should see that the secret was created in the Kubernetes cluster and the ESO instance was deployed successfully.

kubectl get secret -n external-secrets pulumi-access-token -o jsonpath='{.data.PULUMI_ACCESS_TOKEN}' | base64 -d

Step 2: Create an External Secret

Now, we will create a secret in the Pulumi ESC project and synchronize it into the Kubernetes cluster by creating an ESO reference to the external secret.

Create a new ESC environment called esp-esc-app in the eso-esc-demo ESC project.

Add the following YAML into the Environment definition:

values:
 app:
 hello: world
 hello-secret:
 fn::secret: world

If you prefer to use the Pulumi CLI, you can create the environment by running:

pulumi env init <your-org>/eso-esc-demo/esp-esc-app

And set the configuration by running the env edit command and copy the above YAML into the editor:

pulumi env edit <your-org>/eso-esc-demo/esp-esc-app

Either way, you should see following environment configuration in the Pulumi Cloud Console:

We can now create ClusterExternalSecret in the index.ts and reference the ESC secret.

import * as k8s from "@pulumi/kubernetes";
import * as pulumi from "@pulumi/pulumi";

// cut for brevity

const crd = new k8s.apiextensions.CustomResource("cluster-secret-store", {
 apiVersion: "external-secrets.io/v1beta1",
 kind: "ClusterSecretStore",
 metadata: {
 name: "secret-store",
 },
 spec: {
 provider: {
 pulumi: {
 organization: pulumi.getOrganization(),
 project: "eso-esc-demo",
 environment: "esp-esc-app",
 accessToken: {
 secretRef: {
 name: mySecret.metadata.name,
 key: "PULUMI_ACCESS_TOKEN",
 namespace: mySecret.metadata.namespace,
 },
 },
 },
 },
 },
});

We can check that the secret was successfully synchronized by running:

kubectl get clustersecretstore secret-store -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'

Step 3: Deploy an Application and Reference the Secret

Now, we can deploy an application that references the secret from the Kubernetes cluster. I am going to deploy the app called PodInfo, which is an application that exposes the pod information as an API. Add the following code to the index.ts:

import * as k8s from "@pulumi/kubernetes";
import * as pulumi from "@pulumi/pulumi";

// cut for brevity
const podInfo = new k8s.helm.v3.Release("podinfo", {
 chart: "podinfo",
 version: "6.7.0",
 namespace: "podinfo",
 createNamespace: true,
 repositoryOpts: {
 repo: "https://stefanprodan.github.io/podinfo",
 },
 values: {
 extraEnvs: [
 {
 name: "FROM_ESC_VIA_ESO",
 valueFrom: {
 secretKeyRef: {
 name: "esc-secret-store",
 key: "hello-secret",
 }
 }
 },
 {
 name: "FROM_ESC_VIA_ESO_2",
 valueFrom: {
 secretKeyRef: {
 name: "esc-secret-store",
 key: "hello",
 }
 }
 }
 ]
 },
});

const externalSecretPodInfo = new k8s.apiextensions.CustomResource("external-secret-podinfo", {
 apiVersion: "external-secrets.io/v1beta1",
 kind: "ExternalSecret",
 metadata: {
 name: "esc-secret-store",
 namespace: podInfo.namespace,
 },
 spec: {
 dataFrom: [
 {
 extract: {
 conversionStrategy: "Default",
 key: "app",
 }
 }
 ],
 refreshInterval: "1h",
 secretStoreRef: {
 kind: clusterSecretStore.kind,
 name: clusterSecretStore.metadata.name,
 }
 },
});

This will deploy the PodInfo application and create an ExternalSecret resource that references the secret from the Pulumi ESC project. We mount the secret into the pod as an environment variable by using the extraEnvs field in the Helm chart.

In a real-world application, you might use alternate ways to reference the secret, like mounting it as a volume or using fromEnv field in the Helm chart. It depends on your application and how you want to consume the secret.

After deploying the stack, you should see that the PodInfo application is running. To access the application, we can port-forward the service by running:

kubectl port-forward svc/podinfo 9898:9898 -n podinfo

Our PodInfo demo application provides a large selection of different endpoints to interact with. We are interested in the /env endpoint to check if the secret was successfully mounted into the pod. Let’s try that:

curl -s localhost:9898/env | grep FROM
 "FROM_ESC_VIA_ESO=world",
 "FROM_ESC_VIA_ESO_2=world",

And here we see secret values from the ESC project, proxied by ESO, and passed to the PodInfo application as environment variables. That means the secret was successfully mounted into the pod and the application is running as expected.

Step 4: Clean Up

After you are done with the demo, you can clean up the resources by running:

pulumi destroy

Conclusion

The combination of Pulumi ESC and External Secrets Operator provides a secure and efficient way to manage secrets in a cloud-native environment, improving security, efficiency, and compliance when consuming secrets in your Kubernetes cluster. By using Pulumi ESC, you can manage secrets and configurations in a centralized way, and with ESO, you now have a streamlined process to synchronize secrets into your Kubernetes cluster.

Start using Pulumi ESC and ESO today by creating an account on the Pulumi Cloud Console and begin managing your secrets in a secure and efficient way.

Create an Account

Pulumi in a Cloud Native World

Engin Diri — Fri, 02 Aug 2024 00:00:00 +0000

In today’s complex digital landscape, organizations are increasingly turning to platform engineering to optimize their software delivery processes and maximize efficiency. The growing complexity of modern applications, coupled with the need for rapid, secure, and scalable deployments, has created a pressing demand for robust Internal Developer Platforms (IDPs).

IDPs are a key component of modern platform engineering strategies. An IDP is a self-service layer that sits on top of an organization’s infrastructure, abstracting away complexity and providing developers with the tools and environments they need to build, test, and deploy applications efficiently.

Key features of an IDP:

Developer Control Plane. Curated experiences that empower developers by meeting them at their level of expertise, whether it’s an abstracted developer portal, custom CLI, or shared IaC templates.
Integration & Delivery. Automations to version control, test, trace, and deploy all infrastructure from resources, configurations, environments, and secrets as well as orchestration automations to manage provisioning workflows.
Monitoring & Logging. Components to log, monitor, and observe all infrastructure for greater operational control as well as optimize against unnecessary costs.
Security & Identity. Security and compliance guardrails that regulate every piece of infrastructure from policies to fine-grained access controls to secrets.
Resources. Providers that support modern cloud architectures such as Kubernetes, containers, serverless, generative AI, machine learning, data lakes, hybrid cloud/on-premises, and more.

By implementing an IDP, organizations can significantly reduce cognitive load on developers, accelerate time-to-market, improve security and compliance, and foster a culture of innovation. IDPs act as a force multiplier, allowing development teams to focus on creating business value while the platform handles the underlying complexity of cloud-native infrastructure.

The Cloud Native Computing Foundation (CNCF) landscape offers a large ecosystem of tools and technologies that can be leveraged to build these platforms from scratch. However, the sheer number of options can be overwhelming, making it difficult for organizations to determine the best combination of tools for their specific needs.

At Pulumi’s Customer Experience Team, we’ve observed that adopting Infrastructure as Code often serves as a catalyst for organizations to reassess their entire software delivery process. This reassessment frequently leads to the implementation of more streamlined, automated, and secure methodologies – the core principles of platform engineering mentioned above.

To provide guidance for our customers, we have created a reference architecture called the KEBAP stack, this stack offers a structured approach to integrate these technologies into an effective platform engineering solution using Pulumi.

The KEBAP Stack Reference Architecture

The KEBAP stack consists of the following (most commonly used) CNCF projects:

Kubernetes (with Kyverno as a bonus)
External Secrets Operator
Backstage
Argo CD
Pulumi

Have a look on the high-level architectural diagram of the KEBAP stack to get a first idea of what we are going to talk about in the following sections:

Breaking Down the KEBAP Stack

Let’s take a closer look at each component of the KEBAP stack and understand what their role. The KEBAP is layered in such a way that each component builds on the capabilities of the previous one. The five layers of the KEBAP stack are designed to address different capabilities of our internal developer platform:

Kubernetes

In modern cloud computing, the hub-and-spoke architecture pattern is widely used to manage multiple Kubernetes clusters. In this pattern, a central control plane (hub) manages a set of clusters (spokes) that run applications and services. The hub provides a unified view of the entire infrastructure, enabling centralized management, monitoring, and security.

The control plane cluster in the KEBAP stack is the hub cluster and contains the control plane logic for the entire IDP.

Pulumi

Pulumi manages the lifecycle of all cloud infrastructure, providing developers with self-service capabilities. It allows you to use familiar programming languages to define your infrastructure as code, offering more flexibility and power than traditional YAML or domain-specific languages.

The Pulumi Kubernetes Operator enables you to manage your infrastructure directly from within Kubernetes, seamlessly integrating with your existing Kubernetes workflows.

Since we use GitOps as our deployment method, we define the infrastructure in a Git repository and then let Argo CD deploy it to the hub cluster. As soon as the infrastructure definition is deployed, the Pulumi Kubernetes Operator will take over and create the defined resources in the selected cloud provider.

Here is an example Pulumi definition of a new Kubernetes cluster using DigitalOcean:

apiVersion: pulumi.com/v1
kind: Program
metadata:
 name: ${{values.name}}
program:
 variables:
 clusterName: "do-cluster"
 clusterRegion: "nyc1"
 nodePoolName: "default"
 nodeCount: 1
 version: 1.30.1-do.0

 resources:
 do_cluster:
 type: digitalocean:KubernetesCluster
 properties:
 name: "${clusterName}"
 region: "${clusterRegion}"
 version: "${version}"
 destroyAllAssociatedResources: true
 nodePool:
 name: "${nodePoolName}"
 size: "s-2vcpu-2gb"
 nodeCount: "${nodeCount}"
 # cut for brevity
---
apiVersion: pulumi.com/v1
kind: Stack
metadata:
 name: ${{values.name}}
spec:
 stack: ${{values.stack}}
 envRefs:
 DIGITALOCEAN_TOKEN:
 type: Env
 env:
 name: DIGITALOCEAN_TOKEN
 programRef:
 name: ${{values.name}}
 destroyOnFinalize: false

Argo CD

Argo CD handles the deployment layer, ensuring that the current deployment matches what is declaratively defined in the source. It’s emerging as the de facto standard due to its rich front-end and powerful features. Argo CD implements the GitOps model, continuously monitoring your Git repositories and automatically updating the deployed applications to match the desired state.

In the KEBAP stack, we use Argo CD to deploy the Pulumi infrastructure defined in the Git repository to our hub cluster.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
 name: ${{values.name}}-${{values.stage}}-${{ (values.owner | parseEntityRef).name }}
 namespace: argocd
spec:
 destination:
 namespace: pulumi-operator
 server: https://kubernetes.default.svc
 project: ${{ (values.owner | parseEntityRef).name }}
 source:
 path: "gitops/teams/clusters/${{values.name}}-${{values.stage}}-cluster"
 directory:
 recurse: false
 exclude: '{catalog-info.yaml}'
 targetRevision: main
 repoURL: https://github.com/my-backstage-demo/backstage-infrastructure-provisioning-templates-workshop.git
 syncPolicy:
 automated:
 prune: true
 selfHeal: true
 syncOptions:
 - ServerSideApply=true
 - CreateNamespace=true

External Secrets Operator (ESO)

The External Secrets Operator (ESO) addresses the challenge of securely managing secrets in Kubernetes. It integrates with various secret stores (such as Pulumi ESC, AWS Secrets Manager, HashiCorp Vault, Google Secrets Manager, Azure Key Vault). This allows you to configure access to the secret store once for all your required secrets, enhancing security and simplifying management.

We even take it a step further by using ESO to manage the secrets from Pulumi ESC (Environments, Secrets, and Configurations) with the Pulumi Provider for ESO. This way, we use the Pulumi not only for the infrastructure but also for the secrets management reducing the number of tools we need to manage.

To give the Pulumi Kubernetes Operator access to the secrets, we create following ClusterSecretStore object:

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
 name: secret-store
spec:
 provider:
 pulumi:
 organization: <org>
 environment: <env>
 accessToken:
 secretRef:
 name: pulumi-access-token
 key: PULUMI_ACCESS_TOKEN
 namespace: argocd

And then we can define the ExternalSecret with all the secrets that we want to manage and pass to our Operator as environment variables extraEnv:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
 name: pulumi-operator-secrets
spec:
 refreshInterval: 1h
 secretStoreRef:
 kind: ClusterSecretStore
 name: secret-store
 data:
 - secretKey: pulumi-access-token
 remoteRef:
 key: "backstage.pulumi-pat"
 - secretKey: do-token
 remoteRef:
 key: "backstage.do"

Backstage

Backstage is an open platform for building developer portals. It provides a centralized place for managing software catalogs, documentation, and tooling. This layer helps in organizing microservices and infrastructure, streamlining the process for developers to create, manage, and explore services.

In our KEBAB stack, we installed the Pulumi plugin for Backstage, which allows us to manage our infrastructure as code directly from the Backstage UI by cookie-cutting the Pulumi project and making the result available in our source control repository. From there, Argo CD takes over and deploys the definition to the hub cluster as described above.

apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
 name: kubernetes-template
 title: Kubernetes Cluster
 description: |
 A template for creating a new Kubernetes Cluster.
 tags:
 - pulumi
 - kubernetes
spec:
 steps:
 - id: pulumi-new-component
 name: Cookie cut the component Pulumi project
 action: pulumi:new
 input:
 name: "${{ parameters.component_id }}-infrastructure"
 description: ${{ parameters.description | dump }}
 organization: ediri
 stack: ${{ parameters.stack }}
 template: "https://github.com/my-silly-organisation/microservice-civo/tree/main/infrastructure-${{ parameters.cloud }}-${{ parameters.language }}"
 config:
 "node:node_count": "${{ parameters.nodeCount }}"
 folder: .

Bonus: Kyverno

Kyverno can be used to enforce policies and automate security and operational best practices in Kubernetes clusters. Kyverno is a policy engine designed for Kubernetes that allows you to define policies as code and enforce them at runtime. This ensures that your Kubernetes clusters are secure and compliant with your organization’s policies.

In the KEBEP stack, we use Kyverno to create a ClusterPolicy object to automatically add vCluster clusters to the Argo CD as spoke clusters. This will generate a Secret object in the argocd namespace for each vCluster cluster the dev teams create.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
 name: sync-secret-vcluster
 annotations:
 argocd.argoproj.io/sync-wave: "5"
spec:
 generateExistingOnPolicyUpdate: true
 rules:
 - name: sync-secret-vcluster-nonprod
 match:
 any:
 - resources:
 names:
 - "*-dev-*"
 - "*-staging-*"
 - "*-prod-*"
 kinds:
 - Secret
 exclude:
 any:
 - resources:
 namespaces:
 - kube-system
 - default
 - kube-public
 - kyverno
 context:
 - name: namespace
 variable:
 value: "{{ request.object.metadata.namespace }}"
 - name: name
 variable:
 value: "{{ request.object.metadata.name }}"
 - name: ca
 variable:
 value: "{{ request.object.data.\"certificate-authority\" }}"
 - name: cert
 variable:
 value: "{{ request.object.data.\"client-certificate\" }}"
 - name: key
 variable:
 value: "{{ request.object.data.\"client-key\" }}"
 - name: vclusterName
 variable:
 value: "{{ replace_all(name, 'vc-', '') }}"
 jmesPath: 'to_string(@)'
 - name: stage
 variable:
 value: "{{ split(vclusterName, '-')[2] }}"
 jmesPath: 'to_string(@)'
 - name: team
 variable:
 value: "{{ split(vclusterName, '-')[3] }}"
 jmesPath: 'to_string(@)'
 generate:
 kind: Secret
 apiVersion: v1
 name: "{{ vclusterName }}"
 namespace: argocd
 synchronize: true
 data:
 kind: Secret
 metadata:
 labels:
 argocd.argoproj.io/secret-type: cluster
 clusterType: "vcluster"
 env: "{{ stage }}"
 team: "{{ team }}"
 stringData:
 name: "{{ vclusterName }}"
 server: "https://{{ vclusterName }}.{{ namespace }}:443"
 config: |
 {
 "tlsClientConfig": {
 "insecure": false,
 "caData": "{{ ca }}",
 "certData": "{{ cert }}",
 "keyData": "{{ key }}"
 }
 }

Putting all the Pieces Together

As we have seen, each component of the KEBAP stack plays a crucial role in the overall architecture. Let’s how the different workflows look like when we put all the pieces together. We go through the following scenarios:

Ignite the Control Plane Cluster With Pulumi
Ordering a New Development Kubernetes Cluster
Ordering a New Production Kubernetes Cluster
Ordering Other Infrastructure

Ignite the Control Plane Cluster With Pulumi

We use Pulumi code to create the control plane cluster in our cloud provider of choice. After the creation of the control plane, we install the Argo CD in a very bare-bones way:

const argocd = new ArgoCD("argocd", {
 initialObjects: initialObjects
}, {
 providers: {
 kubernetes: k8sProvider,
 },
});

The idea is to let then Argo CD fetch the rest of the configuration from the Git repository and apply it to the cluster. And here comes the power of GitOps: It will also manage the Argo CD itself. This way, we separated the workload running on the cluster from the creation of the cluster itself.

Ordering a New Development Kubernetes Cluster

Starting point is Backstage, where we can order a new development Kubernetes cluster. As we want to keep the costs low for development cluster, we opted here for the vCluster project. The vCluster project allows us to create a Kubernetes cluster on demand, which runs as containers on the control plane cluster. The moment the cluster is created, we add it to the Argo CD as spoke cluster. This way, we can roll out applications to the development cluster in the same way as we would do for production clusters.

See the definition of the ClusterPolicy object in the Kyverno section.

Ordering a New Production Kubernetes Cluster

The Backstage catalog is again the starting point to order a new production Kubernetes cluster. This time, we use the Pulumi code to create the cluster in the cloud provider of choice. After the creation of the cluster, we add it to the Argo CD as a production cluster. This way, we can roll out applications to the production cluster in the same way as we would do for development clusters.

Ordering Other Infrastructure

Any other infrastructure components such as databases, message queues, and storage will be also ordered via Backstage. And similar to the ordering of a new production Kubernetes cluster, we use Pulumi to create the cloud infrastructure.

Conclusion

A lot of organizations are building their internal developer platforms using CNCF projects to fulfill the needs of their internal customers. The KEBAP stack provides a structured approach to integrate these technologies into an effective platform engineering solution. We also saw that Pulumi plays a crucial role in this stack by taking care of the management of the cloud infrastructure.

This KEBAP stack offers several benefits, including:

A secure, automated, and auditable environment
Reproducible and programmable infrastructure-as-code
Unified management of cloud resources, Kubernetes, and application deployments
Faster onboarding for new team members
Easier compliance with regulatory requirements due to increased auditability

But there is also a downside: The KEBAP stack requires a significant investment in time and resources to set up and maintain as you are in full charge of the integration of the components as well as operating them. All the projects under the umbrella of CNCF are not automatically implying that they are compatible with each other out of the box. Any platform engineering team will need to invest continuously time and resources to keep the stack up-to-date and running. This is a not negligible investment to consider for any organization.

For this reason, we have another reference architecture that uses components from all of the Pulumi Platform for an end to end fully-managed solution. This solution takes care of the integration of the different components of an internal developer portal and provides a carefree experience of managing your infrastructure at scale.

Continue to upskill and learn about platform engineering:

Read Pulumi Patterns and Practices (P3): A Pulumi-based reference architecture for large-scale organizations
Learn how to build Developer Portals with Pulumi.
Register for one of our Platform Engineering or DevOps workshops.

Pulumi vs HCL: Understanding the Language Differences in Infrastructure as Code

Engin Diri — Tue, 09 Jul 2024 00:00:00 +0000

The Java Language Architect at Oracle, Brian Goetz, author of Java Concurrency in Practice, has commented how declarative languages can be a double-edged sword:

HashiCorp’s infrastructure as code solution, Terraform, uses a domain-specific language (DSL) to declare cloud resources. Pulumi’s infrastructure as code solution, on the other hand, lets you choose from any number of modern languages – C#, Java, JavaScript, Go, Python, or TypeScript – or the industry-standard markup language YAML, to declare cloud resources. Although both Terraform and Pulumi are declarative infrastructure as code engines at their core, this fundamentally different approach to expression languages has significant consequences.

Real-World Consequences

We’ve worked with 1,000s of companies and 100,000+ developers and infrastructure experts alike to adopt infrastructure as code using Pulumi and modern languages, and we see several significant recurring themes as they migrate away from Terraform and HCL.

Tapping Into Millions of Developers

Pulumi allows developers to use widely-adopted general-purpose languages for infrastructure as code. This approach taps into vast existing developer communities, with JavaScript boasting over 17 million developers, Python 15 million, and Go 3 million worldwide. By utilizing these languages, Pulumi enables teams to leverage their current programming expertise and access extensive resources, libraries, and tools. Meanwhile, HCL maintains a dedicated user base of several hundred thousand developers who appreciate its specialized syntax for infrastructure description.

Pulumi Leverages Rich Ecosystems of General-Purpose Languages

Pulumi harnesses the extensive ecosystems surrounding popular programming languages for infrastructure as code. While domain-specific languages like HCL have their strengths, Pulumi’s approach offers developers access to:

Comprehensive IDE support, including features like autocompletion, syntax highlighting, and real-time error detection, which may be more advanced than those available for specialized IaC languages
Wide array of established linting tools to enforce coding standards and catch potential issues early in the development process
Robust testing frameworks that allow for thorough verification and validation of infrastructure configurations, often more mature than those specific to IaC languages
Extensive libraries and packages that can be seamlessly integrated into infrastructure code, expanding possibilities beyond traditional IaC toolsets

This approach allows teams to utilize familiar tools and workflows, potentially increasing productivity and code quality. Developers can leverage their existing programming expertise and the vast resources available in these language communities when working on infrastructure projects, which may require less additional learning compared to adopting a specialized syntax

Pulumi Embraces Open-Source Languages and Principles

Languages like Python, JavaScript, and Go are developed and maintained by diverse global communities, ensuring transparency and collective innovation. These languages have remained open-source throughout their history, with their core components freely available for use, modification, and distribution.

In contrast, some domain-specific IaC languages have different ownership and licensing models. For example, HCL has always been under the control of a single company. Recently, there have been changes in its licensing, with parts of its execution engine moving to a “source available” license that is no longer considered fully open-source.

By using mainstream open-source languages, Pulumi offers teams the benefits of well-established ecosystems, community-driven development, and the assurance of continued open availability. This approach can provide additional confidence for organizations concerned about long-term access and evolution of their IaC tooling.

Here’s a list of languages Pulumi commonly uses, along with their licenses:

Language	License
C#	MIT (.NET Core)
Java	GNU v2.0 (OpenJDK)
JavaScript	MIT (ECMAScript), BSD-style 3-Clause (V8)
Go	BSD-style 3-Clause
Python	BSD-style Python Software Foundation
TypeScript	Apache 2.0
YAML	Creative Commons Attribution

Developers Onboard Quicker with Pulumi

While HCL serves its purpose in infrastructure as code, many developers find Pulumi’s approach more appealing. Pulumi allows engineers to use familiar programming languages, making it easier to leverage existing skills and tools.

This familiarity often leads to faster onboarding and more efficient debugging processes. Pulumi’s flexibility enables developers to create custom solutions with ease, tapping into the full power of general-purpose languages. Additionally,

Pulumi’s open architecture provides developers with the freedom to integrate various tools and services, fostering a diverse and adaptable ecosystem for infrastructure management.

Enhancing Team Collaboration With Pulumi

Pulumi offers several advantages for fostering better collaboration across teams compared to HCL. While HCL is a powerful tool primarily utilized by operations and DevOps teams, its specialized nature for infrastructure tasks might limit broader engagement with development teams who use other programming languages.

Pulumi, on the other hand, supports multiple languages, allowing development teams to use familiar tools and languages which can enhance collaboration and streamline processes. Additionally, Pulumi’s flexibility in integrating with a variety of tools can help bridge gaps between different teams, ensuring smoother communication and more effective teamwork.

This inclusive approach not only accelerates progress but also democratizes access to infrastructure management, avoiding bottlenecks and enhancing overall team efficiency.

Reducing Redundancy and Copy-Paste Challenges

HCL often encounters a significant challenge with “copy-pasta,” where developers frequently copy and paste code. This practice can lead to redundant configurations across environments, bloating the codebase and complicating management.

Additionally, this increases the risk of errors as small mistakes can propagate across multiple instances, and each one must be manually updated when changes are needed, heightening the chance for inconsistencies. As projects evolve, maintaining a codebase with extensive copy-pasta becomes burdensome, slowing development and elevating the likelihood of bugs.

In contrast, Pulumi reduces the need for such redundancies by enabling more dynamic code reuse and better resource management, streamlining updates and fostering more robust and error-resistant infrastructure development.

Reduce Need for Custom Tooling

As projects expand, managing HCL configurations becomes increasingly complex, often necessitating custom tools for efficient large-scale management. While HCL is adept at defining infrastructure as code, it lacks built-in features to orchestrate, validate, and deploy configurations across extensive networks smoothly. Teams usually need to develop bespoke tools or integrate external solutions like Terragrunt, Atlantis, and Sentinel to automate tasks, ensure consistency, and facilitate integration with other systems. This process can demand significant resources and expertise.

In contrast, Pulumi is designed to scale seamlessly with projects, reducing the need for external tools or extensive custom solutions. Pulumi provides robust capabilities for managing large-scale projects efficiently, streamlining the deployment process, and ensuring consistency across various environments without the overhead of additional tooling. This makes Pulumi a more straightforward and resource-efficient choice for teams looking to manage their infrastructure effectively as they grow.

Future-Proofing Your Infrastructure

HCL, while robust, lacks the future-proof qualities of Pulumi SDKs due to its fixed syntax and usage constraints. In contrast, Pulumi’s support for multiple programming languages like Python, TypeScript, and Go, coupled with the potential to adopt new languages over time, offers unparalleled flexibility and adaptability.

This allows developers to write infrastructure as code in familiar languages, facilitating easier integration with existing projects and workflows. As technology progresses, the capacity to embrace new programming languages and paradigms becomes essential. Pulumi’s adaptable model supports this continuous evolution, ensuring that infrastructure as code practices can keep pace with the dynamic nature of development environments and preferences, thereby offering a more sustainable and scalable solution for long-term projects.

Making Infrastructure Code Simpler

As HCL continues to evolve, its fixed syntax and structured approach are likely to lead to greater complexity. This evolution could make the language increasingly specialized and challenging for newcomers, particularly those already proficient in more widely-used programming languages. As HCL adapts more bespoke features to meet specific needs, it may become less intuitive and harder to handle effectively, potentially resulting in a language that is cumbersome to use and maintain, especially in long-term projects.

In contrast, Pulumi offers a more streamlined and adaptable alternative. Supporting multiple mainstream programming languages like Python, TypeScript, and Go, Pulumi enables developers to work with tools and languages they are already familiar with, greatly reducing the learning curve and enhancing productivity. This flexibility allows Pulumi to integrate seamlessly into existing projects and workflows, making it a more sustainable and scalable choice for modern infrastructure needs.

HCL in the Wild: Practical Comparisons With Pulumi

In this section, we’ll explore real-life examples to compare and contrast HCL with Pulumi programs. By examining how each approach handles similar infrastructure tasks, you’ll quickly appreciate the advantages of leveraging a familiar programming language through Pulumi.

This comparison aims to highlight the practical benefits and increased efficiency that Pulumi offers over traditional HCL configurations.

`for_each` Loop With Conditional Expression

First, let’s look at an HCL example featuring a for_each loop with a conditional statement. At first glance, it may seem straightforward, but we can immediately see the unique way a loop must be defined in HCL.

Initially, you must decide whether to use count or for_each. Following this, you must adhere to the strict format for implementing conditional statements.

This contrasts starkly with how you would use these constructs in a generic programming language.

variable "storage_account_names" {
 type = list(string)
 default = ["mickey", "donald", "goofy"]
}

resource "azurerm_resource_group" "example" {
 name = "storage-rg"
 location = "West Europe"
}

resource "azurerm_storage_account" "my_storage" {
 for_each = {
 for name in var.storage_account_names :
 name => name
 if name != "goofy"
 }
 name = each.value
 resource_group_name = azurerm_resource_group.example.name
 location = azurerm_resource_group.example.location
 account_tier = "Standard"
 account_replication_type = "GRS"
}

Now let’s look at how we can approach the same example using Pulumi.

const storageAccountNames: string[] = [
 "mickey",
 "donald",
 "goofy",
];
const example = new azure.core.ResourceGroup("example", {
 name: "storage-rg",
 location: "West Europe",
});

storageAccountNames.filter((name: string) => name !== "goofy").forEach((name: string) => {
 new azure.storage.Account(`my_storage-${name}`, {
 name: name,
 resourceGroupName: example.name,
 location: example.location,
 accountTier: "Standard",
 accountReplicationType: "GRS",
 });
});

import pulumi
import pulumi_azure as azure

storage_account_names = ["mickey", "donald", "goofy"]

example = azure.core.ResourceGroup(
 "example",
 name="storage-rg",
 location="West Europe"
)

for name in filter(lambda n: n != "goofy", storage_account_names):
 azure.storage.Account(
 f"my_storage-{name}",
 name=name,
 resource_group_name=example.name,
 location=example.location,
 account_tier="Standard",
 account_replication_type="GRS"
 )

package main

import (
 "github.com/pulumi/pulumi-azure/sdk/v5/go/azure/core"
 "github.com/pulumi/pulumi-azure/sdk/v5/go/azure/storage"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 storageAccountNames := []string{"mickey", "donald", "goofy"}

 example, err := core.NewResourceGroup(ctx, "example", &core.ResourceGroupArgs{
 Name: pulumi.String("storage-rg"),
 Location: pulumi.String("West Europe"),
 })
 if err != nil {
 return err
 }

 for _, name := range storageAccountNames {
 if name != "goofy" {
 _, err := storage.NewAccount(ctx, "my_storage-"+name, &storage.AccountArgs{
 Name: pulumi.String(name),
 ResourceGroupName: example.Name,
 Location: example.Location,
 AccountTier: pulumi.String("Standard"),
 AccountReplicationType: pulumi.String("GRS"),
 })
 if err != nil {
 return err
 }
 }
 }

 return nil
 })
}

using Pulumi;
using Pulumi.Azure.Core;
using Pulumi.Azure.Storage;
using System.Collections.Generic;

class Program
{
 static Task<int> Main() => Deployment.RunAsync<MyStack>();
}

class MyStack : Stack
{
 public MyStack()
 {
 var storageAccountNames = new List<string> { "mickey", "donald", "goofy" };

 var resourceGroup = new ResourceGroup("example", new ResourceGroupArgs
 {
 Name = "storage-rg",
 Location = "West Europe",
 });

 foreach (var name in storageAccountNames)
 {
 if (name != "goofy")
 {
 var storageAccount = new Account($"my_storage-{name}", new AccountArgs
 {
 Name = name,
 ResourceGroupName = resourceGroup.Name,
 Location = resourceGroup.Location,
 AccountTier = "Standard",
 AccountReplicationType = "GRS",
 });
 }
 }
 }
}

import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.azurenative.core.ResourceGroup;
import com.pulumi.azurenative.core.ResourceGroupArgs;
import com.pulumi.azurenative.storage.Account;
import com.pulumi.azurenative.storage.AccountArgs;

import java.util.List;

public class Main {
 public static void main(String[] args) {
 Pulumi.run(Main::stack);
 }

 public static void stack(Context ctx) {
 var storageAccountNames = List.of("mickey", "donald", "goofy");

 var resourceGroup = new ResourceGroup("example", ResourceGroupArgs.builder()
 .name("storage-rg")
 .location("West Europe")
 .build());

 for (var name : storageAccountNames) {
 if (!name.equals("goofy")) {
 var storageAccount = new Account("my_storage-" + name, AccountArgs.builder()
 .name(name)
 .resourceGroupName(resourceGroup.name())
 .location(resourceGroup.location())
 .kind("StorageV2")
 .sku(AccountSkuArgs.builder()
 .name("Standard_GRS")
 .tier("Standard")
 .build())
 .build());
 }
 }
 }
}

We can see that we leverage the full potential that the generic programming language offers, using filter to streamline our array without the need for conditional statements, and then iterating through the reduced array with forEach. This approach is less error-prone, faster, and easier to debug.

However, implementing a for_each in HCL can become even more complicated.

`for_each` Loop With Multi-Level Nested Dynamic Blocks

To give you an idea of how HCL handles cascading nested dynamic providers, consider this: dynamic providers are intended to dynamically construct repeatable nested blocks. However, this complexity can make the code difficult to read and maintain. We will examine a specific resource from this GitHub repository to provide a clearer illustration:

variable "enable_auto_scaling" {
 type = bool
 default = false
 description = "Enable node pool autoscaling"
}

variable "agents_pool_kubelet_configs" {
 type = list(object({
 cpu_manager_policy = optional(string)
 cpu_cfs_quota_enabled = optional(bool, true)
 cpu_cfs_quota_period = optional(string)
 image_gc_high_threshold = optional(number)
 image_gc_low_threshold = optional(number)
 topology_manager_policy = optional(string)
 allowed_unsafe_sysctls = optional(set(string))
 container_log_max_size_mb = optional(number)
 container_log_max_line = optional(number)
 pod_max_pid = optional(number)
 }))
 default = [
 {
 cpu_manager_policy = "none"
 cpu_cfs_quota_enabled = true
 cpu_cfs_quota_period = "100ms"
 image_gc_high_threshold = 85
 image_gc_low_threshold = 65
 topology_manager_policy = "none"
 allowed_unsafe_sysctls = ["net.*", "kernel.*"]
 container_log_max_size_mb = 10
 container_log_max_line = 10
 pod_max_pid = 100
 }
 ]
 nullable = false
}

resource "azurerm_resource_group" "example" {
 name = "storage-rg"
 location = "West Europe"
}

resource "azurerm_kubernetes_cluster" "main" {
 name = "example-aks"
 location = azurerm_resource_group.example.location
 resource_group_name = azurerm_resource_group.example.name
 dns_prefix = "exampleaks"

 dynamic "default_node_pool" {
 for_each = var.enable_auto_scaling == true ? [] : ["default_node_pool_manually_scaled"]
 content {
 name = "default"
 vm_size = "Standard_DS2_v2"
 zones = [1, 2, 3]
 enable_auto_scaling = true
 min_count = 1
 max_count = 3
 dynamic "kubelet_config" {
 for_each = var.agents_pool_kubelet_configs
 content {
 allowed_unsafe_sysctls = kubelet_config.value.allowed_unsafe_sysctls
 }
 }
 }
 }
 identity {
 type = "SystemAssigned"
 }
}

As we can see, incorporating even one nested dynamic provider block can make the code difficult to read. We recommend using dynamic blocks sparingly to maintain readability and ease of maintenance, particularly when creating an abstraction with the help of a module. Overuse of these blocks, especially when defining a resource’s arguments and nested blocks directly from input variables, might not provide a useful abstraction.

Have look how this would look like when using Pulumi:

const config = new pulumi.Config();
const enableAutoScaling = config.getBoolean("enableAutoScaling") || false;

const agentsPoolKubeletConfigs = config.getObject("agentsPoolKubeletConfigs") || {
 allowedUnsafeSysctls: [
 "kernel.*",
 "net.*",
 ],
 containerLogMaxLine: 10,
 containerLogMaxSizeMb: 10,
 cpuCfsQuotaEnabled: true,
 cpuCfsQuotaPeriod: "100ms",
 cpuManagerPolicy: "none",
 imageGcHighThreshold: 85,
 imageGcLowThreshold: 65,
 podMaxPid: 100,
 topologyManagerPolicy: "none",
};
const example = new azure.core.ResourceGroup("example", {
 name: "storage-rg",
 location: "West Europe",
});

let kubeletConfigs: azure.types.input.containerservice.KubernetesClusterDefaultNodePoolKubeletConfig = agentsPoolKubeletConfigs;

let defaultNodePool: azure.types.input.containerservice.KubernetesClusterDefaultNodePool = {
 name: "default",
 vmSize: "Standard_DS2_v2",
 zones: [
 "1",
 "2",
 "3",
 ],
 enableAutoScaling: enableAutoScaling,
 minCount: 1,
 maxCount: 3,
 kubeletConfig: kubeletConfigs,
};

import pulumi
import pulumi_azure as azure

config = pulumi.Config()
enable_auto_scaling = config.get_bool("enableAutoScaling") or False

agents_pool_kubelet_configs = config.get_object("agentsPoolKubeletConfigs") or {
 "allowedUnsafeSysctls": [
 "kernel.*",
 "net.*",
 ],
 "containerLogMaxLine": 10,
 "containerLogMaxSizeMb": 10,
 "cpuCfsQuotaEnabled": True,
 "cpuCfsQuotaPeriod": "100ms",
 "cpuManagerPolicy": "none",
 "imageGcHighThreshold": 85,
 "imageGcLowThreshold": 65,
 "podMaxPid": 100,
 "topologyManagerPolicy": "none",
}

example = azure.core.ResourceGroup("example",
 name="storage-rg",
 location="West Europe")

kubelet_configs = azure.containerservice.KubernetesClusterDefaultNodePoolKubeletConfigArgs(
 **agents_pool_kubelet_configs
)

default_node_pool = azure.containerservice.KubernetesClusterDefaultNodePoolArgs(
 name="default",
 vm_size="Standard_DS2_v2",
 zones=["1", "2", "3"],
 enable_auto_scaling=enable_auto_scaling,
 min_count=1,
 max_count=3,
 kubelet_config=kubelet_configs,
)

package main

import (
 "github.com/pulumi/pulumi-azure/sdk/v5/go/azure/core"
 "github.com/pulumi/pulumi-azure/sdk/v5/go/azure/containerservice"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 enableAutoScaling := false

 agentsPoolKubeletConfigs := map[string]interface{}{
 "allowedUnsafeSysctls": []string{
 "kernel.*",
 "net.*",
 },
 "containerLogMaxLine": 10,
 "containerLogMaxSizeMb": 10,
 "cpuCfsQuotaEnabled": true,
 "cpuCfsQuotaPeriod": "100ms",
 "cpuManagerPolicy": "none",
 "imageGcHighThreshold": 85,
 "imageGcLowThreshold": 65,
 "podMaxPid": 100,
 "topologyManagerPolicy": "none",
 }

 example, err := core.NewResourceGroup(ctx, "example", &core.ResourceGroupArgs{
 Name: pulumi.String("storage-rg"),
 Location: pulumi.String("West Europe"),
 })
 if err != nil {
 return err
 }

 kubeletConfigs := &containerservice.KubernetesClusterDefaultNodePoolKubeletConfigArgs{
 AllowedUnsafeSysctls: pulumi.StringArray{
 pulumi.String("kernel.*"),
 pulumi.String("net.*"),
 },
 ContainerLogMaxLine: pulumi.Int(agentsPoolKubeletConfigs["containerLogMaxLine"].(int)),
 ContainerLogMaxSizeMb: pulumi.Int(agentsPoolKubeletConfigs["containerLogMaxSizeMb"].(int)),
 CpuCfsQuotaEnabled: pulumi.Bool(agentsPoolKubeletConfigs["cpuCfsQuotaEnabled"].(bool)),
 CpuCfsQuotaPeriod: pulumi.String(agentsPoolKubeletConfigs["cpuCfsQuotaPeriod"].(string)),
 CpuManagerPolicy: pulumi.String(agentsPoolKubeletConfigs["cpuManagerPolicy"].(string)),
 ImageGcHighThreshold: pulumi.Int(agentsPoolKubeletConfigs["imageGcHighThreshold"].(int)),
 ImageGcLowThreshold: pulumi.Int(agentsPoolKubeletConfigs["imageGcLowThreshold"].(int)),
 PodMaxPid: pulumi.Int(agentsPoolKubeletConfigs["podMaxPid"].(int)),
 TopologyManagerPolicy: pulumi.String(agentsPoolKubeletConfigs["topologyManagerPolicy"].(string)),
 }

 _, err = containerservice.NewKubernetesClusterDefaultNodePool(ctx, "defaultNodePool", &containerservice.KubernetesClusterDefaultNodePoolArgs{
 Name: pulumi.String("default"),
 VmSize: pulumi.String("Standard_DS2_v2"),
 Zones: pulumi.StringArray{pulumi.String("1"), pulumi.String("2"), pulumi.String("3")},
 EnableAutoScaling: pulumi.Bool(enableAutoScaling),
 MinCount: pulumi.Int(1),
 MaxCount: pulumi.Int(3),
 KubeletConfig: kubeletConfigs,
 }, pulumi.Parent(example))
 if err != nil {
 return err
 }

 return nil
 })
}

using Pulumi;
using Pulumi.AzureNative.Resources;
using Pulumi.AzureNative.ContainerService;
using System.Collections.Generic;

class MyStack : Stack
{
 public MyStack()
 {
 var enableAutoScaling = false;

 var agentsPoolKubeletConfigs = new Dictionary<string, object>
 {
 { "allowedUnsafeSysctls", new string[] { "kernel.*", "net.*" } },
 { "containerLogMaxLine", 10 },
 { "containerLogMaxSizeMb", 10 },
 { "cpuCfsQuotaEnabled", true },
 { "cpuCfsQuotaPeriod", "100ms" },
 { "cpuManagerPolicy", "none" },
 { "imageGcHighThreshold", 85 },
 { "imageGcLowThreshold", 65 },
 { "podMaxPid", 100 },
 { "topologyManagerPolicy", "none" }
 };

 var resourceGroup = new ResourceGroup("example", new ResourceGroupArgs
 {
 Name = "storage-rg",
 Location = "West Europe"
 });

 var kubeletConfigArgs = new KubernetesClusterDefaultNodePoolKubeletConfigArgs
 {
 AllowedUnsafeSysctls = agentsPoolKubeletConfigs["allowedUnsafeSysctls"] as string[],
 ContainerLogMaxLine = agentsPoolKubeletConfigs["containerLogMaxLine"] as int?,
 ContainerLogMaxSizeMb = agentsPoolKubeletConfigs["containerLogMaxSizeMb"] as int?,
 CpuCfsQuotaEnabled = agentsPoolKubeletConfigs["cpuCfsQuotaEnabled"] as bool?,
 CpuCfsQuotaPeriod = agentsPoolKubeletConfigs["cpuCfsQuotaPeriod"] as string,
 CpuManagerPolicy = agentsPoolKubeletConfigs["cpuManagerPolicy"] as string,
 ImageGcHighThreshold = agentsPoolKubeletConfigs["imageGcHighThreshold"] as int?,
 ImageGcLowThreshold = agentsPoolKubeletConfigs["imageGcLowThreshold"] as int?,
 PodMaxPid = agentsPoolKubeletConfigs["podMaxPid"] as int?,
 TopologyManagerPolicy = agentsPoolKubeletConfigs["topologyManagerPolicy"] as string
 };

 var defaultNodePool = new KubernetesClusterDefaultNodePool("defaultNodePool", new KubernetesClusterDefaultNodePoolArgs
 {
 Name = "default",
 VmSize = "Standard_DS2_v2",
 Zones = new[] { "1", "2", "3" },
 EnableAutoScaling = enableAutoScaling,
 MinCount = 1,
 MaxCount = 3,
 KubeletConfig = kubeletConfigArgs
 }, new CustomResourceOptions { Parent = resourceGroup });
 }
}

class Program
{
 static Task<int> Main() => Deployment.RunAsync<MyStack>();
}

import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.azure.core.ResourceGroup;
import com.pulumi.azure.core.inputs.ResourceGroupArgs;
import com.pulumi.azure.containerservice.KubernetesClusterDefaultNodePool;
import com.pulumi.azure.containerservice.inputs.KubernetesClusterDefaultNodePoolArgs;
import com.pulumi.azure.containerservice.inputs.KubernetesClusterDefaultNodePoolKubeletConfigArgs;

import java.util.Map;
import java.util.HashMap;

public class MyStack {
 public static void main(String[] args) {
 Pulumi.run(ctx -> {
 boolean enableAutoScaling = false;

 Map<String, Object> agentsPoolKubeletConfigs = new HashMap<>();
 agentsPoolKubeletConfigs.put("allowedUnsafeSysctls", new String[]{"kernel.*", "net.*"});
 agentsPoolKubeletConfigs.put("containerLogMaxLine", 10);
 agentsPoolKubeletConfigs.put("containerLogMaxSizeMb", 10);
 agentsPoolKubeletConfigs.put("cpuCfsQuotaEnabled", true);
 agentsPoolKubeletConfigs.put("cpuCfsQuotaPeriod", "100ms");
 agentsPoolKubeletConfigs.put("cpuManagerPolicy", "none");
 agentsPoolKubeletConfigs.put("imageGcHighThreshold", 85);
 agentsPoolKubeletConfigs.put("imageGcLowThreshold", 65);
 agentsPoolKubeletConfigs.put("podMaxPid", 100);
 agentsPoolKubeletConfigs.put("topologyManagerPolicy", "none");

 var resourceGroup = new ResourceGroup("example", ResourceGroupArgs.builder()
 .name("storage-rg")
 .location("West Europe")
 .build());

 var kubeletConfigArgs = KubernetesClusterDefaultNodePoolKubeletConfigArgs.builder()
 .allowedUnsafeSysctls((String[]) agentsPoolKubeletConfigs.get("allowedUnsafeSysctls"))
 .containerLogMaxLine((Integer) agentsPoolKubeletConfigs.get("containerLogMaxLine"))
 .containerLogMaxSizeMb((Integer) agentsPoolKubeletConfigs.get("containerLogMaxSizeMb"))
 .cpuCfsQuotaEnabled((Boolean) agentsPoolKubeletConfigs.get("cpuCfsQuotaEnabled"))
 .cpuCfsQuotaPeriod((String) agentsPoolKubeletConfigs.get("cpuCfsQuotaPeriod"))
 .cpuManagerPolicy((String) agentsPoolKubeletConfigs.get("cpuManagerPolicy"))
 .imageGcHighThreshold((Integer) agentsPoolKubeletConfigs.get("imageGcHighThreshold"))
 .imageGcLowThreshold((Integer) agentsPoolKubeletConfigs.get("imageGcLowThreshold"))
 .podMaxPid((Integer) agentsPoolKubeletConfigs.get("podMaxPid"))
 .topologyManagerPolicy((String) agentsPoolKubeletConfigs.get("topologyManagerPolicy"))
 .build();

 var defaultNodePool = new KubernetesClusterDefaultNodePool("defaultNodePool", KubernetesClusterDefaultNodePoolArgs.builder()
 .name("default")
 .vmSize("Standard_DS2_v2")
 .zones(new String[]{"1", "2", "3"})
 .enableAutoScaling(enableAutoScaling)
 .minCount(1)
 .maxCount(3)
 .kubeletConfig(kubeletConfigArgs)
 .build(), new CustomResourceOptions().parent(resourceGroup));
 });
 }
}

In Pulumi, using TypeScript as an example, we can easily create new objects of a specific type and populate them with values from our configuration. If we need to add conditions, we can include our usual if statements. This approach greatly simplifies the code, making it easy to read and maintain for any future changes.

Execute Programming Code With `null_resource`, `local_file` and `provisioners`

There are sometimes situations where there is a strong need to execute a script written in a programming language. Most prominent use case is packaging up the source code for an AWS lambda function. Let’s have a look into an example from the Terraform AWS Lambda module

locals {
 python = (substr(pathexpand("~"), 0, 1) == "/") ? "python3" : "python.exe"
}

# Generates a filename for the zip archive based on the content of the files
# in source_path. The filename will change when the source code changes.
data "external" "archive_prepare" {
 count = var.create && var.create_package ? 1 : 0
 program = [local.python, "${path.module}/package.py", "prepare"]
 query = {
 paths = jsonencode({
 module = path.module
 root = path.root
 cwd = path.cwd
 })
 docker = var.build_in_docker ? jsonencode({
 docker_pip_cache = var.docker_pip_cache
 docker_build_root = var.docker_build_root
 docker_file = var.docker_file
 docker_image = var.docker_image
 with_ssh_agent = var.docker_with_ssh_agent
 docker_additional_options = var.docker_additional_options
 docker_entrypoint = var.docker_entrypoint
 }) : null
 artifacts_dir = var.artifacts_dir
 runtime = var.runtime
 source_path = jsonencode(var.source_path)
 hash_extra = var.hash_extra
 hash_extra_paths = jsonencode(
 [
 # Temporary fix when building from multiple locations
 # We should take into account content of package.py when counting hash
 # Related issue: https://github.com/terraform-aws-modules/terraform-aws-lambda/issues/63
 # "${path.module}/package.py"
 ]
 )
 recreate_missing_package = var.recreate_missing_package
 }
}

# This transitive resource used as a bridge between a state stored
# in a Terraform plan and a call of a build command on the apply stage
# to transfer a noticeable amount of data
resource "local_file" "archive_plan" {
 count = var.create && var.create_package ? 1 : 0
 content = data.external.archive_prepare[0].result.build_plan
 filename = data.external.archive_prepare[0].result.build_plan_filename
 directory_permission = "0755"
 file_permission = "0644"
}

# Build the zip archive whenever the filename changes.
resource "null_resource" "archive" {
 count = var.create && var.create_package ? 1 : 0
 triggers = {
 filename = data.external.archive_prepare[0].result.filename
 timestamp = var.trigger_on_package_timestamp ? data.external.archive_prepare[0].result.timestamp : null
 }
 provisioner "local-exec" {
 interpreter = [
 local.python, "${path.module}/package.py", "build",
 "--timestamp", data.external.archive_prepare[0].result.timestamp
 ]
 command = data.external.archive_prepare[0].result.build_plan_filename
 }
 depends_on = [local_file.archive_plan]
}

We see a confusing mix of different providers (local and null_resource) and even the use of provisioner here. This setup is overly complex to read and maintain, leading to a split-brain situation where some logic is in the code and the rest in HCL. It also becomes very challenging to debug, especially if the team lacks experience with the Python code.

In Pulumi, simplicity is key: simply write your Python code and seamlessly integrate it with Pulumi SDK calls. No additional steps are necessary, and there’s no need to run the Python binary separately to execute the file. It’s all about streamlining the process to make your life easier. For instance, consider this example where we read the name of an Azure Resource Group from a JSON file (config.json, illustrating how effortlessly Python can blend with Pulumi’s infrastructure management capabilities.

import json
import pulumi
import pulumi_azure_native as azure_native

# Load the JSON file
with open('config.json') as json_file:
 config = json.load(json_file)

# Extract values from the JSON file
resource_group_name = config['resourceGroupName']
location = config['location']

# Create an Azure Resource Group
resource_group = azure_native.resources.ResourceGroup(
 "resourceGroup",
 resource_group_name=resource_group_name,
 location=location
)

# Export the name of the resource group
pulumi.export('resourceGroupName', resource_group.name)

If Statements Constructed With `count`

variable "mode" {
 default = "not-spoke"
}

variable "restricted_net_hub_project_id" {
 default = "xxx"
}

data "google_compute_network" "vpc_restricted_net_hub" {
 count = var.mode == "spoke" ? 1 : 0
 name = "vpc-c-shared-restricted-hub"
 project = var.restricted_net_hub_project_id
}

In this HCL example, we see the count parameter being used in a way that stretches its original intent—essentially using it to conditionally create or skip resources based on the value of var.mode. This method, while functional, can make the code more difficult to understand and maintain, as it repurposes Terraform constructs in unintended ways.

Contrast this with Pulumi, which allows you to use native if statements from the programming language, such as TypeScript or Python. This native approach enhances readability and maintainability because it aligns with standard programming practices, making it easier for developers to follow and debug. Pulumi’s use of familiar programming constructs means you can write more intuitive and straightforward infrastructure code without bending the tool to fit unusual patterns.

import * as pulumi from "@pulumi/pulumi";
import * as gcp from "@pulumi/gcp";

const config = new pulumi.Config();
const mode = config.get("mode") || "";
const restrictedNetHubProjectId = config.get("restrictedNetHubProjectId") || "";
if (mode == "spoke") {
 const vpcSharedSpoke = gcp.compute.getNetworkOutput({
 name: "vpc-c-shared-spoke",
 project: restrictedNetHubProjectId,
 });
}

import pulumi
import pulumi_gcp as gcp

config = pulumi.Config()
mode = config.get("mode") or ""
restricted_net_hub_project_id = config.get("restrictedNetHubProjectId") or ""

if mode == "spoke":
 vpc_shared_spoke = gcp.compute.get_network_output(
 name="vpc-c-shared-spoke",
 project=restricted_net_hub_project_id,
 )

package main

import (
 "github.com/pulumi/pulumi-gcp/sdk/v7/go/gcp/compute"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 mode := ""

 if mode == "spoke" {
 _, err := compute.LookupNetwork(ctx, &compute.LookupNetworkArgs{
 Name: "vpc-c-shared-spoke",
 Project: "your-restrictedNetHubProjectId",
 })

 if err != nil {
 return err
 }
 }

 return nil
 })
}

using Pulumi;
using Pulumi.Gcp.Compute;

class Program
{
 static Task<int> Main() => Deployment.RunAsync<ComputeStack>();

 class ComputeStack : Stack
 {
 public ComputeStack()
 {
 var mode = "";

 if (mode == "spoke")
 {
 var network = Output.Tuple(Output.Create("vpc-c-shared-spoke"), Output.Create("your-restrictedNetHubProjectId"))
 .Apply(tuple => GetNetwork.InvokeAsync(new GetNetworkArgs
 {
 Name = tuple.Item1,
 Project = tuple.Item2
 }));

 // Handle potential errors if necessary
 network.Apply(n =>
 {
 if (n == null)
 {
 throw new Exception("Network not found");
 }
 });
 }
 }
 }
}

import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.gcp.compute.ComputeFunctions;
import com.pulumi.gcp.compute.inputs.GetNetworkArgs;

public class Main {
 public static void main(String[] args) {
 Pulumi.run(ctx -> {
 String mode = "";

 if ("spoke".equals(mode)) {
 Output<Tuple<String, String>> networkData = Output.tuple(Output.of("vpc-c-shared-spoke"), Output.of("your-restrictedNetHubProjectId"));

 Output<ComputedNetwork> network = networkData.applyValue(tuple ->
 ComputeFunctions.getNetwork(GetNetworkArgs.builder()
 .name(tuple.t1)
 .project(tuple.t2)
 .build()
 )
 );

 network.applyValue(n -> {
 if (n == null) {
 throw new RuntimeException("Network not found");
 }
 return null;
 });
 }
 });
 }
}

HCL Function Frenzy

Another source of extreme confusion and constant re-learning is the heavy usage of built-in functions within HCL expressions. At first glance, while it seems helpful for performing complex operations quickly, it can lead to overly complex expressions.

Here is an example of very complex expressions involving extensive function usage from the Google Network module:

locals {
 local_network_name = element(reverse(split("/", var.local_network)), 0)
 peer_network_name = element(reverse(split("/", var.peer_network)), 0)

 local_network_peering = "${var.prefix}-${local.local_network_name}-${local.peer_network_name}"
 local_network_peering_name = length(local.local_network_peering) < 63 ? local.local_network_peering :
 "${substr(local.local_network_peering, 0, min(58, length(local.local_network_peering)))}-${random_string.network_peering_suffix.result}"
 peer_network_peering = "${var.prefix}-${local.peer_network_name}-${local.local_network_name}"
 peer_network_peering_name = length(local.peer_network_peering) < 63 ? local.peer_network_peering :
 "${substr(local.peer_network_peering, 0, min(58, length(local.peer_network_peering)))}-${random_string.network_peering_suffix.result}"
}

In Pulumi, we can use the same functions from the programming language we pick for our infrastructure. This makes things easier because we don’t need to learn new, special functions like in HCL. We can also use extra libraries if we need them. Compared to HCL, where you have to learn unique functions, Pulumi lets you work with tools you might already know, making the whole process simpler and more flexible.

import * as pulumi from "@pulumi/pulumi";

const config = new pulumi.Config();
const localNetwork = config.require("localNetwork");
const peerNetwork = config.require("peerNetwork");
const prefix = config.require("prefix");

const localNetworkName = localNetwork.split("/").reverse()[0];
const peerNetworkName = peerNetwork.split("/").reverse()[0];

const localNetworkPeering = `${prefix}-${localNetworkName}-${peerNetworkName}`;
const peerNetworkPeering = `${prefix}-${peerNetworkName}-${localNetworkName}`;
const networkPeeringSuffix = "peering";
const localNetworkPeeringName = localNetworkPeering.length < 63
 ? localNetworkPeering
 : `${localNetworkPeering.substring(0, Math.min(58, localNetworkPeering.length))}-${networkPeeringSuffix}`;

const peerNetworkPeeringName = peerNetworkPeering.length < 63
 ? peerNetworkPeering
 : `${peerNetworkPeering.substring(0, Math.min(58, peerNetworkPeering.length))}-${networkPeeringSuffix}`;

export const outputs = {
 localNetworkName,
 peerNetworkName,
 localNetworkPeeringName,
 peerNetworkPeeringName,
};

import pulumi

config = pulumi.Config()
local_network = config.require("localNetwork")
peer_network = config.require("peerNetwork")
prefix = config.require("prefix")

local_network_name = local_network.split("/")[::-1][0]
peer_network_name = peer_network.split("/")[::-1][0]

local_network_peering = f"{prefix}-{local_network_name}-{peer_network_name}"
peer_network_peering = f"{prefix}-{peer_network_name}-{local_network_name}"
network_peering_suffix = "peering"

local_network_peering_name = (
 local_network_peering if len(local_network_peering) < 63
 else f"{local_network_peering[:min(58, len(local_network_peering))]}-{network_peering_suffix}"
)

peer_network_peering_name = (
 peer_network_peering if len(peer_network_peering) < 63
 else f"{peer_network_peering[:min(58, len(peer_network_peering))]}-{network_peering_suffix}"
)

pulumi.export("localNetworkName", local_network_name)
pulumi.export("peerNetworkName", peer_network_name)
pulumi.export("localNetworkPeeringName", local_network_peering_name)
pulumi.export("peerNetworkPeeringName", peer_network_peering_name)

package main

import (
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 localNetwork := "your-local-network"
 peerNetwork := "your-peer-network"
 prefix := "your-prefix"

 localNetworkName := reverseString(getNetworkName(localNetwork))
 peerNetworkName := reverseString(getNetworkName(peerNetwork))

 localNetworkPeering := prefix + "-" + localNetworkName + "-" + peerNetworkName
 peerNetworkPeering := prefix + "-" + peerNetworkName + "-" + localNetworkName
 networkPeeringSuffix := "peering"

 localNetworkPeeringName := condenseName(localNetworkPeering, networkPeeringSuffix)
 peerNetworkPeeringName := condenseName(peerNetworkPeering, networkPeeringSuffix)

 ctx.Export("localNetworkName", pulumi.String(localNetworkName))
 ctx.Export("peerNetworkName", pulumi.String(peerNetworkName))
 ctx.Export("localNetworkPeeringName", pulumi.String(localNetworkPeeringName))
 ctx.Export("peerNetworkPeeringName", pulumi.String(peerNetworkPeeringName))

 return nil
 })
}

func getNetworkName(network string) string {
 parts := strings.Split(network, "/")
 return parts[len(parts)-1]
}

func reverseString(s string) string {
 runes := []rune(s)
 for i, j := 0, len(runes)-1; i < j; i, j = i+1, j-1 {
 runes[i], runes[j] = runes[j], runes[i]
 }
 return string(runes)
}

func condenseName(name string, suffix string) string {
 if len(name) < 63 {
 return name
 }
 return name[:min(58, len(name))] + "-" + suffix
}

func min(a, b int) int {
 if a < b {
 return a
 }
 return b
}

using System;
using System.Linq;
using Pulumi;

class Program
{
 static Task<int> Main() => Deployment.RunAsync<MyStack>();

 class MyStack : Stack
 {
 public MyStack()
 {
 var localNetwork = "your-local-network";
 var peerNetwork = "your-peer-network";
 var prefix = "your-prefix";

 var localNetworkName = ReverseString(GetNetworkName(localNetwork));
 var peerNetworkName = ReverseString(GetNetworkName(peerNetwork));

 var localNetworkPeering = $"{prefix}-{localNetworkName}-{peerNetworkName}";
 var peerNetworkPeering = $"{prefix}-{peerNetworkName}-{localNetworkName}";
 var networkPeeringSuffix = "peering";

 var localNetworkPeeringName = CondenseName(localNetworkPeering, networkPeeringSuffix);
 var peerNetworkPeeringName = CondenseName(peerNetworkPeering, networkPeeringSuffix);

 this.Export("localNetworkName", localNetworkName);
 this.Export("peerNetworkName", peerNetworkName);
 this.Export("localNetworkPeeringName", localNetworkPeeringName);
 this.Export("peerNetworkPeeringName", peerNetworkPeeringName);
 }

 private string GetNetworkName(string network)
 {
 var parts = network.Split('/');
 return parts[parts.Length - 1];
 }

 private string ReverseString(string s)
 {
 var chars = s.ToCharArray();
 Array.Reverse(chars);
 return new string(chars);
 }

 private string CondenseName(string name, string suffix)
 {
 if (name.Length < 63) return name;
 return name.Substring(0, Math.Min(58, name.Length)) + "-" + suffix;
 }
 }
}

import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.resources.Stack;

import java.util.Arrays;

public class Program {
 public static void main(String[] args) {
 Pulumi.run(Program::stack);
 }

 public static void stack(Context ctx) {
 var localNetwork = "your-local-network";
 var peerNetwork = "your-peer-network";
 var prefix = "your-prefix";

 var localNetworkName = reverseString(getNetworkName(localNetwork));
 var peerNetworkName = reverseString(getNetworkName(peerNetwork));

 var localNetworkPeering = String.format("%s-%s-%s", prefix, localNetworkName, peerNetworkName);
 var peerNetworkPeering = String.format("%s-%s-%s", prefix, peerNetworkName, localNetworkName);
 var networkPeeringSuffix = "peering";

 var localNetworkPeeringName = condenseName(localNetworkPeering, networkPeeringSuffix);
 var peerNetworkPeeringName = condenseName(peerNetworkPeering, networkPeeringSuffix);

 ctx.export("localNetworkName", Output.of(localNetworkName));
 ctx.export("peerNetworkName", Output.of(peerNetworkName));
 ctx.export("localNetworkPeeringName", Output.of(localNetworkPeeringName));
 ctx.export("peerNetworkPeeringName", Output.of(peerNetworkPeeringName));
 }

 private static String getNetworkName(String network) {
 var parts = network.split("/");
 return parts[parts.length - 1];
 }

 private static String reverseString(String s) {
 var chars = s.toCharArray();
 for (int i = 0, j = chars.length - 1; i < j; i++, j--) {
 char temp = chars[i];
 chars[i] = chars[j];
 chars[j] = temp;
 }
 return new String(chars);
 }

 private static String condenseName(String name, String suffix) {
 if (name.length() < 63) return name;
 return name.substring(0, Math.min(58, name.length())) + "-" + suffix;
 }
}

Pitfalls of HCL’s `try` Function

HCL’s try function can be problematic because it evaluates each argument until one does not produce an error. While it seems useful for navigating complex data structures, it often masks underlying issues by ignoring errors rather than addressing them.

This behavior can make debugging difficult because it can hide the real reasons behind failures, leaving configurations brittle and error-prone. Here is an example of the overuse of the try function

platform = try(each.value.platform, var.eks_managed_node_group_defaults.platform, "linux")
cluster_endpoint = try(time_sleep.this[0].triggers["cluster_endpoint"], "")
cluster_auth_base64 = try(time_sleep.this[0].triggers["cluster_certificate_authority_data"], "")
cluster_service_ipv4_cidr = var.cluster_service_ipv4_cidr
cluster_ip_family = var.cluster_ip_family
cluster_service_cidr = try(time_sleep.this[0].triggers["cluster_service_cidr"], "")
enable_bootstrap_user_data = try(each.value.enable_bootstrap_user_data, var.eks_managed_node_group_defaults.enable_bootstrap_user_data, false)
pre_bootstrap_user_data = try(each.value.pre_bootstrap_user_data, var.eks_managed_node_group_defaults.pre_bootstrap_user_data, "")
post_bootstrap_user_data = try(each.value.post_bootstrap_user_data, var.eks_managed_node_group_defaults.post_bootstrap_user_data, "")
bootstrap_extra_args = try(each.value.bootstrap_extra_args, var.eks_managed_node_group_defaults.bootstrap_extra_args, "")
user_data_template_path = try(each.value.user_data_template_path, var.eks_managed_node_group_defaults.user_data_template_path, "")
cloudinit_pre_nodeadm = try(each.value.cloudinit_pre_nodeadm, var.eks_managed_node_group_defaults.cloudinit_pre_nodeadm, [])
cloudinit_post_nodeadm = try(each.value.cloudinit_post_nodeadm, var.eks_managed_node_group_defaults.cloudinit_post_nodeadm, [])

Switching to Pulumi provides a clearer, more maintainable approach:

import * as pulumi from "@pulumi/pulumi";

// Example configuration using TypeScript for better error handling
const config = new pulumi.Config();
const platform = config.get("platform") || "linux"; // Default to "linux" if undefined

// Output using straightforward TypeScript logic
pulumi.output({ platform });

import pulumi

# Example configuration using Python
config = pulumi.Config()
platform = config.get("platform") or "linux" # Default to "linux" if undefined

# Output using straightforward Python logic
pulumi.export("platform", platform)

package main

import (
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 platform, ok := ctx.GetConfig("platform")
 if !ok {
 platform = "linux" // Default to "linux" if undefined
 }

 ctx.Export("platform", pulumi.String(platform))
 return nil
 })
}

using Pulumi;

class Program
{
 static Task<int> Main() => Pulumi.Deployment.RunAsync(() =>
 {
 var config = new Config();
 var platform = config.Get("platform") ?? "linux"; // Default to "linux" if undefined

 return new Dictionary<string, object?>
 {
 { "platform", platform }
 };
 });
}

import com.pulumi.Config;
import com.pulumi.Pulumi;

import java.util.HashMap;
import java.util.Map;

public class Main {
 public static void main(String[] args) {
 Pulumi.run(ctx -> {
 Config config = Config.auto();
 String platform = config.get("platform").orElse("linux"); // Default to "linux" if undefined

 Map<String, Object> result = new HashMap<>();
 result.put("platform", platform);

 ctx.export("result", result);
 });
 }
}

This Pulumi example shows how to handle configuration values in a more straightforward way, avoiding the need for complex try functions which potentially sweep errors under the rug. By using standard programming constructs, you can improve the readability and maintainability of your infrastructure code.

In Conclusion

In conclusion, Pulumi’s approach to infrastructure as code offers significant advantages over HCL, positioning it as a more developer-friendly and future-proof solution. By leveraging general-purpose programming languages, Pulumi addresses many of the limitations and complexities inherent in HCL.

Pulumi’s use of languages like TypeScript, Python, and Go brings several key benefits:

Larger ecosystem and community support, providing better tooling, debugging capabilities, and resources.
Familiar syntax and constructs for developers, reducing the learning curve and improving code readability.
Greater flexibility and expressiveness, allowing for more elegant solutions to complex infrastructure challenges.
Better integration with existing development workflows and practices.
Improved collaboration between development and operations teams, helping to break down silos.

The examples provided throughout this comparison demonstrate how Pulumi’s approach leads to cleaner, more intuitive code. Complex operations that require convoluted HCL expressions or multiple custom resources can often be accomplished with straightforward programming constructs in Pulumi.

Moreover, Pulumi’s model is inherently more adaptable to future changes in cloud technologies and programming paradigms. As new languages or cloud services emerge, Pulumi can easily incorporate them, ensuring that your infrastructure-as-code practices remain current and effective.

For organizations looking to modernize their infrastructure management, streamline their DevOps processes, and empower their development teams, Pulumi presents the only alternative to HCL. Its ability to unify application and infrastructure code under a single programming model offers a path to more efficient, maintainable, and scalable cloud engineering practices.

By choosing Pulumi, teams can enjoy the benefits of true programming languages for infrastructure, leading to more robust, flexible, and developer-friendly cloud deployments.

As always, we welcome your feedback and contributions in the Pulumi Community Slack, GitHub repository, and Pulumi Community Discussions.

New to Pulumi? Signing up is easy and free. Get started today!

Interested in migrating? Our cloud experts can help. Learn more about Pulumi’s Migration Services.

Pulumi Blog: Engin Diri

Incident Response as Code: Managing PagerDuty with Pulumi

What you are building

Prerequisites

The provider and the token

Teams, users, and memberships

A schedule people can live with

The escalation policy

The service

The payoff: the alarm and the pager in one program

Azure and Google Cloud, same trick

Breaking a Fargate payment service on purpose

Growing without the drift: a reusable component

Importing what you already clicked together

Conclusion

Knowledge as Code: The Memory File Just Got a Spec

The pattern everyone copied

A wiki only your agent can read

What Google actually shipped: the Open Knowledge Format

The critique is the feature

Knowledge as code

What a standard won’t do for you

Where to start

Fully Automated AI Inference on AWS, Azure, and Google Cloud with Pulumi

What you are building

Prerequisites

Credentials without the copy-paste

One cloud-init, three clouds

The programs

Deploying

Testing the inference server

Cost

Security considerations

What’s next?

Conclusion

Deploy a Private Hermes Agent on Render Securely with Pulumi, Modal, and Tailscale

What is Hermes?

Prerequisites

Understanding the Hermes architecture

Setting up ESC for secrets management

Generating the Render SDK

Securing with Tailscale

Deploying the agent

The complete program

Cost

Running the deployment

Access the agent

Verify the deployment (optional)

Security considerations

What’s next?

Conclusion

Stop Prompting. Design the Loop.

The leverage moved up a layer

The five pieces, and the one that holds them together

What makes a loop hold together

What the loop still won’t do for you

When the loop reaches production

Where to start

Five Stacks Before Lunch: The Parallel Coding Playbook for Pulumi

2x is prompting, 10x is plumbing

The five pillars

What changes when the agents are touching infrastructure

The five catches, infra edition

Where to start, this afternoon

Five stacks before lunch

Stop Tuning Prompts. Build a Harness.

How Claude Code navigates without an index

The AI layer in seven pieces

Lean and layered CLAUDE.md

Hooks that make the harness self-improving

Path-scoped skills

Symbol-level search through LSP and MCP

Subagents for exploration

Don’t let it rot

Assign an owner

Where to start

How Building AI Agents Has Changed in 2026

The old playbook

The shift: three things changed at once

When you still need a framework