Pulumi Blog: Pablo Seibelt

Build a Governed Databricks Workspace with Pulumi

Pablo Seibelt — Thu, 25 Jun 2026 00:00:00 +0000

Platform teams responsible for Databricks often find themselves manually configuring clusters and notebooks for every new data science team. This manual overhead leads to inconsistent cluster policies, runaway costs from oversized instances, and security gaps in workspace operations. Without a standardized way to provision workspace-local resources, data platforms become a fragmented collection of bespoke environments that are impossible to govern at scale.

As Databricks usage grows across the enterprise, the lack of a governed workspace baseline becomes a major operational risk. Inconsistent policies lead to unpredictable billing and audit failures. Standardizing your Databricks environment with Pulumi ensures that every workspace starts with the correct cost controls, cluster policies, notebooks, and automated jobs, allowing your data teams to move faster without compromising governance.

What you’ll build

In this post, you will learn how to provision a governed Databricks workspace baseline using Pulumi. You will build:

Cluster policies to enforce instance types and cost controls.
Workspace notebooks to standardize workload locations.
Secret scopes and permissions to make access reviewable.
Automated jobs that run on policy-constrained compute.

By the end, you will have a reproducible workspace configuration that you can deploy to any new Databricks environment.

The Databricks management boundary

When managing Databricks with Pulumi, understand the boundary between account-level and workspace-level resources.

Account-level: Creating cloud prerequisites such as VPCs, subnets, IAM roles, and Azure resource groups uses the cloud provider for your target platform. On AWS and GCP, Databricks workspace creation and account-level identity use the Databricks provider’s account APIs, such as databricks.MwsWorkspaces; on Azure, the workspace itself is commonly managed with azure-native.
Workspace-level: Managing clusters, jobs, notebooks, permissions, and secret scopes within a specific workspace. This example uses the @pulumi/databricks provider against an Azure Databricks workspace URL, and the same workspace-resource pattern applies once your provider is configured for the target workspace.

For this tutorial, we’ll assume you have an existing Databricks workspace and want to manage the resources within it.

Configuring credentials with Pulumi ESC

Before writing code, we need to configure our credentials. Pulumi ESC (Environments, Secrets, and Configuration) manages these securely.

You can create an environment that maps Databricks credentials into Pulumi configuration for the Databricks provider:

values:
 databricks:
 host: "https://adb-123456789.0.azuredatabricks.net"
 token:
 fn::secret: databricks-token
 pulumiConfig:
 databricks:host: ${databricks.host}
 databricks:token: ${databricks.token}

Once configured, import this environment into your Pulumi stack. The Databricks provider reads configuration values such as databricks:host and databricks:token without requiring static secrets in stack config.

Defining your data infrastructure

The following Pulumi program in TypeScript sets up separate cluster policies for interactive and job compute, a governed shared cluster, a secret scope, explicit permissions, a notebook, and a scheduled job that runs on policy-constrained compute.

import * as databricks from "@pulumi/databricks";

const interactivePolicy = new databricks.ClusterPolicy("interactive-fair-use", {
 name: "Interactive Fair Use Policy",
 definition: JSON.stringify({
 "dbus_per_hour": {
 "type": "range",
 "maxValue": 10,
 },
 "autotermination_minutes": {
 "type": "fixed",
 "value": 20,
 "hidden": true,
 },
 }),
});

const jobPolicy = new databricks.ClusterPolicy("job-fair-use", {
 name: "Job Fair Use Policy",
 definition: JSON.stringify({
 "dbus_per_hour": {
 "type": "range",
 "maxValue": 10,
 },
 }),
});

const sharedCluster = new databricks.Cluster("shared-cluster", {
 clusterName: "Shared Engineering Cluster",
 sparkVersion: "17.3.x-scala2.13",
 nodeTypeId: "Standard_DS3_v2",
 autoterminationMinutes: 20,
 numWorkers: 2,
 policyId: interactivePolicy.id,
});

const clusterPermissions = new databricks.Permissions("shared-cluster-permissions", {
 clusterId: sharedCluster.id,
 accessControls: [{
 groupName: "data-engineers",
 permissionLevel: "CAN_RESTART",
 }, {
 groupName: "analysts",
 permissionLevel: "CAN_ATTACH_TO",
 }],
});

const secretScope = new databricks.SecretScope("etl-secrets", {
 name: "etl-secrets",
});

const secretScopeAcl = new databricks.SecretAcl("etl-secrets-readers", {
 scope: secretScope.name,
 principal: "data-engineers",
 permission: "READ",
});

const etlNotebook = new databricks.Notebook("etl-notebook", {
 path: "/Shared/ETL/NightlyProcess",
 language: "PYTHON",
 contentBase64: Buffer.from("print('Running ETL process...')").toString("base64"),
});

const job = new databricks.Job("nightly-etl", {
 name: "Nightly ETL Job",
 tasks: [{
 taskKey: "etl-task",
 newCluster: {
 numWorkers: 2,
 sparkVersion: "17.3.x-scala2.13",
 nodeTypeId: "Standard_DS3_v2",
 policyId: jobPolicy.id,
 },
 notebookTask: {
 notebookPath: etlNotebook.path,
 },
 }],
 schedule: {
 quartzCronExpression: "0 0 2 * * ?",
 timezoneId: "UTC",
 },
});

This baseline covers the resources a platform team needs to standardize first:

Interactive cluster policy: caps DBU consumption and enforces auto-termination for shared interactive compute.
Job cluster policy: caps DBU consumption for ephemeral job compute without applying interactive auto-termination settings.
Shared cluster: gives teams a governed interactive compute target.
Secret scope and ACL: makes ETL secret access reviewable.
Cluster permissions: assigns group-level access to the shared cluster.
Notebook: stores the workload artifact in the workspace.
Job: schedules the notebook on policy-constrained compute.

Govern Databricks from day one

Define your cluster policies, notebooks, jobs, and permissions in Pulumi, then promote the same governed baseline across every workspace with stacks.

Get started

Validation

After running pulumi up, you can validate your Databricks baseline:

Databricks UI: Navigate to the “Compute” section and verify that the interactive and job fair-use policies exist, with auto-termination fixed only on the interactive policy.
Cluster check: Confirm that the shared cluster references the interactive policy and uses the expected node type.
Permission check: Confirm that cluster and secret-scope permissions match the intended Databricks groups.
Notebook check: Confirm that /Shared/ETL/NightlyProcess exists in the workspace.
Job check: Go to the “Workflows” section and confirm that the “Nightly ETL Job” is scheduled and references the notebook.

Managing your Databricks workspace as code gives every team a governed, cost-aware baseline. Use Pulumi stacks to deploy the same cluster policies, permissions, secret scope, notebook, and job pattern to multiple workspaces with environment-specific configuration.

# Deploy to development
pulumi stack select dev
pulumi up

# Promote to production
pulumi stack select prod
pulumi up

This keeps production ETL jobs, cluster policies, secret scopes, and permission assignments aligned with development while still letting each stack bind to its own Databricks workspace and ESC environment.

Conclusion

Next, connect this workspace baseline to your CI/CD pipeline and promote it through Pulumi stacks so Databricks jobs, cluster policies, and permissions move through the same review path as application code. Start with the Databricks provider docs and Pulumi ESC for credential handling.

Cloudflare-First Networking as Code with Pulumi

Pablo Seibelt — Tue, 23 Jun 2026 00:00:00 +0000

Platform teams managing multi-cloud applications face a dangerous visibility gap. While origin infrastructure is tightly controlled, the edge configuration often drifts through manual console tweaks. DNS records point to stale origins, WAF rules are inconsistent across environments, and Zero Trust policies fail to keep pace with team changes. This edge drift leads to application exposure or routing failures that origin teams only notice after users report them.

As applications span multiple clouds, the edge often becomes the most consistent layer for enforcing security and traffic policies. The cost of waiting for a unified edge strategy is high. Every manual change is a potential security hole or a performance bottleneck that bypasses your standard CI/CD rigor.

What you’ll build

In this post, you will build a standardized Cloudflare edge baseline. You’ll use Pulumi to define:

DNS records to point to your multi-cloud origins.
WAF custom rules to block malicious traffic before it hits your network.
A Worker canary to handle edge logic and header validation.
Zero Trust Access policies to secure internal tools and origin access.

By the end, you will have a version-controlled edge that you can validate and deploy across any environment.

The Pulumi program

We’ll build a Pulumi program that sets up a complete edge environment. This approach ensures that your edge security and routing are always in sync with your application code.

Setting up the provider

First, ensure you have the @pulumi/cloudflare package installed and your credentials configured. The examples below are checked against @pulumi/cloudflare@6.15.0, the current v6 Pulumi Registry API at the time of writing.

npm install @pulumi/cloudflare@6.15.0
pulumi config set --secret cloudflare:apiToken "$CLOUDFLARE_API_TOKEN"

Defining the infrastructure

Here is the complete Pulumi program in TypeScript. If your zone already has a custom WAF ruleset for the same phase, import that ruleset or add the rule to your existing ruleset instead of creating another phase-level ruleset that may fail or conflict.

import * as cloudflare from "@pulumi/cloudflare";

const accountId = "<your-cloudflare-account-id>";
const trustedAdminIp = "198.51.100.10";

const zone = new cloudflare.Zone("my-zone", {
 account: {
 id: accountId,
 },
 name: "example.com",
});

const www = new cloudflare.DnsRecord("www", {
 zoneId: zone.id,
 name: "www",
 content: "1.2.3.4",
 ttl: 1,
 type: "A",
 proxied: true,
});

const admin = new cloudflare.DnsRecord("admin", {
 zoneId: zone.id,
 name: "admin",
 content: "1.2.3.4",
 ttl: 1,
 type: "A",
 proxied: true,
});

const wafRule = new cloudflare.Ruleset("waf-rule", {
 zoneId: zone.id,
 name: "Block suspicious traffic",
 description: "Custom WAF rule to block specific patterns",
 kind: "zone",
 phase: "http_request_firewall_custom",
 rules: [{
 action: "block",
 expression: `(http.request.uri.path contains "/admin" and not ip.src in {${trustedAdminIp}})`,
 description: "Block unauthorized admin access",
 }],
});

const worker = new cloudflare.WorkersScript("canary-worker", {
 accountId: accountId,
 scriptName: "edge-canary",
 compatibilityDate: "2026-01-01",
 mainModule: "worker.js",
 content: `
 export default {
 async fetch(request) {
 return new Response("Edge Baseline Validated", {
 headers: { "x-edge-baseline": "active" },
 });
 },
 };
 `,
});

const workerRoute = new cloudflare.WorkersRoute("canary-route", {
 zoneId: zone.id,
 pattern: "www.example.com/*",
 script: worker.scriptName,
});

const adminGroup = new cloudflare.ZeroTrustAccessGroup("admin-group", {
 accountId: accountId,
 name: "Administrators",
 includes: [{
 email: {
 email: "admin@example.com",
 },
 }],
});

const accessApp = new cloudflare.ZeroTrustAccessApplication("internal-tool", {
 accountId: accountId,
 name: "Internal Admin Tool",
 domain: "admin.example.com",
 type: "self_hosted",
 policies: [{
 name: "Allow administrators",
 precedence: 1,
 decision: "allow",
 includes: [{
 group: {
 id: adminGroup.id,
 },
 }],
 }],
});

Validation

Once you run pulumi up, you can validate your edge baseline with these steps:

DNS check: Run dig www.example.com to confirm the record points to Cloudflare’s proxied IPs.
WAF test: Attempt to access www.example.com/admin from an unauthorized IP and verify you receive a 403 Forbidden response.
Worker canary: Use curl -I https://www.example.com and check for the x-edge-baseline: active header.
Access policy: Navigate to admin.example.com and verify you are redirected to the Cloudflare Access login page.

By managing these resources as code, you eliminate the risk of manual drift and ensure that every environment starts with a secure, validated edge baseline.

Use Your Mac for AI Agents: Self-Host Gemma 4 12 B with Pulumi and Tailscale

Pablo Seibelt — Thu, 04 Jun 2026 00:00:00 +0000

If you run AI tools and agents, you’ve probably accepted three tradeoffs: your data leaves your network, you can’t work offline, and your bill scales with usage.

Open-weight models now run well on consumer hardware. Once the model is on your machine, your data stays local, inference works offline, and tokens cost nothing. If you own a modern Mac, you can run a high-quality model yourself.

Gemma 4 is an open-weights model family from Google. This post focuses on Gemma 4 12 B, released in June 2026, using Unsloth’s Q8_0 GGUF. The 12 B model fits comfortably on a modern Mac while leaving enough headroom for local llama.cpp and a chat UI.

We’ll use llama.cpp for host-native inference, k3d for a local Kubernetes cluster, Pulumi for infrastructure as code, and Tailscale for secure access.

Prerequisites

This setup was validated on the following hardware:

macOS 26 Tahoe, version 26.5
MacBook Pro with Apple M3 Max
36 GB RAM

On this machine, llama.cpp reported about 20 output tokens per second for a 160-token validation response with unsloth/gemma-4-12b-it-GGUF, gemma-4-12b-it-Q8_0.gguf, and a 131,072-token context. Sustained throughput varies by prompt length, thermal state, and llama.cpp settings.

You’ll need brew, docker, pulumi, and tailscale installed. We’ll also install k3d during the process.

Run Gemma 4 with host-native llama.cpp

We use llama.cpp directly on macOS to leverage Apple Metal acceleration. Running the LLM on the host is more efficient than trying to pass GPU access into a local Kubernetes VM.

Install the build tools:

brew install cmake git

Then build llama.cpp from source and download the multimodal projector. In validation, Homebrew llama.cpp 9430 could run text inference, but it could not load the new Gemma 4 12 B projector and failed with unknown projector type: gemma4uv. Building current llama.cpp from source fixed that.

llm_home="$HOME/pulumi-gemma4-llm"
mkdir -p "$llm_home/models" "$llm_home/logs"

if [ ! -d "$llm_home/llama.cpp/.git" ]; then
 git clone --depth 1 https://github.com/ggml-org/llama.cpp.git "$llm_home/llama.cpp"
fi

cmake -S "$llm_home/llama.cpp" \
 -B "$llm_home/llama.cpp/build" \
 -DGGML_METAL=ON \
 -DGGML_BLAS=ON \
 -DCMAKE_BUILD_TYPE=Release

cmake --build "$llm_home/llama.cpp/build" --target llama-server -j 10

curl -L --fail \
 --output "$llm_home/models/mmproj-F16.gguf" \
 https://huggingface.co/unsloth/gemma-4-12b-it-GGUF/resolve/main/mmproj-F16.gguf

Then download and run the model with this command:

"$HOME/pulumi-gemma4-llm/llama.cpp/build/bin/llama-server" \
 --hf-repo unsloth/gemma-4-12b-it-GGUF \
 --hf-file gemma-4-12b-it-Q8_0.gguf \
 --mmproj "$HOME/pulumi-gemma4-llm/models/mmproj-F16.gguf" \
 --host 127.0.0.1 \
 --port 18080 \
 --ctx-size 131072 \
 --parallel 1 \
 --jinja \
 --reasoning off

We use port 18080 because 8080 is commonly used and is likely to conflict with another service you may already have running locally. If your port 8080 is free, you can use it and adjust the Pulumi config later.

The model file is about 12.65 GB, and the projector is about 116 MB. Gemma 4 12 B advertises a 131,072-token context, and this Mac loaded that full context with --parallel 1. llama.cpp projected about 15.1 GiB of Apple Metal device memory for the text model and about 258 MiB worst-case memory for the projector, leaving enough headroom for Open WebUI and the rest of the local stack. The --reasoning off flag keeps OpenAI-compatible chat responses visible in clients that do not read separate reasoning fields.

With --mmproj, /v1/models advertised capabilities: ["completion","multimodal"]. In local validation, Open WebUI accepted an uploaded Pulumi logo image and Gemma 4 described it correctly. A small WAV file also worked through the OpenAI-compatible input_audio request shape, though llama.cpp logs still mark audio input as experimental.

Verify the LLM API

Open a new terminal and check if llama.cpp is responding:

curl http://127.0.0.1:18080/v1/models

The /v1/models endpoint should return the model ID unsloth/gemma-4-12b-it-GGUF. Now try a chat completion:

curl http://127.0.0.1:18080/v1/chat/completions \
 -H "Content-Type: application/json" \
 -d '{
 "model": "unsloth/gemma-4-12b-it-GGUF",
 "messages": [{"role": "user", "content": "Reply with exactly: OK"}],
 "temperature": 0,
 "max_tokens": 32
 }'

The chat prompt Reply with exactly: OK should return content OK. In validation, llama.cpp reported an output token velocity of about 20 tokens per second for a longer 160-token response.

Keep llama.cpp running after reboot

For a permanent setup, put the llama.cpp startup script and logs under a folder in your home directory and let launchd restart it when you sign in:

llm_home="$HOME/pulumi-gemma4-llm"
mkdir -p "$llm_home/logs" "$HOME/Library/LaunchAgents"

cat > "$llm_home/start-llama-server.sh" <<'EOF'
#!/bin/zsh
set -euo pipefail

export PATH="/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin"

exec "$HOME/pulumi-gemma4-llm/llama.cpp/build/bin/llama-server" \
 --hf-repo unsloth/gemma-4-12b-it-GGUF \
 --hf-file gemma-4-12b-it-Q8_0.gguf \
 --mmproj "$HOME/pulumi-gemma4-llm/models/mmproj-F16.gguf" \
 --host 127.0.0.1 \
 --port 18080 \
 --ctx-size 131072 \
 --parallel 1 \
 --jinja \
 --reasoning off
EOF

chmod +x "$llm_home/start-llama-server.sh"

cat > "$HOME/Library/LaunchAgents/com.pulumi.gemma4.llama-server.plist" <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
 <key>Label</key>
 <string>com.pulumi.gemma4.llama-server</string>
 <key>ProgramArguments</key>
 <array>
 <string>$llm_home/start-llama-server.sh</string>
 </array>
 <key>WorkingDirectory</key>
 <string>$llm_home</string>
 <key>RunAtLoad</key>
 <true/>
 <key>KeepAlive</key>
 <true/>
 <key>StandardOutPath</key>
 <string>$llm_home/logs/llama-server.out.log</string>
 <key>StandardErrorPath</key>
 <string>$llm_home/logs/llama-server.err.log</string>
</dict>
</plist>
EOF

launchctl bootout gui/$(id -u)/com.pulumi.gemma4.llama-server 2>/dev/null || true
launchctl bootstrap gui/$(id -u) "$HOME/Library/LaunchAgents/com.pulumi.gemma4.llama-server.plist"
launchctl kickstart -k gui/$(id -u)/com.pulumi.gemma4.llama-server

Check the launchd service and llama.cpp logs:

launchctl print gui/$(id -u)/com.pulumi.gemma4.llama-server
tail -f "$HOME/pulumi-gemma4-llm/logs/llama-server.err.log"

If you want to stop llama.cpp later, unload the launchd service:

launchctl bootout gui/$(id -u)/com.pulumi.gemma4.llama-server

Deploy Open WebUI with Pulumi and k3d

Now we’ll deploy Open WebUI into a local Kubernetes cluster. This provides a polished chat interface that connects to our host-native LLM.

First, install k3d if you haven’t already:

brew install k3d

Create a new cluster for this project:

k3d cluster create pulumi-gemma4-blog-qa

We’ll use the Pulumi program in pulumi/examples. This program defaults to runtimeMode=host, which creates a Kubernetes ExternalName service pointing to your host machine.

Why not run the LLM inside Kubernetes on this Mac? Pulumi can do that, and the example supports it with runtimeMode=cluster, but that path is meant for Linux hosts with NVIDIA or AMD GPU device plugins.

On macOS, llama.cpp enables Metal by default, and Metal acceleration is available to native macOS processes. k3d runs Linux containers through Docker Desktop, so those pods do not get direct access to the Mac’s Metal device. Docker’s own vLLM Metal announcement calls out the same boundary: Metal-backed inference runs natively on the host because there is no Metal GPU passthrough for containers. That is why this setup keeps inference host-native and lets Pulumi manage the Kubernetes UI, service wiring, and optional Tailscale access around it.

Clone the examples repo, navigate to the program directory, and initialize a new stack:

git clone https://github.com/pulumi/examples.git
cd examples/kubernetes-py-self-host-gemma4-llm
pulumi stack init gemma4-local

Configure the program to match your local setup:

pulumi config set hostLlmPort 18080
pulumi config set llmBaseUrl http://llm-server:18080/v1

The Kubernetes service named llm-server maps to host.k3d.internal. In our validation, we confirmed that a disposable k3d pod could reach the Mac’s llama.cpp API at http://llm-server:18080/v1/models after a CoreDNS restart.

kubectl rollout restart deployment coredns -n kube-system

Run pulumi up to deploy Open WebUI and connect it to host-native llama.cpp:

pulumi up

In our validation environment, this command successfully reached the resource synthesis phase without requiring Tailscale credentials because Tailscale exposure is opt-in.

Access Open WebUI through Tailscale

Tailscale allows you to access your private Open WebUI instance from any device on your tailnet. Note that we only expose the web interface, not the raw LLM API, to keep the system secure.

The base Open WebUI deployment works without Tailscale credentials. To expose the web UI on your tailnet, enable Tailscale resources and provide an explicit api_key or OAuth/identity token:

pulumi config set enableTailscale true
pulumi config set tailscale:apiKey YOUR_API_KEY --secret

Once configured, Pulumi will create a Tailscale device or proxy that routes traffic to your Open WebUI service.

Use the model with Pi

Open WebUI gives you a browser-based chat interface, but local models are also useful from coding agents. Pi is the local coding agent used for this validation; if you do not use Pi, treat this section as an example of how any OpenAI-compatible client can point at the same local endpoint. Pi can point at the same OpenAI-compatible llama.cpp endpoint and use the model running on your Mac.

For a fresh Pi config, create ~/.pi/agent/models.json with a local provider that points at the llama.cpp API:

{
 "providers": {
 "local-llama": {
 "baseUrl": "http://127.0.0.1:18080/v1",
 "api": "openai-completions",
 "apiKey": "local",
 "compat": {
 "supportsDeveloperRole": false,
 "supportsReasoningEffort": false
 },
 "models": [
 {
 "id": "unsloth/gemma-4-12b-it-GGUF",
 "name": "Gemma 4 12B Q8 (local llama.cpp)",
 "reasoning": false,
 "input": ["text"],
 "contextWindow": 131072,
 "maxTokens": 1024,
 "cost": {
 "input": 0,
 "output": 0,
 "cacheRead": 0,
 "cacheWrite": 0
 }
 }
 ]
 }
 }
}

Then set Pi to use that provider and model by default in ~/.pi/agent/settings.json:

{
 "defaultProvider": "local-llama",
 "defaultModel": "unsloth/gemma-4-12b-it-GGUF",
 "defaultThinkingLevel": "off",
 "hideThinkingBlock": true
}

If you already have Pi configuration files, merge the local-llama provider and defaults into your existing JSON instead of replacing the files.

Manage your local AI stack as code

Let Pulumi provision the Kubernetes cluster, service wiring, and Tailscale access around your self-hosted model, so you can rebuild the whole setup with pulumi up.

Get started

Advanced: Linux GPU in-cluster serving

If you’re running on a Linux host with an NVIDIA or AMD GPU, you can run the LLM directly inside the Kubernetes cluster. This requires the NVIDIA or ROCm device plugins.

The Pulumi program supports this through runtimeMode=cluster. In this mode, it deploys a LlmServer pod that manages the llama.cpp process within the cluster, using GPU resource requests to ensure hardware acceleration.

Cleanup

When you’re done, you can tear down the resources:

pulumi destroy
k3d cluster delete pulumi-gemma4-blog-qa
# Stop llama.cpp using the PID from your terminal
kill <PID>

Build an EKS Environment Factory with Pulumi and vCluster

Pablo Seibelt — Thu, 04 Jun 2026 00:00:00 +0000

AWS reports in an AWS Architecture Blog case study that Deloitte’s move to a virtual cluster model on Amazon EKS resulted in 89% faster testing environment provisioning. By consolidating dozens of disparate clusters into a single host cluster with over 50 vCluster instances, the case study says Deloitte saved about 500 QA hours per year. This “Environment Factory” pattern allows platform teams to provide isolated, ephemeral Kubernetes environments on demand without the cost or lag of full cluster provisioning.

This post adapts that general architecture with Pulumi to orchestrate Amazon EKS Auto Mode and vCluster.

The problem: environment sprawl and provisioning lag

Traditional development workflows often rely on one full EKS cluster per developer or feature branch. While this provides strong isolation, it introduces major pain points. Provisioning a full cluster can take 15 minutes or more, which slows down CI/CD pipelines. Managing dozens of clusters also leads to high costs and significant operational overhead.

Platform teams need a “soft multi-tenancy” model. This model should feel like a dedicated cluster to the developer but run on shared infrastructure to keep costs low and startup times fast.

Architecture overview: the host and the tenants

The environment factory architecture consists of two main layers.

Host cluster: A single, reliable EKS cluster managed with EKS Auto Mode. This cluster provides the underlying compute, networking, and storage.
Tenant environments: Virtual clusters (vCluster) running as pods within host namespaces.

According to the vCluster architecture, the virtual control plane handles API requests while a syncer maps virtual resources to the host cluster. This separation allows tenants to manage their own CRDs, namespaces, and RBAC while platform teams use quotas, NetworkPolicies, pod security, IAM boundaries, and node isolation controls to protect the host and other tenants.

Implementation: the EKS Auto Mode host

EKS Auto Mode simplifies the host cluster by automating infrastructure management. It handles node provisioning, scaling, and updates based on pod requirements.

The following snippet shows how to define an EKS cluster with Auto Mode enabled using Pulumi.

import * as awsx from "@pulumi/awsx";
import * as eks from "@pulumi/eks";
import * as k8s from "@pulumi/kubernetes";
import { SubnetType } from "@pulumi/awsx/ec2";

const clusterName = "environment-factory";

const vpc = new awsx.ec2.Vpc("environment-factory", {
 enableDnsHostnames: true,
 cidrBlock: "10.0.0.0/16",
 subnetSpecs: [
 {
 type: SubnetType.Public,
 tags: {
 [`kubernetes.io/cluster/${clusterName}`]: "shared",
 "kubernetes.io/role/elb": "1",
 },
 },
 {
 type: SubnetType.Private,
 tags: {
 [`kubernetes.io/cluster/${clusterName}`]: "shared",
 "kubernetes.io/role/internal-elb": "1",
 },
 },
 ],
 subnetStrategy: "Auto",
});

// Create an EKS cluster with Auto Mode enabled.
const hostCluster = new eks.Cluster("host-cluster", {
 name: clusterName,
 authenticationMode: eks.AuthenticationMode.Api, // Use API authentication mode for EKS access entries.
 vpcId: vpc.vpcId,
 publicSubnetIds: vpc.publicSubnetIds,
 privateSubnetIds: vpc.privateSubnetIds,
 autoMode: {
 enabled: true,
 },
});

const hostProvider = new k8s.Provider("host-provider", {
 kubeconfig: hostCluster.kubeconfig,
});

Implementation: the environment factory

Once the host cluster is ready, we can build the factory that stamps out tenant environments. Each tenant needs a dedicated namespace, resource quotas, and the vCluster itself.

Tenant guardrails

Before installing vCluster, we set up a namespace and resource quotas to ensure one tenant cannot consume all host resources.

import * as k8s from "@pulumi/kubernetes";

// Define a tenant namespace on the host cluster.
const tenantNamespace = new k8s.core.v1.Namespace("tenant-alpha", {
 metadata: { name: "tenant-alpha" },
}, { provider: hostProvider });

// Apply resource quotas to the tenant namespace.
const quota = new k8s.core.v1.ResourceQuota("tenant-quota", {
 metadata: { namespace: tenantNamespace.metadata.name },
 spec: {
 hard: {
 pods: "20",
 "requests.cpu": "4",
 "requests.memory": "8Gi",
 "limits.cpu": "8",
 "limits.memory": "16Gi",
 },
 },
}, { provider: hostProvider });

// Define a Role for the tenant within their namespace.
const tenantRole = new k8s.rbac.v1.Role("tenant-role", {
 metadata: { namespace: tenantNamespace.metadata.name },
 rules: [{
 apiGroups: [""],
 resources: ["pods", "services", "configmaps", "secrets"],
 verbs: ["get", "list", "watch", "create", "update", "patch", "delete"],
 }],
}, { provider: hostProvider });

// Bind the Role to a tenant user or group.
const tenantRoleBinding = new k8s.rbac.v1.RoleBinding("tenant-role-binding", {
 metadata: { namespace: tenantNamespace.metadata.name },
 subjects: [{
 kind: "User",
 // Replace "tenant-user" with the IAM-mapped user or group for this tenant.
 name: "tenant-user",
 apiGroup: "rbac.authorization.k8s.io",
 }],
 roleRef: {
 kind: "Role",
 name: tenantRole.metadata.name,
 apiGroup: "rbac.authorization.k8s.io",
 },
}, { provider: hostProvider });

For production use, map these Kubernetes identities to IAM principals using EKS Access Entries, with the legacy aws-auth ConfigMap still appearing in older clusters.

Deploying vCluster with Helm

We use the kubernetes.helm.v3.Release resource to install vCluster. This resource provides controlled Helm lifecycle management for the vCluster release. The values block should be adjusted for each tenant profile to control resource synchronization and control plane behavior. Review the vCluster release notes when changing chart versions because values schema and generated secret names can change across releases.

import * as k8s from "@pulumi/kubernetes";

// Install vCluster using the Helm Release resource.
const vcluster = new k8s.helm.v3.Release("vcluster-alpha", {
 name: "vcluster-alpha",
 chart: "vcluster",
 version: "0.20.0", // Tested with vCluster 0.20.x; review release notes before changing versions.
 repositoryOpts: {
 repo: "https://charts.loft.sh",
 },
 namespace: tenantNamespace.metadata.name,
 values: {
 // Explicit sync configuration; adjust per tenant profile.
 sync: {
 toHost: {
 pods: { enabled: true },
 },
 },
 },
}, { provider: hostProvider });

Accessing the virtual cluster

The vCluster generates a kubeconfig that allows developers to interact with the virtual API server. We must treat this kubeconfig as a secret, and the endpoint in that kubeconfig must be reachable from the Pulumi runner before using it to create resources inside the virtual cluster.

import * as pulumi from "@pulumi/pulumi";
import * as k8s from "@pulumi/kubernetes";

// Retrieve the vCluster kubeconfig from the generated secret.
// The vCluster-generated secret can lag behind Helm release readiness on first creation,
// so teams may choose an explicit readiness check or rerun after the virtual control plane initializes.
const vclusterKubeconfig = k8s.core.v1.Secret.get("vcluster-secret",
 pulumi.interpolate`${tenantNamespace.metadata.name}/vc-vcluster-alpha`,
 {
 provider: hostProvider,
 dependsOn: [vcluster],
 }
).data.apply(data => Buffer.from(data["config"], "base64").toString());

// Export the kubeconfig as a secret.
export const tenantKubeconfig = pulumi.secret(vclusterKubeconfig);

// Create a provider for the virtual cluster using the secret kubeconfig.
const vclusterProvider = new k8s.Provider("vcluster-provider", {
 kubeconfig: tenantKubeconfig,
});

Provision environments on demand

Model your host cluster, vCluster tenants, and guardrails in Pulumi so your platform team can hand developers isolated test environments and clean them up as code.

Get started

Operational caveats

RBAC and permissions: vCluster generates default RBAC rules that work for most scenarios. However, if your host cluster is heavily locked down, you may need to provide additional permissions to the vCluster service account.
Helm release previews: When using kubernetes.helm.v3.Release, Pulumi previews may not show every detail of the rendered Kubernetes resources. It primarily tracks the state of the Helm release itself.
EKS Auto Mode node lifetime: EKS Auto Mode uses immutable AMIs and has a 21-day node lifetime. Kubernetes reschedules vCluster pods and tenant workloads when nodes are replaced, so configure replicas, PodDisruptionBudgets, requests, and persistent storage for disruption tolerance.

Conclusion: ephemeral environments at scale

By combining Pulumi with EKS Auto Mode and vCluster, you can build a scalable environment factory. This approach provides the isolation developers need while maintaining the speed and cost-efficiency required by platform teams.

The snippets provided here are adapted for illustration. In a production environment, you would likely wrap these resources into a Pulumi ComponentResource to provide a clean, reusable API for your internal developers. When a feature branch is merged, deleting the Pulumi stack removes the resources managed by that stack, but validate namespace finalizers, persistent volume reclaim policies, and external cloud artifacts as part of cleanup.

For more on managing EKS with Pulumi, see the EKS guide.

Why Choose Pulumi Over Terraform?

Pablo Seibelt — Tue, 02 Jun 2026 00:00:00 +0000

Terraform is a proven infrastructure as code tool with a large provider and module ecosystem. Many teams choose Pulumi when they want to keep that infrastructure as code model, but write and maintain infrastructure with general-purpose programming languages, familiar package managers, IDEs, testing, and software engineering patterns, while still understanding the refactoring tradeoffs in Terraform’s own module refactoring guidance.

Why choose Pulumi over Terraform? Pulumi’s language SDKs let teams define cloud infrastructure in TypeScript, Python, Go, C#, Java, or YAML while adding first-class workflows for refactoring with Pulumi aliases, secrets, protect, retainOnDelete, deleteBeforeReplace, replaceOnChanges, provider resources, Pulumi stacks, testing, and incremental migration with pulumi import. Pulumi does not remove every hard problem in cloud infrastructure, but it gives teams stronger tools for many day-to-day pain points.

Executive summary

Pulumi is often a better fit when infrastructure code needs to behave like application code: reviewed, tested, packaged, refactored, and shared across teams. The biggest advantages show up when teams need safer refactors, encrypted secret values, reusable components, clearer provider resources, and guardrails around destructive changes.

The tradeoff is important: Pulumi is still an infrastructure as code engine. Provider bugs, cloud API eventual consistency, drift, preview-time unknowns, and poorly designed abstractions still require engineering discipline. The advantage is not magic. The advantage is a stronger programming model and a more familiar developer workflow.

Need	Terraform pattern	Pulumi advantage	What still needs care
Languages and tooling	HCL plus Terraform-specific functions and expressions	Pulumi supports general-purpose languages, YAML, and now HCL natively, with the normal IDE, test, and package workflows for each	Teams still need code review and shared conventions
Refactoring	Moved blocks or state commands for resource identity changes	Pulumi aliases can map old resource identities to new ones during refactors	Aliases must model the old identity correctly
Secrets	Sensitive values can still require careful state and plan handling	Pulumi tracks secrets and encrypts secret values in state	Secrets are still available to code at runtime
Lifecycle safety	Lifecycle meta-arguments and plan review	Pulumi resource options such as protect, retainOnDelete, deleteBeforeReplace, and replaceOnChanges make intent explicit	Provider behavior and replacement semantics still matter
Environments	Workspaces or separate configurations	Pulumi stacks model environments with per-stack config, secrets, history, and outputs	Stack boundaries still need thoughtful design
Code reuse	Modules and HCL composition patterns	Pulumi components and packages use normal language abstractions	Over-abstracted components can become hard to use
Imports and migration	Import blocks, generated config, and state operations	`pulumi import` and migration tooling support gradual adoption	Imported code still needs review and cleanup
Terraform state	S3, Azure Blob, or Terraform Cloud backends	Pulumi Cloud can serve as a Terraform state backend with encrypted storage, locking, history, and RBAC, while you keep using the Terraform or OpenTofu CLI	State still needs a clean migration plan as resources move to Pulumi
Provider wiring	Provider inheritance and aliases inside modules	Explicit provider resources make multi-region and multi-account wiring visible in code review	Provider versions and bugs can still affect deployments
Testing	Validation, plan review, and external test harnesses	Pulumi programs can use normal unit and integration test frameworks	Tests complement previews, they do not replace them
Drift detection	Refresh and refresh-only plans, plus external scheduling	`pulumi refresh` reconciles state, and Pulumi Cloud adds scheduled drift detection and remediation on a configurable cadence	Detection still depends on provider behavior and a healthy state backend
Caveats	Declarative planning still has unknowns and drift	Pulumi improves the workflow around many pain points	It does not eliminate provider bugs or eventual consistency

Use programming languages and familiar tools

Terraform modules are powerful, but larger HCL codebases can require teams to maintain separate conventions for composition, validation, and reuse. Pulumi lets infrastructure teams use the features of whichever supported programming language they choose, such as classes, functions, types, loops, package managers, linters, and test frameworks.

Teams with HCL muscle memory are not left out, either. Pulumi now supports HCL natively as one of its languages, alongside Python, TypeScript, Go, C#, Java, and YAML. Part of a team can keep authoring in HCL while the project still benefits from Pulumi components, testing, and the same engine and package workflows as every other language.

For example, a platform team can wrap a standard storage pattern in a ComponentResource and share it like any other TypeScript abstraction:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

interface WorkQueueArgs {
 visibilityTimeoutSeconds: number;
}

class WorkQueue extends pulumi.ComponentResource {
 public readonly queueUrl: pulumi.Output<string>;

 constructor(name: string, args: WorkQueueArgs, opts?: pulumi.ComponentResourceOptions) {
 super("platform:queue:WorkQueue", name, {}, opts);

 const queue = new aws.sqs.Queue(`${name}-queue`, {
 visibilityTimeoutSeconds: args.visibilityTimeoutSeconds,
 }, { parent: this });

 this.queueUrl = queue.url;
 this.registerOutputs({ queueUrl: this.queueUrl });
 }
}

const jobs = new WorkQueue("image-jobs", {
 visibilityTimeoutSeconds: 60,
});

Refactor infrastructure without surprise replacement

Renaming a resource, moving it into a component, or reorganizing a project should not automatically mean replacing production infrastructure. Pulumi aliases let you tell Pulumi how a resource used to be addressed, so refactors can preserve resource identity when the old and new resources represent the same cloud object.

const queue = new aws.sqs.Queue("app-jobs", {
 visibilityTimeoutSeconds: 60,
}, {
 aliases: [{ name: "jobs" }],
});

Aliases are not a substitute for review. They work when the old identity is modeled correctly, including details such as name, parent, type, project, and stack when those changed.

Handle secrets with encrypted configuration and secret outputs

Secrets are one of the most common places where infrastructure workflows become risky. Terraform has sensitive values and backend guidance, but teams still need to understand how plans, state, outputs, and backend access interact. Pulumi treats secrets as first-class values, encrypts them in state, and preserves secrecy as values flow through outputs.

const config = new pulumi.Config();
const dbPassword = config.requireSecret("dbPassword");

const passwordParameter = new aws.ssm.Parameter("db-password", {
 type: "SecureString",
 value: dbPassword,
});

This improves the default experience, but it’s not runtime isolation. In the TypeScript SDK, config.requireSecret("dbPassword") retrieves secret configuration, and your program can still access the decrypted value while it runs, so reviews, least privilege, and secret-provider choices still matter. If a value starts as a plain input instead of coming from requireSecret, pulumi.secret(...) can mark it as secret.

For secrets that shouldn’t live in stack config at all, Pulumi ESC (Environments, Secrets, and Configuration) centralizes them in environments that your stacks consume through config. ESC can pull secrets dynamically from external stores such as HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, and 1Password, and mint short-lived OIDC credentials for AWS, Azure, and GCP. That lets you wire an environment into a Pulumi program instead of storing long-lived credentials in state.

Add safer lifecycle controls for destructive changes

Infrastructure mistakes are expensive when they replace or delete stateful resources. Pulumi gives teams explicit resource options for safety-sensitive intent, such as protect, retainOnDelete, deleteBeforeReplace, and replaceOnChanges.

const database = new aws.rds.Instance("orders-db", {
 instanceClass: "db.t4g.micro",
 allocatedStorage: 20,
 engine: "postgres",
 username: "app",
 password: dbPassword,
}, {
 protect: true,
 retainOnDelete: true,
});

These controls are guardrails, not guarantees. Provider bugs, cloud API eventual consistency, and replacement behavior still require preview review and operational judgment. replaceOnChanges applies to custom resources only, not component resources.

Model environments with stacks

Terraform workspaces can represent environments, but many teams eventually need stronger boundaries for configuration, secrets, history, and cross-environment outputs. Pulumi stacks make environment boundaries explicit and pair them with per-stack config and outputs.

const networking = new pulumi.StackReference("acme/networking/prod");
const vpcId = networking.requireOutput("vpcId");

const appSecurityGroup = new aws.ec2.SecurityGroup("app-sg", {
 vpcId,
 ingress: [{
 protocol: "tcp",
 fromPort: 443,
 toPort: 443,
 cidrBlocks: ["10.0.0.0/8"],
 }],
 egress: [{
 protocol: "-1",
 fromPort: 0,
 toPort: 0,
 cidrBlocks: ["0.0.0.0/0"],
 }],
});

Cross-stack references are cleaner than sharing an entire remote state file, but they are still dependencies. Keep stack outputs intentional and avoid turning one stack into a global variable bag.

Test infrastructure with normal language tooling

Because Pulumi programs are real programs, infrastructure testing can use familiar test runners and assertions. Teams can test component defaults, naming conventions, required tags, and policy assumptions before a deployment reaches production.

pulumi.runtime.setMocks({
 newResource: (args) => ({
 id: `${args.name}_id`,
 state: args.inputs,
 }),
 call: (args) => args.inputs,
});

Testing does not replace previews. It catches a different class of problems: broken abstractions, missing required inputs, invalid defaults, and regressions in shared components.

Wire providers explicitly

Provider configuration is another place where explicit code can reduce ambiguity. In Terraform, provider inheritance and aliases are often managed across module boundaries. In Pulumi, provider resources are normal resources that can be passed through resource options, which makes multi-region or multi-account deployments easier to follow in code review.

const west = new aws.Provider("west", {
 region: "us-west-2",
});

const east = new aws.Provider("east", {
 region: "us-east-1",
});

const primary = new aws.sqs.Queue("primary-jobs", {}, {
 provider: west,
});

const replica = new aws.sqs.Queue("replica-jobs", {}, {
 provider: east,
});

The provider object does not remove provider versioning or schema-change risk, but it makes provider wiring visible in the same language and review flow as the rest of the program.

Detect and reconcile drift

Infrastructure drifts when something changes outside your IaC tool: a hotfix in the console, another controller, or a manual break-glass change. Pulumi gives you first-class tools to find and fix it instead of leaving it to chance. pulumi refresh reconciles your state with what is actually running in the cloud, and pulumi preview --diff shows what the next update would change.

For teams that want this continuously, Pulumi Cloud adds scheduled drift detection and remediation that runs on a configurable cadence and can automatically remediate drift when it is detected. That turns finding and correcting out-of-band changes into a managed workflow rather than something a person has to remember to check.

Drift detection still depends on accurate provider behavior and a healthy state backend, so it complements review and operational discipline rather than replacing them.

Adopt Pulumi at your own pace

Bring your existing Terraform infrastructure across one stack at a time, using general-purpose languages and the software workflows your team already relies on.

Get started

Import and migrate incrementally

Teams rarely get to rebuild infrastructure from scratch. Pulumi supports incremental adoption with pulumi import, generated code, and Terraform interoperability paths. That makes it possible to start with one resource, one component, or one stack instead of forcing a big-bang migration.

Pulumi also supports interoperability paths for teams that need to bring existing Terraform assets forward over time, including Terraform provider access and migration workflows. That makes migration an engineering sequence, not an all-or-nothing rewrite.

const importedRepository = new aws.ecr.Repository("app", {
 name: "existing-app-repo",
}, {
 import: "existing-app-repo",
});

The CLI flow can also generate declarations with pulumi import. Generated code is a starting point, not a finished architecture. Review names, options, providers, secrets, and component boundaries before treating imported resources as production-ready Pulumi code.

You also don’t have to do it alone. Pulumi provides self-serve conversion tools and expert migration services, and for teams leaving HashiCorp, credits you can apply toward Pulumi to avoid paying for two tools at once, so budget and timing don’t have to block getting started.

Manage Terraform state in Pulumi Cloud

You don’t have to rewrite anything to start getting value from Pulumi. Pulumi Cloud can serve as a Terraform state backend, letting you store and manage Terraform state alongside your Pulumi stacks. Your team can keep using the Terraform or OpenTofu CLI for day-to-day operations while gaining encrypted state storage, update history, state locking, role-based access control, audit policies, and unified resource visibility through Pulumi Insights — with agentic infrastructure coding through Neo on top.

It uses Terraform’s standard remote backend, so you point the CLI at Pulumi Cloud without changing your infrastructure code:

terraform {
 backend "remote" {
 hostname = "api.pulumi.com"
 organization = "your-pulumi-org"

 workspaces {
 name = "_dev"
 }
 }
}

From there, Pulumi can also read outputs from that state, so a new Pulumi stack can consume Terraform-managed values such as VPC IDs and subnet IDs while you migrate one piece at a time. When you’re ready to bring resources fully across, pulumi import --from terraform reads a .tfstate file and generates matching declarations, and pulumi convert --from terraform translates the surrounding configuration into your chosen language.

Storing Terraform state in Pulumi Cloud keeps both tools pointed at one system of record, which is what makes a gradual, low-risk migration practical.

What Pulumi does not magically fix

Pulumi does not eliminate cloud API eventual consistency. A deployment can finish successfully while downstream reads, controllers, or other operators still see stale state for a short window. That is a property of the cloud control plane, not of the IaC tool.

Pulumi also does not eliminate provider bugs. If a provider has a schema issue, a bad default, or a flaky update path, Pulumi still has to ride that provider behavior. Drift will still happen when infrastructure changes outside Pulumi, too. The difference is that Pulumi gives you first-class tooling to detect and reconcile it, as covered in Detect and reconcile drift above — but the drift itself, and the discipline to act on it, do not disappear.

Pulumi does not eliminate preview-time unknowns either. Some values are not known until deployment, so the plan can still contain uncertainty. Bad project decomposition and side-effect-heavy deployment code remain risks too, which is why testing, clear stack boundaries, and disciplined component design still matter.

OpenTofu compatibility caveat

OpenTofu users face many of the same infrastructure-as-code concerns around state, providers, drift, secrets, lifecycle behavior, and migration planning. This post stays Terraform-focused because that is where most teams begin the Pulumi comparison, but the same Pulumi capabilities are relevant when evaluating OpenTofu alternatives or hybrid migration paths.

Pulumi can also work with Terraform provider ecosystems, including long-tail providers that may not have a native Pulumi package yet. Treat that as an interoperability path, not a promise that every Terraform or OpenTofu workflow maps one-for-one without design work.

Get started with Pulumi

If your team is evaluating infrastructure as code options, start with the workflow that creates the most leverage: write infrastructure in the language your team already uses, test shared components, protect critical resources, and migrate one stack at a time.

To go deeper, get started with Pulumi or read the Terraform migration guide.

Pulumi Cloud Now Supports Google Sign-In

Pablo Seibelt — Tue, 10 Mar 2026 00:00:00 +0000

Many developers and platform engineers already use Google accounts daily for email, cloud console access, and collaboration. Until now, signing in to Pulumi Cloud required a GitHub, GitLab, or Atlassian account, or an email/password combination. Today, we’re adding Google as a first-class identity provider, so you can sign in to Pulumi Cloud with the same Google account you already use for everything else.

Adding Google as an identity provider brings several benefits:

Use the account you already have. If your team already lives in Google Workspace, you can sign in to Pulumi Cloud with a single click, no new credentials required.
Inherit your existing security policies. If you’ve already configured two-factor authentication, device management, and other protections in a Google Workspace, you can carry them over to Pulumi Cloud automatically.

How it works

Signing up or signing in

On the Pulumi Cloud sign-in page, you’ll see a new Sign in with Google button alongside the existing GitHub, GitLab, and Atlassian options. If you are a new user, select it, authenticate with your Google account, and you’re in.

If you already have an existing Pulumi Cloud account, make sure to associate to your existing account as described in the next section.

Connecting Google to an existing account

If you already have a Pulumi Cloud account, you can link your Google identity from your account settings:

Navigate to your Account Settings.
Scroll to the Identity providers section.
Under Available identities, select Connect Google.

Once connected, you can use Google to sign in to your existing Pulumi Cloud account.

Google sign-in lets you authenticate with Pulumi Cloud using your individual Google account. It does not enable Google as a single sign-on (SSO) identity provider for your Pulumi Cloud organization.

If your team uses Google Workspace and needs centralized membership governance for Pulumi Cloud, configure SAML SSO with Google Workspace instead. SAML SSO is available on Pulumi Enterprise and Business Critical editions.

Get started

Google sign-in is available now for all new and existing Pulumi Cloud users:

New users: Sign up with Google on the Pulumi Cloud sign-up page.
Existing users: Connect your Google account in your account settings.

For more details, see the Pulumi Cloud accounts documentation.

We’d love to hear your feedback. Join the conversation in the Pulumi Community Slack or open an issue on GitHub.

How We Load Data into Snowflake in Seconds with Pulumi

Pablo Seibelt — Mon, 23 Feb 2026 00:00:00 +0000

When you manage dozens of data-loading pipelines, copying and pasting IaC configurations between them is a recipe for mishap. IAM policies can drift, naming conventions diverge, and every new source is a new opportunity to make a mistake — not to mention compound the problem of duplication. In this post, we’ll show you how you can identify and encapsulate common patterns into composable components and walk through the production lessons we’ve learned running 25+ pipelines for over three years.

What we’ll cover

If you’re loading data into Snowflake and want reusable, composable infrastructure, this post is for you. Here’s what we’ll cover:

Handling and validating GitHub webhooks with AWS Lambda
Streaming webhook payloads directly into Snowflake with Amazon Data Firehose
Wiring it all up with a reusable Pulumi ComponentResource

The companion template also includes S3 auto-ingest and batch loading patterns, which we’ll cover in upcoming posts. We also use Pulumi ESC to handle authentication to both AWS and Snowflake using OpenID Connect.

Our own Josh Kodroff wrote an excellent introduction to Snowpipe with Pulumi. This post builds on his work using the newest Snowflake and AWS provider APIs and the direct Firehose-to-Snowflake destination, which wasn’t available when Josh wrote his post. Some resource names and grant patterns will also differ if you’re comparing the two.

Architecture overview

The following diagram shows the architecture in more detail:

GitHub sends webhook events to a Lambda Function URL.
Lambda validates the HMAC (Hash-based Message Authentication Code) signature and forwards the payload to Amazon Data Firehose.
Firehose streams records directly into Snowflake via the Snowpipe Streaming API. Data appears in Snowflake within seconds.
S3 is used only as a backup destination for failed records.

The direct Firehose-to-Snowflake destination is an AWS-native feature that works with any Snowflake account.

Project setup

Start a new Pulumi Python project and choose uv for dependency management when prompted:

mkdir snowpipe-data-loading && cd snowpipe-data-loading
pulumi new python

Notice Pulumi.yaml shows uv as your selected toolchain:

name: snowpipe-data-loading
runtime:
 name: python
 options:
 toolchain: uv

Add the provider dependencies for AWS, Snowflake, GitHub, and the Random and TLS providers:

uv add pulumi-aws pulumi-snowflake pulumi-github pulumi-random pulumi-tls

That’s it. uv creates the virtual environment and lockfile automatically, and Pulumi uses uv run under the hood to execute your program.

All examples in this post are in Python, but Pulumi supports multiple languages. You can implement the same components in TypeScript, Go, .NET, Java, or YAML.

Managing credentials with Pulumi ESC

This project needs credentials for AWS, Snowflake, and GitHub. Rather than managing long-lived secrets locally, we can use Pulumi ESC to obtain dynamic, short-lived OIDC credentials for AWS and Snowflake at runtime. When you run pulumi up, ESC exchanges a Pulumi-issued OIDC token for temporary credentials from each provider and injects them into your stack config automatically. If you prefer not to use ESC, you can set credentials directly with pulumi config set --secret.

Here is a single ESC environment that handles all three providers:

values:
 aws:
 login:
 fn::open::aws-login:
 oidc:
 duration: 1h
 roleArn: arn:aws:iam::123456789012:role/pulumi-esc-oidc
 sessionName: pulumi-snowpipe-demo
 snowflake:
 login:
 fn::open::snowflake-login:
 oidc:
 account: <your-snowflake-account>
 user: ESC_SERVICE_USER
 organizationName: <your-org-name>
 accountName: <your-account-name>
 environmentVariables:
 AWS_ACCESS_KEY_ID: ${aws.login.accessKeyId}
 AWS_SECRET_ACCESS_KEY: ${aws.login.secretAccessKey}
 AWS_SESSION_TOKEN: ${aws.login.sessionToken}
 SNOWFLAKE_USER: ${snowflake.login.user}
 SNOWFLAKE_TOKEN: ${snowflake.login.token}
 pulumiConfig:
 aws:region: us-west-2
 snowflake:organizationName: ${snowflake.organizationName}
 snowflake:accountName: ${snowflake.accountName}
 snowflake:authenticator: OAUTH
 snowflake:role: PULUMI_DEPLOYER
 github:token:
 fn::secret: <your-github-pat>
 github:owner: <your-github-org-or-user>

Then reference the environment from your stack config:

# Pulumi.<stack>.yaml
environment:
 - <project>/my-snowpipe-env
config:
 snowpipe-data-loading:database: LANDING_ZONE_WEBHOOKS
 snowpipe-data-loading:environment: dev
 snowpipe-data-loading:webhook-repo: <your-test-repo-name>
 snowflake:previewFeaturesEnabled:
 - snowflake_table_resource

Depending on your preferences, you can split credentials into separate per-provider environments and compose them with imports and reuse across stacks.

To set up OIDC trust for each provider, see the AWS OIDC guide and the Snowflake OIDC login guide. For GitHub authentication options (fine-grained PATs, classic PATs, or GitHub Apps), see the pulumi-github provider docs.

Shared infrastructure

The direct streaming pipeline needs an S3 bucket for backup/errors, a Snowflake database, and a schema.

Building the direct ingestion ComponentResource

Amazon Data Firehose supports Snowflake as a native destination via the Snowpipe Streaming API. Firehose streams records directly into Snowflake.

The Lambda handler

The Lambda function is the entry point for GitHub webhooks. It validates the HMAC-SHA256 signature, wraps the payload in an envelope with the event type, and forwards it to Firehose. Create this as lambda/webhook_handler.py:

import hashlib
import hmac
import json
import os

import boto3

firehose = boto3.client("firehose")

STREAM_NAME = os.environ["FIREHOSE_STREAM_NAME"]
WEBHOOK_SECRET = os.environ["WEBHOOK_SECRET"]


def handler(event, context):
 body = event.get("body", "")
 signature = (event.get("headers") or {}).get("x-hub-signature-256", "")

 # Validate HMAC-SHA256 signature
 expected = "sha256=" + hmac.new(
 WEBHOOK_SECRET.encode(), body.encode(), hashlib.sha256
 ).hexdigest()

 if not hmac.compare_digest(expected, signature):
 return {"statusCode": 401, "body": "Invalid signature"}

 github_event = (event.get("headers") or {}).get("x-github-event", "unknown")

 # Wrap in envelope - newline-delimited for S3 backup where Firehose concatenates records
 record = json.dumps({
 "github_event": github_event,
 "payload": json.loads(body),
 }) + "\n"

 firehose.put_record(
 DeliveryStreamName=STREAM_NAME,
 Record={"Data": record.encode()},
 )

 return {"statusCode": 200, "body": "OK"}

The envelope format {"github_event": "<type>", "payload": {...}}\n is important. The github_event field (e.g., push, pull_request, star) comes from the x-github-event header and lets downstream queries filter by event type. The trailing newline delimits records in the S3 backup destination, where Firehose concatenates them into files.

Another interesting part of this snippet is that instead of manually creating a secret which would have to be copy-pasted into both your webhook configuration and your Lambda environment, we are using random.RandomPassword to generate and store it securely it in Pulumi state.

The secret is automatically wired to both the Lambda env var and the GitHub webhook config, and it rotates cleanly if you ever need to replace it.

Why direct Firehose to Snowflake?

Data like this is normally written to and loaded from Amazon S3. But with an S3 intermediate path, you must wait for Firehose to buffer records (60 seconds), then for Snowpipe to detect the new file and load it. Total latency: about two minutes. With the direct Snowflake destination, Firehose uses the Snowpipe Streaming API to insert records as soon as they arrive, in seconds.

The direct path also removes the need for S3 event notifications, SQS queues, external stages, and pipe resources. S3 is still used, but only as a backup destination for failed records.

The DirectSnowflakeIngestion component

The component creates everything needed for the direct path: a TLS key pair for Snowflake authentication, a Snowflake service user with least-privilege grants, the landing table, a Firehose delivery stream with destination="snowflake", and a Lambda function with a public URL. Create an empty components/__init__.py so that Python treats the directory as a package.

The full component lives in components/direct_snowflake_ingestion.py:

import json
from dataclasses import dataclass

import pulumi
import pulumi_aws as aws
import pulumi_snowflake as snowflake
import pulumi_tls as tls

# In the example repository, you will find this class imported from a common library file instead
@dataclass
class ColumnDef:
 """Column definition for a Snowflake table."""

 name: str
 type: str
 nullable: bool = True

def strip_pem_headers(pem: str) -> str:
 """Remove PEM header/footer lines, returning only the base64 content."""
 lines = pem.strip().split("\n")
 return "".join(lines[1:-1])


@dataclass
class DirectSnowflakeIngestionArgs:
 bucket_arn: pulumi.Input[str]
 bucket_name: pulumi.Input[str]
 database: pulumi.Input[str]
 schema_name: pulumi.Input[str]
 table_name: str
 table_columns: list[ColumnDef]
 snowflake_account_url: pulumi.Input[str]
 snowflake_role_name: str
 lambda_code: pulumi.Archive
 lambda_handler: str
 lambda_environment: dict[str, pulumi.Input[str]]
 table_comment: str = ""
 s3_prefix: str = "direct-webhooks"
 s3_backup_mode: str = "FailedDataOnly"
 buffering_interval: int = 0
 buffering_size: int = 1
 retry_duration: int = 60
 data_loading_option: str = "VARIANT_CONTENT_AND_METADATA_MAPPING"
 content_column_name: str = "CONTENT"
 metadata_column_name: str = "METADATA"

class DirectSnowflakeIngestion(pulumi.ComponentResource):
 function_url: pulumi.Output[str]
 firehose_stream_name: pulumi.Output[str]
 snowflake_user_name: pulumi.Output[str]

 def __init__(
 self,
 name: str,
 args: DirectSnowflakeIngestionArgs,
 opts: pulumi.ResourceOptions | None = None,
 ):
 super().__init__(
 "snowpipe:direct:DirectSnowflakeIngestion", name, {}, opts
 )

 # --- TLS key pair for Snowflake auth ---
 key_pair = tls.PrivateKey(
 f"{name}-keypair",
 algorithm="RSA",
 rsa_bits=2048,
 opts=pulumi.ResourceOptions(parent=self),
 )

 # --- Snowflake role, user, and grants ---
 sf_role = snowflake.AccountRole(
 f"{name}-sf-role",
 name=args.snowflake_role_name,
 opts=pulumi.ResourceOptions(parent=self),
 )

 user_name = f"FIREHOSE_{name.upper().replace('-', '_')}_USER"
 sf_user = snowflake.ServiceUser(
 f"{name}-sf-user",
 name=user_name,
 login_name=user_name,
 default_role=sf_role.name,
 rsa_public_key=key_pair.public_key_pem.apply(strip_pem_headers),
 opts=pulumi.ResourceOptions(parent=self),
 )

 # Landing table
 table = snowflake.Table(
 f"{name}-table",
 name=args.table_name,
 database=args.database,
 schema=args.schema_name,
 comment=args.table_comment,
 columns=[
 snowflake.TableColumnArgs(
 name=col.name, type=col.type, nullable=col.nullable,
 )
 for col in args.table_columns
 ],
 opts=pulumi.ResourceOptions(parent=self),
 )

 # Grants: DB USAGE, schema USAGE, table INSERT+SELECT
 snowflake.GrantPrivilegesToAccountRole(
 f"{name}-grant-db-usage",
 account_role_name=sf_role.name,
 privileges=["USAGE"],
 on_account_object=snowflake.GrantPrivilegesToAccountRoleOnAccountObjectArgs(
 object_type="DATABASE",
 object_name=args.database,
 ),
 opts=pulumi.ResourceOptions(parent=self),
 )

 snowflake.GrantPrivilegesToAccountRole(
 f"{name}-grant-schema-usage",
 account_role_name=sf_role.name,
 privileges=["USAGE"],
 on_schema=snowflake.GrantPrivilegesToAccountRoleOnSchemaArgs(
 schema_name=pulumi.Output.all(
 args.database, args.schema_name
 ).apply(lambda parts: f'"{parts[0]}"."{parts[1]}"'),
 ),
 opts=pulumi.ResourceOptions(parent=self),
 )

 table_name = args.table_name
 snowflake.GrantPrivilegesToAccountRole(
 f"{name}-grant-table",
 account_role_name=sf_role.name,
 privileges=["INSERT", "SELECT"],
 on_schema_object=snowflake.GrantPrivilegesToAccountRoleOnSchemaObjectArgs(
 object_type="TABLE",
 object_name=pulumi.Output.all(
 args.database, args.schema_name
 ).apply(
 lambda parts: f'"{parts[0]}"."{parts[1]}"."{table_name}"'
 ),
 ),
 opts=pulumi.ResourceOptions(parent=self, depends_on=[table]),
 )

 # --- Firehose IAM role (S3 backup write) ---
 firehose_role = aws.iam.Role(
 f"{name}-firehose-role",
 assume_role_policy=json.dumps({
 "Version": "2012-10-17",
 "Statement": [{
 "Effect": "Allow",
 "Action": "sts:AssumeRole",
 "Principal": {"Service": "firehose.amazonaws.com"},
 }],
 }),
 opts=pulumi.ResourceOptions(parent=self),
 )

 aws.iam.RolePolicy(
 f"{name}-firehose-s3-policy",
 role=firehose_role.id,
 policy=args.bucket_arn.apply(
 lambda arn: json.dumps({
 "Version": "2012-10-17",
 "Statement": [{
 "Effect": "Allow",
 "Action": [
 "s3:AbortMultipartUpload",
 "s3:GetBucketLocation",
 "s3:GetObject",
 "s3:ListBucket",
 "s3:ListBucketMultipartUploads",
 "s3:PutObject",
 ],
 "Resource": [arn, f"{arn}/*"],
 }],
 })
 ),
 opts=pulumi.ResourceOptions(parent=self),
 )

 # --- Firehose delivery stream (Snowflake destination) ---
 stream = aws.kinesis.FirehoseDeliveryStream(
 f"{name}-firehose",
 destination="snowflake",
 snowflake_configuration=aws.kinesis.FirehoseDeliveryStreamSnowflakeConfigurationArgs(
 account_url=args.snowflake_account_url,
 database=args.database,
 schema=args.schema_name,
 table=args.table_name,
 role_arn=firehose_role.arn,
 user=sf_user.name,
 private_key=key_pair.private_key_pem_pkcs8.apply(
 strip_pem_headers
 ),
 data_loading_option=args.data_loading_option,
 content_column_name=args.content_column_name,
 metadata_column_name=args.metadata_column_name,
 s3_backup_mode=args.s3_backup_mode,
 buffering_size=args.buffering_size,
 buffering_interval=args.buffering_interval,
 retry_duration=args.retry_duration,
 snowflake_role_configuration=aws.kinesis.FirehoseDeliveryStreamSnowflakeConfigurationSnowflakeRoleConfigurationArgs(
 enabled=True,
 snowflake_role=args.snowflake_role_name,
 ),
 s3_configuration=aws.kinesis.FirehoseDeliveryStreamSnowflakeConfigurationS3ConfigurationArgs(
 bucket_arn=args.bucket_arn,
 role_arn=firehose_role.arn,
 prefix=f"{args.s3_prefix}/backup/",
 error_output_prefix=f"{args.s3_prefix}/errors/",
 ),
 ),
 opts=pulumi.ResourceOptions(parent=self, depends_on=[table]),
 )

 # --- Lambda function + Function URL ---
 lambda_role = aws.iam.Role(
 f"{name}-lambda-role",
 assume_role_policy=json.dumps({
 "Version": "2012-10-17",
 "Statement": [{
 "Effect": "Allow",
 "Action": "sts:AssumeRole",
 "Principal": {"Service": "lambda.amazonaws.com"},
 }],
 }),
 opts=pulumi.ResourceOptions(parent=self),
 )

 aws.iam.RolePolicyAttachment(
 f"{name}-lambda-basic-execution",
 role=lambda_role.name,
 policy_arn="arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole",
 opts=pulumi.ResourceOptions(parent=self),
 )

 aws.iam.RolePolicy(
 f"{name}-lambda-firehose-policy",
 role=lambda_role.id,
 policy=stream.arn.apply(
 lambda arn: json.dumps({
 "Version": "2012-10-17",
 "Statement": [{
 "Effect": "Allow",
 "Action": ["firehose:PutRecord"],
 "Resource": [arn],
 }],
 })
 ),
 opts=pulumi.ResourceOptions(parent=self),
 )

 env_vars = {
 **args.lambda_environment,
 "FIREHOSE_STREAM_NAME": stream.name,
 }

 fn = aws.lambda_.Function(
 f"{name}-handler",
 runtime="python3.11",
 handler=args.lambda_handler,
 role=lambda_role.arn,
 timeout=30,
 code=args.lambda_code,
 environment=aws.lambda_.FunctionEnvironmentArgs(
 variables=env_vars,
 ),
 opts=pulumi.ResourceOptions(parent=self),
 )

 fn_url = aws.lambda_.FunctionUrl(
 f"{name}-function-url",
 function_name=fn.name,
 authorization_type="NONE",
 opts=pulumi.ResourceOptions(parent=self),
 )

 aws.lambda_.Permission(
 f"{name}-function-url-permission",
 action="lambda:InvokeFunctionUrl",
 function=fn.name,
 principal="*",
 function_url_auth_type="NONE",
 opts=pulumi.ResourceOptions(parent=self),
 )

 # --- Outputs ---
 self.function_url = fn_url.function_url
 self.firehose_stream_name = stream.name
 self.snowflake_user_name = sf_user.name

 self.register_outputs({
 "function_url": self.function_url,
 "firehose_stream_name": self.firehose_stream_name,
 "snowflake_user_name": self.snowflake_user_name,
 })

A few things to note:

TLS key pair for authentication. The component generates an RSA key pair using the pulumi-tls provider. The public key is assigned to the Snowflake service user; the private key (PKCS#8 format, base64-encoded) is passed to Firehose. No passwords or OAuth tokens are stored.
ServiceUser instead of User. Snowflake service users can’t log in interactively. They authenticate only via key pair, which is exactly what Firehose needs.
destination="snowflake" on Firehose. This tells Firehose to use the Snowpipe Streaming API rather than writing to S3. The s3_configuration block is still required, but only for backup/error records.
Immediate flushing. buffering_interval=0 and buffering_size=1 ensure records are sent to Snowflake as soon as they arrive, minimizing latency. Tune according to your needs.

Amazon Data Firehose does not connect from fixed IP addresses, so you cannot use Snowflake network policies to restrict access by IP. If your Snowflake account uses network policies, you have three options: use AWS PrivateLink (requires Snowflake Business Critical edition), allow public internet access for the Firehose service user, or switch to S3 auto-ingest via Snowpipe which does not require direct network access to Snowflake from Firehose.

Wiring it together

With the component defined, __main__.py wires the direct ingestion pipeline:

import pulumi
import pulumi_aws as aws
import pulumi_github as github
import pulumi_random as random
import pulumi_snowflake as snowflake
from components.direct_snowflake_ingestion import (
 DirectSnowflakeIngestion, DirectSnowflakeIngestionArgs,
)
from components.snowpipe_pipeline import ColumnDef

config = pulumi.Config()
database_name = config.get("database") or "LANDING_ZONE_WEBHOOKS"
environment = config.get("environment") or "dev"

# --- Shared infrastructure ---

# S3 bucket for backup/errors
bucket = aws.s3.Bucket(
 "data-landing-bucket",
)

# Snowflake database and schema
database = snowflake.Database("demo-database", name=database_name)
schema = snowflake.Schema(
 "demo-schema",
 name="GITHUB",
 database=database.name,
)

# --- Direct ingestion pipeline ---

webhook_repo = config.require("webhook-repo")

# Snowflake account URL for Firehose configuration
snowflake_config = pulumi.Config("snowflake")
snowflake_account_url = (
 f"https://{snowflake_config.require('organizationName')}"
 f"-{snowflake_config.require('accountName')}"
 f".snowflakecomputing.com"
)

# Landing table columns: CONTENT (webhook JSON) + METADATA (Firehose metadata)
DIRECT_COLUMNS = [
 ColumnDef(name="CONTENT", type="VARIANT", nullable=True),
 ColumnDef(name="METADATA", type="VARIANT", nullable=True),
]

# Step 1: Generate webhook secret for HMAC validation
direct_webhook_secret = random.RandomPassword(
 "github-direct-webhook-secret", length=32, special=False
)

# Step 2: Direct ingestion pipeline - Lambda validates, Firehose streams to Snowflake
direct = DirectSnowflakeIngestion(
 "github-webhooks-direct",
 DirectSnowflakeIngestionArgs(
 bucket_arn=bucket.arn,
 bucket_name=bucket.bucket,
 database=database.name,
 schema_name=schema.name,
 table_name="REPOSITORY_EVENTS_DIRECT",
 table_columns=DIRECT_COLUMNS,
 table_comment="GitHub webhook events loaded via direct Firehose to Snowflake",
 snowflake_account_url=snowflake_account_url,
 snowflake_role_name="FIREHOSE_DIRECT_LOADER",
 lambda_code=pulumi.AssetArchive({
 "webhook_handler.py": pulumi.FileAsset("lambda/webhook_handler.py"),
 }),
 lambda_handler="webhook_handler.handler",
 lambda_environment={"WEBHOOK_SECRET": direct_webhook_secret.result},
 ),
)

# Step 3: GitHub webhook - sends events to the Lambda Function URL
github.RepositoryWebhook(
 "github-direct-webhook",
 repository=webhook_repo,
 configuration=github.RepositoryWebhookConfigurationArgs(
 url=direct.function_url,
 content_type="json",
 secret=direct_webhook_secret.result,
 ),
 events=["push", "pull_request", "issues", "star"],
)

# Exports
pulumi.export("webhook_url", direct.function_url)
pulumi.export("firehose_stream", direct.firehose_stream_name)

That’s the entire pipeline. One component, one GitHub webhook, one secret. The DirectSnowflakeIngestion component handles the TLS key pair, Snowflake service user, landing table, Firehose stream, and Lambda function internally, and now you can reuse this component for as many pipelines as you need.

The full code for this example is available on GitHub:

github.com/pulumi-demos/examples/tree/main/python/aws-snowflake-data-loading-real-time

Testing the pipeline

Deploy the stack:

pulumi up

The entire stack deploys in about two minutes. Immediately after deployment, you’ll start seeing GitHub events flowing into Snowflake.

Before querying, grant the DATA_READER role to your Snowflake user:

GRANT ROLE DATA_READER TO USER <your-user>;

In production, you can manage this grant through Pulumi, manually, or automatically via SCIM provisioning from your identity provider.

No need to craft test payloads. Just interact with the test repo. Star it, push a commit, or open an issue, then wait about 30 seconds and query Snowflake using the least-privilege reader role:

USE ROLE DATA_READER;

SELECT CONTENT:github_event::STRING AS event_type,
 CONTENT:payload:repository:full_name::STRING AS repo,
 METADATA:IngestionTime::TIMESTAMP AS ingested_at
FROM LANDING_ZONE_WEBHOOKS.GITHUB.REPOSITORY_EVENTS_DIRECT
ORDER BY ingested_at DESC;

You should see rows with event types like star, push, or issues, real GitHub events flowing through the entire pipeline. The METADATA column includes Firehose metadata like IngestionTime, which you can use to track end-to-end latency.

Other loading patterns

Direct streaming is the fastest path, but two other patterns are available in the companion template for different requirements:

S3 auto-ingest via Snowpipe. Firehose buffers to S3, and Snowpipe auto-ingests new files. Latency is about two minutes. Best when you need S3 as the system of record or can’t use direct Snowpipe Streaming.
Batch loading. Your orchestrator (Airflow, Prefect, cron, etc.) runs COPY INTO on a schedule. Best for full control over timing and deduplication.

We’ll walk through both patterns in detail in upcoming posts.

Stamp out data pipelines with components

Encapsulate your Snowflake loading logic in a Pulumi component, then create new pipelines with a handful of configuration values instead of copied files.

Get started

Publishing as reusable components

Once your components are battle-tested, you can share them across teams and projects instead of copying files around.

The most straightforward approach: push your components to a Git repository with a PulumiPlugin.yaml file at the root:

runtime: python
name: snowpipe-components
version: 1.0.0

Consumers add the package to their project with pulumi package add:

pulumi package add github.com/your-org/pulumi-snowpipe@v1.0.0

Pulumi downloads the package and generates typed SDKs automatically. The consumer’s __main__.py imports your components as if they were local, but they’re versioned and pinned.

Pulumi Cloud Private Registry

For organization-level discoverability, publish to the Pulumi Cloud Private Registry with pulumi package publish:

pulumi package publish ./schema.json

This gives you auto-generated API docs, usage tracking across teams, and cross-language SDK generation. Your Python components become usable from TypeScript, Go, and C# without any extra work. Teams browse available components in the Pulumi Cloud console, see who’s using what, and get notified when new versions are published. As an additional benefit, Neo will be able to use these components and build new pipelines in minutes from a natural language request.

Conclusion

ComponentResource is the key abstraction that makes this architecture scale. Instead of copying and pasting resources for each new data source, you instantiate a component with a handful of configuration parameters.

The DirectSnowflakeIngestion component in this post delivers data from GitHub webhooks into Snowflake in seconds: Lambda validates the HMAC signature, Firehose streams directly to Snowflake via the Snowpipe Streaming API, and the TLS key pair is managed entirely within Pulumi. No S3 intermediate, no SQS queues.

The component accepts pluggable Lambda handlers, so swapping GitHub for Stripe webhooks or any other source is just a matter of providing different lambda_code and lambda_environment arguments. We’ve been running this pattern in production for over three years across dozens of pipelines without significant changes to the infrastructure code.

You’ll find the complete example in the GitHub repository.

How We Built Platybot: An AI-Powered Analytics Assistant

Pablo Seibelt — Wed, 11 Feb 2026 00:00:00 +0000

Before Platybot, our #analytics Slack channel was a support queue. Every day, people from every team would ask questions: “Which customers use feature X?”, “What’s our ARR by plan type?”, “Do we have a report for template usage?” Our two-person data team was a bottleneck.

Our #analytics channel, before Platybot (dramatized).

We didn’t want to just throw an LLM at our Snowflake warehouse either. Without guardrails, large language models generate SQL that may work but silently gets the answer wrong. Different join logic, wrong filters, missing snapshot handling, incorrect summarization. We needed something that could answer reliably for most queries, otherwise we’d switch to fixing LLM SQL queries.

So we built Platybot (platypus + bot, named after our mascot), an AI-powered analytics assistant that any Pulumi employee can use to query our Data Warehouse in natural language. It’s available as a Web App, a Slack bot, and a Model Context Protocol (MCP) server. The infrastructure is deployed with Pulumi IaC (Infrastructure as Code). But the most important thing we learned building it is that the AI was the easy part. The semantic layer is what makes it work.

The problem with throwing AI at your Data Warehouse

The naive solution is obvious: connect an LLM to your database and let it write SQL. But this fails in practice, and the failure mode is insidious. Consider a few examples from our warehouse:

ARR is a snapshot metric. If you query ARR (Annual Recurring Revenue) without filtering by end-of-period dates (last day of month or quarter), you get duplicate rows and wildly inflated numbers. An LLM doesn’t automatically know this.
Account queries need exclusions. Most queries should exclude Pulumi’s own internal accounts and deleted ones. Without these filters, you’re counting test data alongside real customers.
User queries need employee filters. Querying active users without excluding Pulumi employees inflates adoption metrics.

The danger shows up when results feel decision-ready before anyone has validated how the numbers were derived. A confidently wrong ARR figure presented to leadership is worse than no answer at all.

Many organizations run into the same constraint once data usage spreads. Dashboards answer yesterday’s questions, not today’s. Ad-hoc LLM queries answer today’s questions, but incorrectly. The gap between “I have a question” and “I have a trustworthy answer” is where data teams can get stuck.

Why we built a semantic layer first

Before writing a single line of AI code, we built a semantic layer using Cube (open source). This was the hardest, least glamorous, and most important part of the entire project.

A semantic layer is a shared, versioned definition of what your business metrics mean.

“Monthly active users” starts with COUNT(DISTINCT user_id), but the aggregation is only the outer layer. It has to be applied to the right table (fct_pulumi_operations), with the right filters (exclude Pulumi employees, exclude deleted organizations), scoped to a calendar month, and counting only users who performed real operations — not just previews. The semantic layer encodes all of this once, and the AI can use it to build queries without needing to guess which tables are related to each other, whether it’s one-to-one or many-to-many, etc.

We organized our data into seven domains: Revenue, Cloud, Core, Clickstream, Community, Support, and People. Each domain contains cubes (think of them as well-defined, composable views) with explicit measures, dimensions, and joins. Here’s a real example from our Cloud domain (trimmed for readability):

cubes:
 - name: fct_pulumi_operations
 sql_table: CLOUD.FCT_PULUMI_OPERATIONS
 description: >
 Pulumi CLI operations (update, preview, destroy, refresh).
 Each row is one operation with resource changes, duration,
 and CLI environment.

 joins:
 - name: dim_organization
 sql: '{CUBE}.ORGANIZATION_HK = {dim_organization}.ORGANIZATION_HK'
 relationship: many_to_one
 - name: dim_stacks
 sql: '{CUBE}.STACK_PROGRAM_HK = {dim_stacks}.STACK_PROGRAM_HK'
 relationship: many_to_one
 - name: dim_user
 sql: '{CUBE}.USER_HK = {dim_user}.USER_HK'
 relationship: many_to_one

 measures:
 - name: count
 type: count
 - name: resource_count
 sql: '{CUBE}.RESOURCE_COUNT'
 type: sum
 description: Number of resources active when the operation finished
 - name: operations_succeeded
 sql: "CASE WHEN {CUBE}.OPERATION_STATE = 'succeeded' THEN 1 ELSE 0 END"
 type: sum
 description: Count of operations that succeeded
 # ... plus other measures

 dimensions:
 - name: operation_type
 sql: '{CUBE}.OPERATION_TYPE'
 type: string
 description: Type of operation (Refresh, Update, Destroy, etc)
 - name: operation_state
 sql: '{CUBE}.OPERATION_STATE'
 type: string
 description: Last known state of this operation
 # ... 20+ more dimensions

The key insight: the semantic layer makes the AI’s job tractable. Instead of generating arbitrary SQL from scratch, where the search space is “any possible SQL query against hundreds of tables,” the AI picks from a defined set of measures, dimensions, and joins. The search space shrinks from almost infinite to a well-bounded set of valid combinations.

A semantic layer is to an AI data assistant what a type system is to a programming language. It doesn’t eliminate errors, but it makes entire categories of mistakes structurally impossible. The AI can’t calculate ARR wrong because it doesn’t calculate ARR at all. It references a pre-defined measure that already encodes the correct logic.

Building the semantic layer was mainly data engineering work. We already had agreed-upon metric definitions across the company. The challenge was encoding those definitions into Cube: specifying the correct joins between tables, wiring up the right filters, and making sure every measure matched the logic our dashboards already used. Tedious, but essential.

Adding the AI layer

With the semantic layer in place, the AI becomes a translation problem: convert natural language into a Cube query.

Platybot supports multiple models: Claude Opus 4.6, Claude Sonnet 4.5, and Gemini 3 Pro. Users can choose which model to use. We found that different models have different strengths. Claude excels at structured data queries, while Gemini performs very well on text-heavy tasks like analyzing call transcriptions.

The system prompt gives the model awareness of available cubes, their measures, dimensions, and joins. When a user asks “What’s the ARR breakdown by plan type?”, the model doesn’t write SQL. Instead, it constructs a Cube query, selecting the total_arr measure from the ARR table and grouping by the sku from the subscriptions dimension. Cube handles the SQL generation, the joins, and the filters. For edge cases the semantic layer doesn’t cover, the model can fall back to direct (read-only) SQL against Snowflake, but it may already have a basic query that is already close to what it needs.

Query generation follows a workflow that maps to the same tools a human analyst would use:

flowchart TB
subgraph Entry["Entry points"]
Web["Web UI"]
Slack["Slack (@platybot)"]
MCP["MCP Server"]
end
subgraph Core["Platybot core"]
BE["Backend (Express + LLM)"]
end
subgraph Data["Data layer"]
Cube["Cube semantic layer"]
SF["Snowflake"]
end
Web --> BE
Slack --> BE
MCP --> BE
BE -->|"Structured queries"| Cube
BE -->|"Fallback SQL (read-only)"| SF
Cube --> SF

The workflow is: discover domains, explore cubes, understand the schema, construct and execute the query. This mirrors how a data analyst would work. You don’t jump straight to SQL; you first understand what data is available and what the metrics mean.

The system’s role is narrower than people expect. It’s a translator between human intent and a well-defined data model, not a general-purpose data scientist. This is a feature, not a limitation. By constraining Platybot to operate within the semantic layer, we get reliability. The creativity goes into understanding the question, not into inventing SQL.

Meeting users where they are

The backend solved query generation. It did not solve usage and if people don’t use it, it doesn’t matter. We launched Platybot across three interfaces, each designed for a different workflow.

The web UI

The web app is the primary interface: a React 19 + TypeScript + Vite + Tailwind conversational UI where employees can explore data through multi-turn conversations. It supports table visualization, data export, and conversation history so you can pick up where you left off.

Every analysis produces a shareable report with a permanent link, protected behind company authentication. The report shows the agent’s reasoning so you can verify its approach, and lists every query and table used along with a sample of the data. Each query includes a direct link to run it in Metabase, our reporting tool, so any useful query can be saved, scheduled, or extended without starting from scratch.

The web UI sees the highest adoption of all three access points. It’s where people go for deeper data exploration: multi-step analyses, follow-up questions, and comparing metrics across different dimensions.

One fun detail we’ve added during our last hackathon: while queries iterate through analysis steps (discovering domains, exploring cubes, executing queries), users are entertained by a platypus-themed runner game. Think Chrome’s dinosaur game, but with our mascot on a skateboard. Sometimes Platybot can take a long time iterating, so you can try to beat your high score in the meantime!

Slack

Platybot is also available as a Slack bot. Users @mention it in any channel, and it replies in a thread with the answer, keeping the channel clean while making results visible to the whole team. It’s best suited for quick lookups — “what’s the ARR for account X?” — where the answer fits in a message rather than a deep analysis. Most usage stayed in the web UI, but the Slack bot fills a different niche: answers that benefit from being shared in context.

MCP: Making Platybot usable by Agents

The Model Context Protocol (MCP) server is the newest addition, launched February 4, 2026. It exposes six tools that any MCP-compatible client can use:

list_domains - discover available data domains
list_cubes - explore cubes within a domain
get_cube_details - get measures, dimensions, and joins for a cube
execute_cube_query - run a structured query
execute_sql_query - raw SQL fallback (read-only)
get_generated_sql - preview SQL without executing

Setup is a single command:

claude mcp add --transport http platybot <platybot-mcp-url>

This is a paradigm shift. Platybot goes from a destination (open the app, ask a question) to a capability that any AI tool can use. An engineer writing a postmortem can pull live metrics without leaving their terminal. An analyst building a report in Claude Code can query data inline. The Data Warehouse becomes ambient, always available, never in the way.

Security uses OAuth 2.0 Device Authorization Flow with PKCE for CLI-friendly authentication, with @pulumi.com domain restriction, read-only enforcement, rate limiting, and full audit logging.

A meta moment: we used the Platybot MCP server to query our Slack messages table to find the real usage data cited above.

Deploying with Pulumi

Platybot’s infrastructure runs on AWS and is managed entirely with Pulumi IaC. The stack includes ECS Fargate for the application, an Application Load Balancer for traffic routing, EFS for persistent storage, ECR for container images, and Route 53 for DNS. The setup is intentionally simple and we use a small instance size since it’s just for internal usage.

Here’s a representative snippet of the Fargate service definition:

const service = new awsx.ecs.FargateService("platybot", {
 cluster: cluster.arn,
 assignPublicIp: false,
 taskDefinitionArgs: {
 container: {
 name: "platybot",
 image: image.imageUri,
 essential: true,
 portMappings: [{
 containerPort: 3000,
 targetGroup: targetGroup,
 }],
 environment: [
 { name: "CUBE_API_URL", value: cubeApiUrl },
 { name: "NODE_ENV", value: "production" },
 ],
 secrets: [
 { name: "ANTHROPIC_API_KEY", valueFrom: anthropicKeySecret.arn },
 { name: "SNOWFLAKE_PASSWORD", valueFrom: snowflakePasswordSecret.arn },
 ],
 logConfiguration: {
 logDriver: "awslogs",
 options: {
 "awslogs-group": logGroup.name,
 "awslogs-region": aws.config.region!,
 "awslogs-stream-prefix": "platybot",
 },
 },
 },
 },
});

Dogfooding Pulumi for our internal tools has real benefits beyond the obvious. Staging environments are trivial: spin up a full copy of the stack with different parameters. Secrets management is built in, so API keys and database credentials never touch a config file. And when something needs to change, the diff-and-preview workflow (pulumi preview) catches mistakes before they hit production.

Results

Since launch in September 2025: over 1,700 questions from 83 employees across every team. Usage grew steadily — from around 8 questions per day in the first month to 18 per day by January 2026, with 51 unique users that month alone. This was real production work, not experimentation. Customer analysis for sales calls, resource breakdowns for account managers, blog performance metrics for marketing, policy adoption research for product, ARR deep-dives for leadership.

The impact on the data team was immediate. Questions that used to land in the #analytics channel and wait for a human now get answered in seconds. The data team shifted from answering routine queries to building better models and improving data quality. We went from being a help desk to being a platform team.

Rumors of our replacement have been greatly exaggerated. Platybot just freed us up for the work that doesn't fit in a Slack reply.

Accuracy is harder to quantify, but the semantic layer gives us confidence. Because Platybot uses pre-defined measures instead of generating arbitrary SQL, entire categories of errors (wrong joins, missing filters, incorrect aggregations) are structurally less likely. When the model does get something wrong, it’s usually in interpreting the question, not in the data — and users can verify that through the report’s reasoning trace.

What we learned

The semantic layer matters more than the model. We spent more time defining metrics in Cube than we did on any AI work. That investment pays for itself: swap the model, and the answers stay correct. Swap the semantic layer, and nothing works.

Meet users where they already are. Different UIs handle different cognitive loads. Slack catches the quick questions, the web UI handles deep exploration, and MCP makes data ambient for AI-native workflows. Each interface serves a different mode of thinking.

Transparency builds trust. Showing the AI’s reasoning, the queries it ran, and linking to Metabase for verification turned skeptics into regular users. People don’t trust a black box, but they’ll trust a tool that shows its work.

Tap-Pulumi-Cloud: Simplifying Pulumi Cloud Data Integration

Pablo Seibelt — Tue, 03 Sep 2024 09:00:00 -0700

Integrating various infrastructure data sources into your data warehouse has long been a challenge for Platform Teams. Whether it’s dealing with multiple API endpoints, managing complex authentication processes, or just trying to get a consistent, reliable data feed, the process can be daunting and time-consuming. Especially when you factor in the various cloud providers, and the inconsistency in data formats across them all.

These pain points can slow down your ability to get actionable insights from your infrastructure data, leaving you with more questions than answers.

The tap-pulumi-cloud connector is designed to address these challenges head-on by offering a simple solution for automating the process of accessing infrastructure data.

Leveraging Pulumi Cloud data about your infrastructure instead of going directly to the provider eliminates the need for custom API integrations and handles the data consistency problem. And while all this data existed before by using the Pulumi Cloud console or Pulumi Cloud REST API directly, the tap-pulumi-cloud connector handles complex interactions with the API, transforming the raw data into a structured format that’s ready for analysis as soon as it lands in your data warehouse. This means less time spent on data wrangling and more time on generating insights.

A few examples of the types of analytics you can build on top Pulumi Cloud data:

Average resources under management overtime (code example)
The average time to deploy changes (code example)
See the total updates per user (code example)
Join with CI/CD provider data to generate DORA metrics and reporting
Join with resource cost data to generate infrastructure cost reporting

As it is built on the Meltano SDK, you can use tap-pulumi-cloud with loaders such as target-snowflake, target-bigquery, target-redshift or even target-postgres; allowing us to load the data into our own data warehouse easily.

Setup

Firstly, set up your environment by following Meltano’s installation guide. Once this is done, continue by adding tap-pulumi-cloud and your choice of loader into your environment, we’ll use target-duckdb for this example:

meltano add extractor tap-pulumi-cloud
meltano add loader target-duckdb

Then switch your tap variant to use Pulumi’s instead of the default, open meltano.yml and change the pip_url to git+https://github.com/pulumi/tap-pulumi-cloud.git

version: 1
default_environment: dev
project_id: be90b150-3e48-4ecd-be9d-161ef7417e3c
environments:
- name: dev
- name: staging
- name: prod
plugins:
 extractors:
 - name: tap-pulumi-cloud
 variant: meltanolabs
 pip_url: git+https://github.com/pulumi/tap-pulumi-cloud.git
 loaders:
 - name: target-duckdb
 variant: jwills
 pip_url: target-duckdb~=0.6

Then run meltano install to make sure the right version is installed.

Configure it

To configure all settings available for this tap, you can use Meltano’s interactive config running:

meltano config tap-pulumi-cloud set --interactive

The required variables are the token (Get one from Pulumi Cloud) and organizations is a list of Pulumi Cloud organizations you want to extract data from.

Load data

When your configuration has been finished, you can run the data pipeline:

meltano run tap-pulumi-cloud target-duckdb

After the run finishes, you can access the exported tables and create whichever analytics you need on top of them, for example, you could combine this dataset with tap-github in order to calculate DORA Metrics.

Orchestration

Once you have this working, you’ll need to establish some way of running this regularly in an automated fashion, to keep the data fresh. A popular way to do this is to use an orchestration tool such as Airflow, Dagster or Mage, among others. An easy way to set up Meltano to run in this way is to use a Docker container with your Meltano project.. Another option is to use Arch which is based on Meltano, which path you choose will depend on your requirements and existing stack.

Since each table is created with primary keys, when the process is run again, each run will “upsert” (update+insert) new data, replacing rows which match the same primary keys; e.g. stream Stacks has a primary key by org_name, project_name and stack_name, so if the same combination of keys arrives in the next update, the row will be updated instead of a new row being inserted.

If building a landing zone with all historical changes instead of just the current status, this behavior might not be the desired one, to ignore primary keys and only insert new data, you can add this setting to meltano.yml on the plugin configuration for tap-pulumi-cloud:

 metadata:
 '*':
 table-key-properties: []
 key-properties: []

Generate metrics

With all of the data in one place, you can generate metrics based on Pulumi Cloud data, and combine it with other data sources, for example if you wanted to look at your average Resources Under Management (RUM) monthly (Using DuckDB’s SQL Flavor, adjust to your specific database):

See average resources under management

WITH monthly_rum_average AS (
 SELECT
 DATE_TRUNC('month', MAKE_DATE(CAST(year AS INT), CAST(month AS INT), CAST(day AS INT))) AS rum_month,
 org_name,
 AVG(resources) AS monthly_avg_rum
 FROM
 pulumicloud.daily_rum_usage
 GROUP BY
 rum_month, org_name
)
SELECT
 rum_month,
 org_name,
 monthly_avg_rum,
 LAG(monthly_avg_rum) OVER (ORDER BY rum_month) AS previous_month_avg_rum,
 CASE
 WHEN LAG(monthly_avg_rum) OVER (ORDER BY rum_month) IS NULL THEN NULL
 ELSE ((monthly_avg_rum - LAG(monthly_avg_rum) OVER (ORDER BY rum_month)) / LAG(monthly_avg_rum) OVER (ORDER BY rum_month)) * 100
 END AS month_over_month_rum_growth_percentage
FROM
 monthly_rum_average

See the average time to deploy changes

If you want to see the average time to deploy changes, excluding console-initiated deploys (Similar to what is tracked for “Lead time for changes”):

WITH stack_deployments AS (
 SELECT
 CAST(json_extract(jobs, '$[0].started') AS TIMESTAMP) AS start_time,
 CAST(json_extract(jobs, '$[0].last_updated') AS TIMESTAMP) AS end_time,
 initiator
 FROM
 pulumicloud.stack_deployments
 WHERE
 initiator IS NOT NULL
 AND initiator <> 'console'
)
SELECT
 initiator,
 AVG(DATE_DIFF('second', start_time, end_time)) / 60 AS avg_deploy_time_in_minutes
FROM
 stack_deployments
GROUP BY
 initiator

See the total updates per user

WITH operations_by_members AS (
 SELECT
 org_name,
 REPLACE(CAST(json_extract(requested_by, '$.github_login') AS STRING), '"', '') AS github_login,
 COUNT(*) AS total_updates
 FROM
 pulumicloud.stack_updates
 GROUP BY
 org_name, github_login
 ORDER BY
 total_updates DESC
)
SELECT
 om.org_name,
 om.role,
 om.user_github_login,
 om.user_name,
 obm.total_updates
FROM
 pulumicloud.organization_members om
INNER JOIN
 operations_by_members obm
ON
 om.org_name = obm.org_name
 AND om.user_github_login = obm.github_login
ORDER BY
 total_updates DESC

Conclusion

Integrating Pulumi Cloud data into your data warehouse has never been easier with tap-pulumi-cloud. Whether you’re tracking costs, monitoring deployments, or improving security, the new tap connector empowers your team to make informed, data-driven decisions that drive your cloud strategy forward.

Pulumi Blog: Pablo Seibelt

Build a Governed Databricks Workspace with Pulumi

What you’ll build

The Databricks management boundary

Configuring credentials with Pulumi ESC

Defining your data infrastructure

Validation

Conclusion

Cloudflare-First Networking as Code with Pulumi

What you’ll build

The Pulumi program

Setting up the provider

Defining the infrastructure

Validation

Use Your Mac for AI Agents: Self-Host Gemma 4 12 B with Pulumi and Tailscale

Prerequisites

Run Gemma 4 with host-native llama.cpp

Verify the LLM API

Keep llama.cpp running after reboot

Deploy Open WebUI with Pulumi and k3d

Access Open WebUI through Tailscale

Use the model with Pi

Advanced: Linux GPU in-cluster serving

Cleanup

Build an EKS Environment Factory with Pulumi and vCluster

The problem: environment sprawl and provisioning lag

Architecture overview: the host and the tenants

Implementation: the EKS Auto Mode host

Implementation: the environment factory

Tenant guardrails

Deploying vCluster with Helm

Accessing the virtual cluster

Operational caveats

Conclusion: ephemeral environments at scale

Why Choose Pulumi Over Terraform?

Executive summary

Use programming languages and familiar tools

Refactor infrastructure without surprise replacement

Handle secrets with encrypted configuration and secret outputs

Add safer lifecycle controls for destructive changes

Model environments with stacks

Test infrastructure with normal language tooling

Wire providers explicitly

Detect and reconcile drift

Import and migrate incrementally

Manage Terraform state in Pulumi Cloud

What Pulumi does not magically fix

OpenTofu compatibility caveat

Get started with Pulumi

Pulumi Cloud Now Supports Google Sign-In

How it works

Signing up or signing in

Connecting Google to an existing account

Google sign-in vs. SAML SSO

Get started

How We Load Data into Snowflake in Seconds with Pulumi

What we’ll cover

Architecture overview

Project setup

Managing credentials with Pulumi ESC

Shared infrastructure

Building the direct ingestion ComponentResource

The Lambda handler

Why direct Firehose to Snowflake?

The DirectSnowflakeIngestion component

Wiring it together

Testing the pipeline

Other loading patterns

Publishing as reusable components

Git-based sharing

Pulumi Cloud Private Registry

Conclusion

How We Built Platybot: An AI-Powered Analytics Assistant

The problem with throwing AI at your Data Warehouse

Why we built a semantic layer first

Adding the AI layer

Meeting users where they are

The web UI

Slack

MCP: Making Platybot usable by Agents