Pulumi Blog: Best practices

GitOps Best Practices I Wish I Had Known Before

Engin Diri — Thu, 19 Feb 2026 00:00:00 +0000

Getting started with GitOps can feel like trying to herd cats through a YAML factory while the factory is on fire. It’s one of those things that seems like it ought to be simple (just use Git!), but in practice is much more complex — and you may not realize how much more complex until you’re weeks or more into a project. After years of running GitOps workflows in production across dozens of clusters, I’ve collected a list of best practices that I’m hoping can save you from having to make many of the mistakes I’ve made. Think of it as the GitOps cheat sheet I wish I’d had from Day 1.

Since this post was published, Pulumi has added first-class support for HashiCorp Configuration Language (HCL). You can now write Pulumi programs in HCL directly, alongside general-purpose languages like TypeScript, Python, Go, and C#. To see how it works, see Pulumi HCL.

If you’re not familiar with the formal definition, the OpenGitOps project distills it into four principles:

Declarative desired state
Versioned and immutable storage
Automatic pulling
Continuous reconciliation

But those principles only define the what. This post is also about the how — the practical lessons that can make or break a GitOps implementation.

In this post, I’ll walk you through the GitOps best practices I’ve picked up from production experience, community talks, and more than a few late-night incident calls. Whether you’re just getting started with GitOps or looking to level up, these tips should help you avoid the potholes.

The GitOps mindset in a nutshell.

1. Git is your single source of truth (no, really)

This is the bedrock that the principles of GitOps are built on: every piece of your environment’s desired state lives in a Git repository. No exceptions. No “I’ll just fix it real quick with kubectl edit.” No “let me patch this configmap by hand because the PR process takes too long.”

The moment you make a manual change to your cluster, you’ve created drift between what Git says the world should look like and what it actually looks like. Drift will quietly wreck your GitOps workflows if you let it.

Put everything in Git: Kubernetes manifests, Helm values, Kustomize overlays, policy rules, even your GitOps tool configuration itself.
No manual kubectl edits. If it’s not in Git, it doesn’t exist. Period. Train your team to treat direct cluster changes like touching a hot stove.
You get an audit trail for free. Git gives you a complete history of who changed what, when, and why. That’s your compliance audit trail baked right in.

Pro Tip: Enable branch protection rules on your GitOps repos from Day 1. This prevents anyone from pushing directly to main and bypassing the review process. Future you will thank present you during the next audit.

2. Declarative over imperative, always

If you’re still running sequences of kubectl create, kubectl patch, and kubectl delete commands to manage your cluster, you’re not really doing GitOps yet. Declarative means you define what you want the end state to look like, not the step-by-step instructions to get there.

Think of it like ordering at a restaurant. Imperative: “Go to the kitchen, grab flour, knead dough, preheat the oven to 200 degrees, shape the pizza, add sauce…” Declarative: “One margherita pizza, please.” Let the system figure out how to make it happen.

Your manifests describe the end result. The GitOps operator reconciles reality to match.
It’s idempotent by design. Apply the same manifest ten times, get the same result. No side effects, no surprises.
Rollbacks are easier: just revert a commit. The operator sees the previous desired state and reconciles. A caveat though: this only works for stateless resources; database schema migrations, CRD version changes, persistent volume modifications, and rotated secrets don’t always revert cleanly by rolling back to a previous commit. Be sure to plan rollbacks involving stateful resources carefully.

Pro Tip: If you find yourself writing shell scripts that run a sequence of kubectl commands, stop and ask yourself: “Can I express this as a declarative manifest instead?” Nine times out of ten, the answer is yes.

3. Pull-based deployments are the way

Traditional CI/CD is push-based: your pipeline builds an artifact and then pushes it to the cluster. GitOps flips this. An agent running inside your cluster continuously polls a Git repository and pulls changes when it detects them.

Why does this matter? With push-based deployments, your CI system needs credentials to access your cluster. That’s a wide attack surface. With pull-based, the agent already lives inside the cluster and only needs read access to your Git repo.

Tools like ArgoCD and FluxCD run controllers inside your cluster that watch your Git repos.
Your CI pipeline never needs kubeconfig access. The agent handles deployment.
The agent doesn’t just deploy once; it continuously ensures the cluster matches the desired state in Git.

Pro Tip: You may be tempted to set your reconciliation interval to something aggressive like 1 minute so you always know exactly which version is deployed. That works for a while, but at scale (200+ applications polling every minute) you can blow through GitHub’s API rate limits (5,000 requests/hour for authenticated users) and put real pressure on the Kubernetes API server.

A better approach: set up webhook receivers. Both ArgoCD and FluxCD support incoming webhooks from GitHub, GitLab, and Bitbucket. Your Git provider pings the GitOps controller on every push, so reconciliation kicks off in seconds instead of waiting for the next poll. That alone kills most of the API rate limit pain at scale. You can then relax polling to a 5 or 10 minute fallback for the rare case where a webhook doesn’t fire.

4. Separate app code from deployment config (when it hurts not to)

Let me be upfront: if you’re a small team with one or two services, keeping your Kubernetes manifests next to your application source code in the same repo is fine. A monorepo is simpler to manage and one less thing to automate. Don’t split repos just because a blog post told you to.

That said, ArgoCD’s official best practices call separate repos “highly recommended,” and there are real reasons for that once you grow. Application code and deployment configuration have different lifecycles. You might bump resource limits in a Helm values file without touching a single line of app code. Or you might refactor your entire codebase without changing any deployment parameters. In a shared repo, every config tweak triggers your full CI pipeline, and ArgoCD invalidates the manifest cache for all applications in the repo on every commit, not just the ones that changed. That cache invalidation alone can become a performance problem with dozens of apps in one repo.

When config tweaks start blocking app code from getting through CI, it may be time to split.
When different teams need different access levels (not everyone who pushes app code should modify production deployment config), it’s time to split.
When you’re running microservices with independent release cadences that step on each other, it’s time to split.

If none of those apply yet, don’t split. You’ll know when the pain arrives.

Pro Tip: When you do split, a common pattern is two repos: one for application source code, one for deployment configuration (Helm charts, Kustomize overlays, environment-specific values). Some larger orgs go further with a third repo for environment overrides, following the fleet repo model from the FluxCD maintainers. Either way, pair it with image update automation (Flux Image Automation Controller or ArgoCD Image Updater) so image tag bumps across repos don’t turn into manual toil.

5. Use directories, not branches, for environments

This is the number one anti-pattern I encounter in the wild. Teams create a dev branch, a staging branch, and a prod branch, then cherry-pick or merge between them for promotions. It sounds logical, but it’s a trap. Branches diverge, cherry-picks get missed, configs meant for dev sneak into staging or prod, and before you know it, your environments have drifted, and you can’t tell what’s actually different between them.

A folder structure like environments/dev/, environments/staging/, environments/prod/ makes differences visible in a single diff command.
Moving a change from dev to prod is just copying or updating files across directories, reviewed via a standard pull request.
You’ll never accidentally skip a commit or introduce an environment-specific change where it doesn’t belong. No more cherry-pick roulette.

One thing to note: the commonly used rendered manifests pattern actually does use per-environment branches, but not in a human-managed way. In that pattern, config changes are still committed to main (using named directories as above), and an automated CI process pushes environment YAML to read-only environment branches, which ArgoCD reads from. If you want to try it, ArgoCD’s built-in source hydrator and Kargo Render can automate the rendering for you.

To keep things DRY, you can look to a tool like Kustomize, which lets you use a shared base environment and then apply per-environment patches. Your base directory holds the common manifests, and each environment directory contains only the differences. For example:

gitops-repo/
├── base/
│ ├── deployment.yaml
│ ├── service.yaml
│ └── kustomization.yaml
├── environments/
│ ├── dev/
│ │ ├── kustomization.yaml # replicas: 1, limits: 256Mi
│ │ └── patches/
│ ├── staging/
│ │ ├── kustomization.yaml # replicas: 2, limits: 512Mi
│ │ └── patches/
│ └── prod/
│ ├── kustomization.yaml # replicas: 3, limits: 1Gi
│ └── patches/

This minimizes duplication while keeping environment boundaries clear.

Pro Tip: If you’re working at the IaC layer, Pulumi stack configuration solves a similar problem: each stack (dev, staging, prod) has its own config file with environment-specific values, while the program logic stays shared. It’s the same principle of “one source, per-environment overrides” applied above the Kubernetes manifest level.

6. Validate before you merge

Catching errors after they’ve been applied to your cluster is expensive. Catching them in CI before the merge is cheap. Shift left as hard as you can.

Your GitOps CI pipeline should validate everything it can before the change reaches your cluster: YAML syntax, Kubernetes schema validation, policy compliance, dry-run rendering.

Tools like yamllint and kubeconform catch syntax errors and schema violations before they become runtime failures. (Note: kubeval, which you’ll see in older guides, is no longer maintained — its own README points to kubeconform as the replacement.)
Run helm template or kustomize build in CI to verify that your templates render without errors.
Use OPA/Conftest or Kyverno to enforce organizational policies (no privileged containers, required labels, resource limits set) before merge.

Here’s a minimal GitHub Actions workflow that validates your GitOps manifests on every PR:

name: Validate GitOps Manifests
on:
 pull_request:
 paths: ["environments/**"]

jobs:
 validate:
 runs-on: ubuntu-latest
 steps:
 - uses: actions/checkout@v4
 - name: Validate YAML schemas
 run: kubeconform --strict environments/**/*.yaml
 - name: Build and verify Kustomize output
 run: |
 for env in environments/*/; do
 kustomize build "$env" > /dev/null
 done
 - name: Diff against live cluster
 run: |
 kustomize build environments/staging/ | \
 kubectl diff -f - || true

Pro Tip: Add a kubectl diff step in your CI pipeline that shows exactly what would change in the cluster if the PR were merged. This gives reviewers a concrete view of the impact, not just the YAML diff.

7. Tag with commit SHAs, not “latest”

Using latest as your image tag in a GitOps workflow is playing fast and loose with your deployments. You have no idea what version is actually running, you can’t reliably roll back, and your Git history becomes meaningless because the same manifest could deploy completely different code depending on when it was synced.

Tag your container images with the Git commit SHA that produced them. This creates a direct link between your source code, your build, and your deployment.
Once a tag is pushed, it should never be overwritten. No re-tagging, no “oops let me push a fix to the same tag.”
Need to roll back? Just revert the commit that updated the image tag. The previous SHA still points to the exact same image it always did.

Pro Tip: Set up an admission controller or CI policy that rejects any manifest using latest or any other mutable tag. Make it impossible to deploy without a pinned version.

8. Automate drift detection and reconciliation

Drift is when your cluster’s actual state doesn’t match the desired state in Git. It happens when someone runs a manual kubectl command, when an autoscaler changes a replica count, or when a CRD controller modifies a resource behind your back.

The whole point of GitOps is continuous reconciliation. Your operator should constantly compare the live state against Git and bring things back in line.

Configure your GitOps tool to automatically revert manual changes, but build your exclusion lists (the set of fields and resources you intentionally skip during reconciliation) before you turn on auto-sync, not after. Controllers like Istio, cert-manager, and Crossplane legitimately modify resources, and auto-remediating those changes creates reconciliation loops that can destabilize a cluster. Start with a conservative exclusion list and tighten it over time.
Even with auto-remediation, you want to know when drift happens. Set up alerts. It might indicate a process problem or a team member who needs coaching.

Pro Tip: Be precise about what you reconcile. HPAs, VPAs, cert-manager annotations, and external-dns records all legitimately modify resources. Use ignore rules to exclude fields that are intentionally dynamic. In ArgoCD, that looks like this:

spec:
 ignoreDifferences:
 - group: apps
 kind: Deployment
 jsonPointers:
 - /spec/replicas
 - group: admissionregistration.k8s.io
 kind: MutatingWebhookConfiguration
 jqPathExpressions:
 - .webhooks[]?.clientConfig.caBundle

Also, don’t overlook resource ordering. ArgoCD sync waves and Flux’s dependsOn let you enforce that CRDs are applied before custom resources and namespaces before workloads. Getting ordering wrong is one of the most common causes of failed syncs in production.

9. Practice progressive delivery

Progressive delivery lets you gradually roll out changes, verify they’re working, and automatically roll back if something goes wrong.

GitOps and progressive delivery fit well together. Your Git repo describes the desired rollout strategy, and controllers in the cluster execute it.

Canary deployments: send a small percentage of traffic to the new version. If error rates spike, automatically roll back.
Blue-green deployments: run two identical environments, switch traffic once the new one is verified healthy.
Argo Rollouts extends Kubernetes with canary, blue-green, and analysis-driven rollout strategies that integrate with ArgoCD.
Flagger is the Flux ecosystem equivalent, with automated canary analysis, A/B testing, and blue-green deployments using service mesh or ingress controllers.
Kargo takes it a step further by orchestrating promotions across multiple stages and environments. Instead of manually wiring up pipelines to move changes from dev to staging to prod, Kargo automates the entire promotion workflow on top of ArgoCD. I did a deep dive on GitOps promotion tools and why they belong in your toolkit:

Pro Tip: Combine progressive delivery with automated analysis. Tools like Argo Rollouts and Kargo can query Prometheus metrics during a canary and automatically promote or roll back based on error rates, latency percentiles, or custom metrics.

10. Policy-as-code: your automated guardrails

Reviews are great, but humans make mistakes. Policy-as-code adds an automated layer that enforces organizational rules before a change can land in your cluster. Think of it as guardrails on a mountain road: they don’t slow you down, they just keep you from going over the edge.

OPA/Gatekeeper lets you define policies in Rego that the admission controller enforces at deploy time (e.g., no containers running as root, all deployments must have resource limits).
Kyverno is a Kubernetes-native policy engine that uses familiar YAML to define and enforce policies, including mutation and generation.
Starting with Kubernetes 1.26+, you can write CEL-based ValidatingAdmissionPolicies natively without any external tooling. A lightweight option if you don’t need the full feature set of OPA or Kyverno.
Pulumi CrossGuard lets you write policy packs in TypeScript or Python that validate infrastructure before it’s deployed. If your IaC and GitOps layers are bridged (see the next section), CrossGuard can enforce rules across both.
Run policy checks in your CI pipeline so violations are caught during the PR, not after deployment.

Pro Tip: Start with a small set of high-impact policies (required labels, no privileged containers, resource limits) and expand over time. Trying to enforce 50 policies on Day 1 will create so much friction that teams will revolt.

Try infrastructure as code with Pulumi

Define your cloud resources in TypeScript, Python, or Go, then pass stack outputs straight into your GitOps manifests so both layers stay in sync.

Get started

11. Bridge your IaC and GitOps (don’t choose one)

I hear this all the time: “We use Terraform for infrastructure and ArgoCD for deployments, and they live in completely separate worlds.” Sound familiar? Most teams treat IaC and GitOps as independent workflows, but they’re really two halves of the same story.

IaC handles “Day 0,” creating the cloud resources your cluster depends on: VPCs, the Kubernetes cluster itself, IAM roles, databases, DNS zones. GitOps handles “Day 2,” managing what runs inside the cluster: applications, addons, configurations. The gap between them is where things get messy. Your Helm charts need IAM role ARNs. Your GitOps-deployed addons need to know the cluster’s OIDC provider endpoint. That metadata has to flow from the IaC layer to the GitOps layer somehow.

The gitops-bridge pattern solves this cleanly: IaC provisions cloud resources and writes metadata (role ARNs, account IDs, endpoints) into Kubernetes resources (like ConfigMaps or Secrets) that your GitOps tool can consume directly.

Keep each tool in its lane. Don’t force Terraform’s Helm provider to fight with ArgoCD over resource ownership.
Pulumi lets you define your cloud infrastructure in real programming languages (TypeScript, Python, Go), which makes it natural to compute and pass metadata to your GitOps layer. You can use stack outputs to expose values like role ARNs and cluster endpoints, then feed them into your GitOps manifests. The Pulumi Kubernetes Operator even lets ArgoCD reconcile Pulumi stacks via GitOps.
Statsig went from “1-2 devs clicking around cloud consoles with SEVs left and right” to fully self-service by using Pulumi to generate manifests and ArgoCD to deploy them.
As you grow beyond a handful of clusters, think multi-cluster from the start. Patterns like ArgoCD ApplicationSets with cluster generators and Flux’s Kustomization targeting become important for fleet management. The gitops-bridge pattern works well here because your IaC layer can register new clusters and write their metadata into Kubernetes resources, which ApplicationSets automatically pick up.

Here’s how the gitops-bridge looks in practice. Pulumi provisions cloud resources and writes metadata into a ConfigMap that ArgoCD consumes:

import * as k8s from "@pulumi/kubernetes";

const clusterMetadata = new k8s.core.v1.ConfigMap("cluster-metadata", {
 metadata: {
 name: "cluster-metadata",
 namespace: "argocd",
 labels: {
 "gitops-bridge": "true",
 },
 },
 data: {
 aws_account_id: "123456789012",
 cluster_name: "prod-us-east-1",
 iam_role_arn: "arn:aws:iam::123456789012:role/app-role",
 oidc_provider: "oidc.eks.us-east-1.amazonaws.com/id/EXAMPLE",
 },
});

import pulumi_kubernetes as k8s

cluster_metadata = k8s.core.v1.ConfigMap(
 "cluster-metadata",
 metadata=k8s.meta.v1.ObjectMetaArgs(
 name="cluster-metadata",
 namespace="argocd",
 labels={
 "gitops-bridge": "true",
 },
 ),
 data={
 "aws_account_id": "123456789012",
 "cluster_name": "prod-us-east-1",
 "iam_role_arn": "arn:aws:iam::123456789012:role/app-role",
 "oidc_provider": "oidc.eks.us-east-1.amazonaws.com/id/EXAMPLE",
 },
)

Your ArgoCD Applications or Helm values can then reference this ConfigMap, closing the loop between IaC and GitOps without hardcoding cloud-specific values.

Pro Tip: If you’re using Terraform today, the gitops-bridge pattern works there too. But if you’re tired of wrangling HCL and want type-safe infrastructure code with real programming constructs, give Pulumi a look. The bridge from IaC to GitOps becomes much more natural when both sides speak a real programming language.

12. Be pragmatic, not dogmatic

Michael Crenshaw from Intuit gave a talk titled “How GitOps Should I Be?” at a CNCF conference. Intuit runs 45 ArgoCD instances managing 20,000 applications across 200 clusters. The approach they took: “Be strategic, be pragmatic. GitOps most of the time.”

One of the largest GitOps adopters in the world explicitly deviates from pure GitOps when the situation calls for it. Dogmatic adherence to principles that don’t serve you is just another form of technical debt.

Intuit found that GitHub’s SLA of 20 minutes meant they couldn’t rely on GitOps for region evacuation, so they wrote a kubectl cron job that bypasses Git for failover. Pragmatic.
Massive Jenkins ecosystems don’t get rewritten overnight. Intuit’s compromise: the last step in the Jenkins pipeline calls argocd app sync. It’s push-based, yes, but the source of truth stays in Git. The Jenkins pipeline doesn’t own the desired state; it just tells ArgoCD “go reconcile now” instead of waiting for the next poll. It’s a stepping stone while teams migrate off Jenkins, not a permanent architecture.
Continuous reconciliation with self-heal (auto-reverting manual changes) is where most teams deviate. In practice, many teams start with automated drift detection and alerting (so you know when someone runs a manual kubectl command) but leave auto-remediation off until they’ve built confidence in their exclusion lists. That’s a reasonable middle ground.

Pro Tip: Document your intentional deviations. Create an ADR (Architecture Decision Record) for each case where you’ve chosen not to follow pure GitOps, explaining why and what the plan is to close the gap (if ever). This turns exceptions into conscious decisions rather than accidental shortcuts.

Final thoughts

GitOps isn’t a silver bullet, but it’s one of the best approaches we have for managing Kubernetes at scale. The practices in this list aren’t theoretical. They’re what I’ve learned from running GitOps in production, watching talks from teams at Intuit and Statsig, and making every mistake in the book at least once.

If I had to boil it all down to one sentence: treat your GitOps repo like production code, because it is.

Start with the basics (Git as source of truth, declarative config, pull-based delivery) and layer on the advanced practices as your team matures. Don’t try to implement all 12 on Day 1. Pick the three that address your biggest pain points and iterate from there.

And remember, even the largest GitOps adopters in the world aren’t doing it perfectly. They’re being pragmatic. You should be too.

How Pulumi fits into your GitOps workflow

Several of the practices in this post touch on problems Pulumi is built to solve. If you’re bridging IaC and GitOps, managing environment-specific configuration, or enforcing policy across both layers, here’s where to start:

Pulumi IaC: Define cloud infrastructure in TypeScript, Python, or Go and use stack outputs to feed metadata into your GitOps manifests.
Pulumi ESC: Manage secrets and configuration across environments with fine-grained access controls, integrated with Kubernetes via the External Secrets Operator or the Secret Store CSI Driver.
Pulumi Kubernetes Operator: Let ArgoCD reconcile Pulumi stacks via GitOps, closing the loop between your IaC and Day 2 workflows.

Kubernetes Best Practices I Wish I Had Known Before

Engin Diri — Mon, 20 Jan 2025 00:00:00 +0000

Kubernetes best practices are the production-tested rules that keep clusters reliable, secure, and affordable: set resource requests and limits on every container, isolate workloads with namespaces and NetworkPolicies, enforce least-privilege RBAC, automate health checks, ship via GitOps, validate every change with policy-as-code, generate an SBOM for every image, and manage Kubernetes itself with infrastructure as code instead of hand-rolled YAML. The 20 practices below cover what production teams actually do in 2026 — not what tutorials suggest.

TL;DR: 20 Kubernetes best practices for 2026

Set resource requests and limits on every container.
Use namespaces plus ResourceQuota and LimitRange for isolation.
Run one container per Pod unless you genuinely need a sidecar.
Manage manifests with Helm, Kustomize, or Pulumi — never raw copies.
Use the Gateway API (ingress-nginx is retired) for north-south traffic.
Configure liveness, readiness, and startup probes for every workload.
Enforce RBAC with the principle of least privilege from day one.
Apply Pod Security Admission at the restricted level by default.
Lock down east-west traffic with NetworkPolicy (default-deny).
Store secrets in an external secret manager (Pulumi ESC, Vault, AWS Secrets Manager) — not as plain Kubernetes Secrets.
Autoscale with HPA, VPA, and Cluster Autoscaler / Karpenter.
Monitor with Prometheus, Grafana, and OpenTelemetry; alert before users do.
Deploy with GitOps (Argo CD or Flux) — Git is the only source of truth.
Gate every change with policy-as-code (Pulumi CrossGuard, OPA/Gatekeeper, Kyverno).
Generate and verify an SBOM for every image; sign with Sigstore/cosign.
Manage cost with FinOps practices: right-size, schedule, and chargeback by namespace.
Keep clusters and add-ons patched on a documented cadence.
Use labels and annotations consistently for ownership, cost, and SLO tracking.
Separate dev, staging, and prod clusters — or virtualize with vCluster.
Treat clusters as cattle, not pets: rebuild from code, never edit live.

Last updated April 30, 2026 — added GitOps, policy-as-code, SBOM/supply-chain, autoscaling, NetworkPolicy, and FinOps sections; added an anti-pattern table; restructured headings around the questions teams actually ask.

Going to KubeCon Europe 2026? Tame Kubernetes complexity with code.

Pulumi demos, platform engineering talks, AI-powered Neo in action, and yes, free plushies. Booth 784.

The ‘Kubernetes is easy’-iceberg meme is a classic example of how Kubernetes can be deceivingly complex

What are Kubernetes best practices?

Kubernetes best practices are the operational patterns that keep workloads available, secure, observable, and cost-efficient on a Kubernetes cluster. They span four areas: workload configuration (resources, probes, images), security (RBAC, NetworkPolicy, Pod Security, secrets, supply chain), delivery (GitOps, policy-as-code, IaC), and platform operations (autoscaling, monitoring, upgrades, FinOps). The rest of this post explains each one with a copy-pasteable rule, a definition of the underlying Kubernetes object, and a common anti-pattern to avoid.

Common Kubernetes anti-patterns vs. the correct approach

Common mistake	What goes wrong	Correct approach
No resource requests or limits	Noisy-neighbor evictions, OOMKills, unschedulable Pods	Set requests for every container; use VPA recommendations to tune
Everything in the `default` namespace	No isolation, no quota, no RBAC boundary	One namespace per team or environment + ResourceQuota + RBAC
`cluster-admin` for service accounts	Full-cluster blast radius on compromise	Per-namespace Role + RoleBinding (least privilege)
Plain Kubernetes `Secret` objects	Base64 ≠ encryption; secrets leak via etcd, kubectl, audit logs	External Secrets Operator + Pulumi ESC, Vault, or AWS Secrets Manager
Open east-west traffic	One compromised Pod can reach every service	Default-deny NetworkPolicy; allow only required flows
`kubectl apply -f` from a laptop	No history, no review, no rollback	GitOps with Argo CD or Flux; Git is the source of truth
Manual `latest` image tags	Non-reproducible deploys, supply-chain risk	Pinned digests (`@sha256:...`) with cosign verification
Ingress-nginx in 2026	Project retired March 2026 — no security fixes	Migrate to the Gateway API (Envoy Gateway, Istio, Kgateway)
Hand-rolled YAML in dozens of repos	Drift, copy-paste bugs, no testing	Pulumi, Helm, or Kustomize with reviews
One shared `prod` cluster for everything	Blast radius spans every team	Separate clusters or vCluster virtual clusters

1. How do you set Kubernetes resource requests and limits?

What are resource requests and limits? A resource request is the minimum CPU and memory the kube-scheduler guarantees a container; a resource limit is the maximum the kubelet allows it to consume before throttling CPU or OOMKilling memory. Pods without requests are scheduled blind, so a noisy neighbor can evict your workload at any time.

The rule: every container in production sets a request for both CPU and memory, and a memory limit. CPU limits are optional and frequently counter-productive — they cause throttling spikes that look like outages.

Start at 100–200m CPU and 128–512Mi memory for a typical web service.
Tune from real data using Vertical Pod Autoscaler with updateMode: Off (recommendation-only) or Prometheus histograms.
Use a per-namespace LimitRange so teams cannot ship a container with no requests at all.

2. How should you structure Kubernetes namespaces?

What is a Kubernetes namespace? A namespace is a logical partition of cluster resources. It scopes object names, RBAC bindings, NetworkPolicies, and quotas — but does not provide hard isolation by itself.

The rule: never deploy production workloads to default. Map namespaces to ownership boundaries — usually one per team, environment, or tenant — and pair every namespace with three things:

A ResourceQuota to cap aggregate CPU, memory, and object counts.
A LimitRange to enforce per-container defaults.
A RoleBinding that grants only the permissions that team needs.

This is the smallest unit that gives you predictable cost, blast-radius containment, and an RBAC perimeter.

3. Should you run multiple containers in one Pod?

What is a Pod? A Pod is the smallest deployable unit in Kubernetes — one or more containers that share a network namespace, IPC, and storage volumes, and that are scheduled and scaled as a single unit.

The rule: one container per Pod by default. Add a second container only when it must share the Pod’s network or volumes — for example, a service-mesh sidecar (Istio/Linkerd), a logging shipper, or an init container that prepares state before the main process starts. Multi-container Pods couple lifecycles and scaling, so reach for them deliberately, not by default.

You might also like:

4. Should you use Helm, Kustomize, or Pulumi for Kubernetes manifests?

The rule: never copy-paste raw YAML across services. Pick one of these:

Helm — Go-templated charts; the de-facto distribution format for off-the-shelf software (Prometheus, cert-manager, Argo CD).
Kustomize — overlay-based, no templating; ships in kubectl. Best for “same app, different environment” diffs.
Pulumi — real programming languages (TypeScript, Python, Go, Java, .NET) for Kubernetes plus the cloud resources around it (EKS, GKE, AKS, DNS, IAM). Type checking, tests, and the Pulumi Kubernetes Operator for GitOps-style reconciliation.

If you are already managing cloud infrastructure with code, extending the same Pulumi program to Kubernetes is the smallest cognitive jump — the cluster, its IAM, its DNS, and its workloads live in one stack and one review.

5. How should you handle ingress and networking on Kubernetes in 2026?

What is the Gateway API? The Gateway API is the successor to the legacy Ingress resource. It splits responsibility between infrastructure (GatewayClass, Gateway) and application (HTTPRoute, GRPCRoute, TLSRoute) teams, and it is the SIG-Network direction for north-south traffic.

The rule: new clusters should standardize on the Gateway API. The widely deployed ingress-nginx controller retired in March 2026, and remaining deployments should migrate to Envoy Gateway, Istio, Linkerd Gateway, or Kgateway.

Terminate TLS at the gateway and automate certificates with cert-manager.
Use path-based and host-based routing instead of one gateway per service.
Protect every gateway with a WAF (managed cloud WAF or Coraza on Envoy).

ingress-nginx is retiring in March 2026.

Here is a guide on How to Move to the Gateway API: post ingress-nginx Retirement.

6. What’s the difference between liveness, readiness, and startup probes?

Probes tell the kubelet whether to restart a container, route traffic to it, or wait for it. Configure them per workload:

Liveness probe — restarts a hung container. Use sparingly; a bad liveness probe is a self-inflicted DoS.
Readiness probe — gates traffic from Services and Gateway routes. Use for every network-facing workload.
Startup probe — gives slow-booting apps (JVM, ML models) time to come up before liveness checks begin. Use whenever cold-start exceeds 10 seconds.

See the upstream guide for HTTP, TCP, and exec probe syntax. Tune failureThreshold, periodSeconds, and timeoutSeconds based on real latency, not defaults.

7. How do you secure a Kubernetes cluster?

Cluster security is layered: identity, workload posture, network, and secrets. Get all four right.

What is RBAC in Kubernetes?

RBAC (Role-Based Access Control) is the Kubernetes authorization model that maps subjects (users, groups, ServiceAccounts) to verbs (get, list, create, delete) on resources, scoped to a namespace (Role) or the whole cluster (ClusterRole).

Grant least privilege — never bind humans or workloads to cluster-admin.
Prefer per-namespace Role + RoleBinding over ClusterRoleBinding.
Audit with kubectl auth can-i --as=... and tools like rbac-lookup.

What is Pod Security Admission?

Pod Security Admission is the built-in admission controller that enforces the three Pod Security Standards (privileged, baseline, restricted) at the namespace level.

Label every namespace pod-security.kubernetes.io/enforce=restricted by default.
Drop NET_RAW and all capabilities; run as non-root with a read-only root filesystem.
Use seccompProfile: RuntimeDefault and a tight securityContext for every container.

What is a NetworkPolicy?

A NetworkPolicy is a Kubernetes object that defines allowed ingress and egress traffic for selected Pods at the IP/port level. Without one, every Pod can reach every other Pod by default.

Apply a default-deny ingress and egress policy in every namespace.
Allow only the flows your service actually needs (DNS, the database, the upstream API).
For richer L7 controls (mTLS, JWT auth), layer a service mesh on top.

How should you manage Kubernetes secrets?

Built-in Secret objects are base64-encoded, not encrypted. Treat them as a transport, not as storage.

Store the source of truth in Pulumi ESC, HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, or Azure Key Vault.
Project secrets into Pods with the External Secrets Operator or the Secret Store CSI Driver.
Enable etcd encryption-at-rest with a KMS provider so leaked etcd snapshots stay opaque.
Rotate automatically; never commit secrets to Git, even encrypted.

8. How do you set up Kubernetes observability?

Observability rests on the three pillars: metrics, logs, and traces. The 2026 default stack is OpenTelemetry-first.

Metrics — Prometheus (or a Prometheus-compatible TSDB like Mimir, Thanos, VictoriaMetrics) plus Grafana dashboards.
Logs — Fluent Bit or the OpenTelemetry Collector shipping structured JSON to Loki, Elastic/OpenSearch, or a managed service. Never depend on kubectl logs for incidents — Pods are ephemeral.
Traces — OpenTelemetry SDKs in your apps emitting to Tempo, Jaeger, or a vendor backend. Correlate trace IDs with logs.
Alerts — page on SLO burn rate, not on every crashed Pod. Burn-rate alerting catches real user impact without firing on every restart.

9. How do you automate Kubernetes deployments with GitOps?

What is GitOps? GitOps is a deployment model where the desired cluster state lives in a Git repository and a controller continuously reconciles the live cluster to match. Changes happen through pull requests, not kubectl apply.

The rule: every cluster has a controller — Argo CD, Flux, or the Pulumi Kubernetes Operator — that owns reconciliation. Humans never kubectl apply to production.

Use app-of-apps (Argo) or Kustomization trees (Flux) to bootstrap whole clusters from a single root.
Keep manifests, Helm values, and Pulumi stack references in Git, with environment promotion via PR.
Combine GitOps with Pulumi Deployments so cloud infrastructure (VPCs, EKS clusters, IAM, DNS) and the workloads on top promote through the same review pipeline. See improving GitOps with the Pulumi Operator for a worked example.
Configure automated rollback on health-check failure, and drift detection alerts when someone edits live state.

10. How often should you upgrade Kubernetes?

Kubernetes ships a minor release roughly every four months and supports each one for about 14 months. Treat upgrades as routine maintenance, not a project.

Stay within two minor versions of upstream — older versions miss CVE patches.
Test upgrades in a non-prod cluster, run a conformance test, then promote.
Back up etcd before control-plane changes; managed services (EKS, GKE, AKS) handle most of this for you.
Upgrade add-ons (CNI, CSI, Gateway controller, cert-manager, metrics-server) on the same cadence — pin versions in code so the upgrade is auditable.
Watch the Kubernetes deprecation guide and run pluto or kubent to find deprecated APIs before they break.

11. How should you label and annotate Kubernetes resources?

Labels and annotations are the metadata layer that everything — Services, NetworkPolicies, monitoring, cost tools, GitOps — keys off of. Consistency matters more than cleverness.

Adopt the recommended labels: app.kubernetes.io/name, app.kubernetes.io/instance, app.kubernetes.io/version, app.kubernetes.io/component, app.kubernetes.io/part-of, app.kubernetes.io/managed-by.
Add organization-specific labels for owner, cost-center, environment, and SLO tier — your FinOps and on-call processes will need them.
Reserve annotations for non-identifying metadata: build SHA, change-ticket, last-deployed timestamp, ingress controller hints.

Manage Kubernetes with code

Provision your clusters, their cloud resources, and the workloads on them from one Pulumi program, and use stacks to promote changes from dev to production.

Get started

12. How should you separate Kubernetes environments?

The rule: production is its own cluster. Dev and staging can share — sometimes. Mixing prod with anything else couples blast radius, upgrade cadence, and quotas.

Separate clusters per environment is the safe default. With managed control planes (EKS, GKE, AKS) the cost is small.
Virtual clusters with vCluster give each team a Kubernetes API of their own without the operational cost of a real cluster.
Namespace-only segregation can work for dev/staging if you enforce strict NetworkPolicy, RBAC, and ResourceQuota boundaries.
Provision all of them from the same Pulumi program with different stacks — dev, staging, prod — so promotion is a config change, not a copy.

13. How do you optimize Kubernetes container images?

Start from a distroless (gcr.io/distroless) or chainguard/static base. Alpine is fine if you accept musl.
Build with multi-stage Dockerfiles so build tools never ship in the final image.
Pin the base image by digest (@sha256:...), not by tag — latest and 1.21 move under you.
Scan every image with Trivy, Grype, or your registry’s built-in scanner. Block builds on Critical/High CVEs.
Smaller images mean faster pulls, faster autoscaling, and a smaller attack surface.

14. What’s a reliable Kubernetes logging strategy?

Centralize to Loki, Elastic/OpenSearch, or a managed service via Fluent Bit or the OpenTelemetry Collector running as a DaemonSet.
Structure logs as JSON with consistent field names (trace_id, span_id, service, level).
Retain by tier — 7 days hot, 30–90 days warm, archive to object storage for compliance.
Redact secrets and PII at the collector, not after the fact.

You might also like:

15. How should you autoscale workloads on Kubernetes?

What is the Horizontal Pod Autoscaler? The HPA scales the replica count of a Deployment or StatefulSet up and down based on CPU, memory, or custom/external metrics.

What is the Vertical Pod Autoscaler? The VPA adjusts a Pod’s CPU and memory requests over time based on observed usage. Run it with updateMode: Off (recommendation-only) in production and apply the recommendations through your IaC tool.

What are Cluster Autoscaler and Karpenter? Cluster Autoscaler adds and removes nodes to match Pod demand. Karpenter is a faster, group-less node provisioner now widely used on EKS and gaining traction on other clouds.

The rule: every production workload has an HPA. Every cluster has a node-level autoscaler. Set sensible min/max replicas, define a PodDisruptionBudget, and gate scale-down behavior so traffic spikes don’t pile up at restart time. Pair with KEDA for event-driven scaling (queues, Kafka lag, scheduled scale).

16. How do you enforce policy-as-code on Kubernetes?

What is policy-as-code? Policy-as-code expresses governance rules — security baselines, tagging, region restrictions, cost guardrails — as code that runs in CI and at admission time, blocking non-compliant changes before they reach the cluster.

The rule: every change goes through at least one policy engine.

Pre-deploy — validate Pulumi or Helm output with Pulumi CrossGuard or Conftest in CI.
Admission-time — install OPA Gatekeeper or Kyverno and require, for example, signed images, a team label, and a non-root securityContext on every Pod.
Continuous — scan running clusters with Pulumi Insights or kubescape for drift from policy.

See the benefits of policy-as-code and enforcing policy-as-code on discovered resources for end-to-end examples.

17. How do you secure the Kubernetes software supply chain?

What is an SBOM? A Software Bill of Materials is a machine-readable inventory of every component in a container image — packages, versions, licenses, hashes — typically in SPDX or CycloneDX format.

The rule: every image you run in production has a verified SBOM and a verified signature.

Generate SBOMs at build time with Syft or docker sbom.
Sign images and SBOMs with Sigstore/cosign keylessly via GitHub OIDC.
Verify signatures at admission with Kyverno or Connaisseur.
Generate provenance attestations (SLSA) so you can prove which build pipeline produced a given image.
Pin base images by digest and rebuild on a cadence so CVE patches actually land.

This is what regulators and customers ask for in 2026 — Executive Order 14028, the EU Cyber Resilience Act, and most enterprise procurement checklists all require it.

18. How do you manage Kubernetes cost (FinOps)?

The rule: Kubernetes cost is a labeling and right-sizing problem before it is a discount problem.

Right-size with VPA recommendations, Goldilocks, or your APM’s recommendation engine. Most workloads request more CPU than they actually use; right-sizing is usually the largest single saving.
Schedule non-prod clusters off overnight and on weekends.
Spot/preemptible nodes for fault-tolerant workloads; use Karpenter’s consolidation to compact bin-packing.
Show back / charge back by namespace and label using OpenCost or Kubecost. Without per-team attribution, no team owns cost.
Provision the underlying cluster with IaC — see Pulumi for Kubernetes — so node types, autoscaling limits, and reserved capacity are reviewable artifacts. Pulumi’s hidden-cost analysis walks through where the money actually goes.

19. Why should you treat Kubernetes clusters as cattle, not pets?

The old “cattle, not pets” adage applies more strictly to Kubernetes than to VMs. Manual edits to a live cluster will be reconciled away, drift between environments, or vanish on the next Pod restart.

Fix problems in code — YAML, Helm chart, or Pulumi program — and let the controller redeploy.
If you can’t redeploy a cluster from Git in under an hour, you have a pet.
Use ephemeral preview environments for PRs; use blue/green or progressive delivery (Argo Rollouts, Flagger) for production cutovers.

20. Why use Pulumi to manage Kubernetes?

Native YAML scales until your team doesn’t. Once you have more than a handful of services, dependencies between cloud resources and Kubernetes objects, or more than one cluster, infrastructure as code in a real programming language wins on every axis.

Real languages — TypeScript, Python, Go, Java, .NET — for type-safe, testable Kubernetes infrastructure. Loops, conditionals, and unit tests instead of templating.
One stack, full topology — manage the cloud (EKS/GKE/AKS, VPC, IAM, DNS) and the workloads in it together. See easily create and manage AWS EKS clusters with Pulumi.
Reusable components — abstract platform patterns into Pulumi packages other teams import instead of copy-pasting.
GitOps reconciliation — the Pulumi Kubernetes Operator reconciles a cluster to a Pulumi stack on every Git push.
Policy-as-code built in — CrossGuard blocks non-compliant changes before pulumi up.
Secrets done right — Pulumi ESC federates secrets and configuration across stacks, environments, and Kubernetes clusters.

For a deeper comparison of hand-written YAML, Terraform, and Pulumi for Kubernetes, see YAML, Terraform, Pulumi: what’s the smart choice for deployment automation with Kubernetes, and the beyond YAML write-up on where Kubernetes automation is heading in 2026.

See how Pulumi helps you manage Kubernetes Services and Deployments with code instead of YAML—bringing structure, reuse, and type safety to your configs.

By adopting Pulumi, you can avoid the complexity of juggling endless YAML files and gain a more streamlined, maintainable workflow for your Kubernetes infrastructure.

Final thoughts

Kubernetes rewards discipline. The 20 practices above — resource hygiene, namespaced isolation, RBAC, NetworkPolicy, Pod Security, external secrets, probes, the Gateway API, observability, GitOps, policy-as-code, signed images and SBOMs, autoscaling, FinOps, multi-cluster strategy, and IaC — are the dial settings that separate clusters that page their owners every week from clusters that don’t.

Pick the three weakest spots in your environment and fix those first. Then promote the fixes through code, not through kubectl.

Want to learn how to put these practices into action? Meet us at KubeCon Europe 2026 (Booth 784) or register for our upcoming Zero to Production in Kubernetes workshop.

Try Pulumi for Free

Your Perfect Infrastructure May Not Be So Perfect

Simen A. W. Olsen — Mon, 16 Dec 2024 10:43:07 +0000

Guest Article: Simen A. W. Olsen from Bjerk, is here to share his lessons learned on why designing the perfect architecture for your future needs might be a mistake

I remember standing in front of our engineering team in 2018, proudly presenting what I believed was the future-proof architectural design for our new distributed system. The diagrams were immaculate, the technology choices were cutting-edge, and the scalability patterns were ready for any possible future scenario.

I was basically the Leonardo da Vinci of system design… if Leonardo had been really into Kubernetes and had a concerning addiction to coffee. But six months later, that “future-proof” architecture had become a constraint rather than an enabler, and my masterpiece was looking more like a finger painting done by a caffeinated raccoon.

This experience taught me something crucial: trying to build the perfect system that anticipates every future need is often worse than creating a system designed to change quickly. It’s like trying to predict what your kid will want to be when they grow up and pre-buying all the necessary equipment. Congrats, you now own a space suit, a stethoscope, and a dragon costume — and they decided to become a software engineer anyway.

The Over-Planning

Many teams fall into a common trap: they try to design systems that anticipate every possible future requirement. This happens even in agile teams, where we convince ourselves we need to “get the architecture right” before we can start iterating. You know, because nothing says “agile” like spending three months in a room drawing boxes and arrows while muttering “microservices” under your breath like it’s a magic spell.

In 2008, Netflix faced a choice: build the perfect data center that could handle all their anticipated future needs, or move to the cloud with a simpler architecture that could evolve. They chose the latter, focusing on making their system easy to change rather than trying to make it perfect. Smart move — unlike those my past self made who probably would’ve insisted on building a data center capable of streaming to Mars, just in case Elon asked nicely.

You might also like:

The Core Principles of Change-Ready Architecture

Through both failures and successes, I’ve identified three principles that define truly adaptable architecture:

Embrace simplicity. I think it makes sense to start with the simplest architecture that could possibly work for your current needs. Complexity should be earned, not presumed. If your architecture diagram looks like a plate of spaghetti that’s been hit by lightning, you might be doing it wrong.
Make change cheap. Instead of trying to avoid change, make it inexpensive. This means investing in automated testing, continuous deployment, and monitoring. When change is cheap, you don’t need to fear it.
Learn through action. Rather than trying to predict the future, build mechanisms that help you learn quickly about real needs. It includes feature toggles (Protip: Try Unleash.), A/B testing, and robust monitoring of how your system is actually being used. You know, actual data, not just what that one loud guy in planning insists will definitely happen.

The rise of AI and machine learning systems has made one thing clear: we can’t predict how our systems will need to evolve. The most successful teams aren’t those that try to build the perfect AI architecture upfront, but those that can rapidly experiment and adapt their systems based on real-world feedback.

The biggest pushback I hear is, “But what if we need to scale?” or “What about future requirements?” These fears often drive teams to over-architect their solutions. But here’s the reality: the cost of changing a simple system is usually lower than the cost of maintaining an over-engineered one. The key is understanding that good architecture isn’t about predicting the future — it’s about making future changes as painless as possible.

Conclusion

That over-engineered system I was so proud of in 2018? Its most significant flaw wasn’t in what it got wrong about the future — it was that it tried too hard to be right about the future in the first place. It’s like bringing a fully packed suitcase to a first date. Today, I know that the best architecture isn’t one that anticipates every need, but one that makes it easy to respond to needs as they emerge.

The next time you’re tempted to design for every possible future scenario, remember: the goal isn’t to build a perfect system, but to build one that’s perfectly easy to change. And if someone tells you they’ve designed the perfect future-proof architecture, they’re either lying, or they’ve discovered time travel — and in that case, they should be sharing lottery numbers, not system designs.

You might also like:

IaC Best Practices: Summarizing Key Learnings

Scott Lowe — Mon, 08 Apr 2024 00:00:00 +0000

Welcome to the final post in our series of articles focused on Infrastructure as Code best practices. In this post, we’ll wrap up our recommendations for IaC with Pulumi, summarizing the best practices from previous posts as well as highlighting some areas of potential future growth for the team at Zephyr Archaeotech Emporium—the fictional company at the center of the story throughout this series.

Recapping IaC Best Practices

As this series has progressed, we’ve been showing you how Zephyr’s use of Pulumi has changed in response to how the organization, the team, and the application has changed. In case you haven’t been following along—or in case you need a refresher—here’s a brief summary of what’s happened in the series:

Zephyr started with a single project in a single repository with two stacks, one for production and one for testing. (Details are available in the first blog post in the series.)
Zephyr quickly added per-developer stacks as a way to enhance developer productivity. (You can read about per-developer stacks in the second post in the series.)
As Zephyr continued to grow, their single Pulumi project grew into three different Pulumi projects: one for base infrastructure, one for their Kubernetes platform, and one for the online store application. (The third post in the series describes the reasoning for adopting multiple projects.)
Accompanying the switch to multiple projects, Zephyr implemented stack references (to pass information between stacks in different projects) and applied role-based access control (RBAC) to the stacks in Pulumi Cloud. (Refer back to the fourth post and the fifth post in the series for more details on each of these areas.)
Zephyr realized a need to split out their data layer, going from three projects to four projects, and along the way saw an opportunity to add a higher level of orchestration using the Pulumi Automation API. (Refer back to our sixth post for more details on Zephyr’s initial use of Automation API.)

Since the last blog post, the Zephyr team has continued to explore Automation API, and has added an Automation API program that allows their developers to deploy the Zephyr online store directly from the associated GitHub repositories—developers don’t even need to clone down the repositories first.

To see the latest Automation API programs for Zephyr, refer to the zephyr-automation GitHub repository.

Areas of future growth for Zephyr

We’ve stated before that best practices are always point-in-time recommendations based on the requirements of the organization/team/individual using Pulumi. What this means, in practice, is that there is almost always room for growth and evolution. Organizations, teams, and individuals rarely remain static, and therefore their use of Pulumi is also unlikely to remain entirely static.

So, what are some potential areas of future growth for Zephyr?

Splitting the deployment of the microservices: We alluded to this in the previous post. Right now, all the microservices that comprise the online store are deployed together. What if one particular team needs to deploy only their service? The current approach can certainly accommodate that—Pulumi is declarative and will only change what needs to be changed—but what’s better for the Zephyr team? Does Zephyr need to optimize for allowing service teams to move independently? Is it about reducing blast radius, so that undesired changes aren’t introduced to another service? These are the sorts of considerations that drive this decision (and the third post in our series discusses structuring Pulumi projects in more detail).
Using component resources: Currently, the Zephyr team is using the standard API objects supplied by the Pulumi provider. For example, the various microservices for the Zephyr online store are defined using standard Kubernetes objects, like this:
```
const assetsNs = new k8s.core.v1.Namespace("assets-ns", {
 metadata: ...})

const assetsService = new k8s.core.v1.Service("assets-service", {
 metadata: ...})

const assetsDeployment = new k8s.apps.v1.Deployment("assets-deployment", {
 metadata: ...})
```
Component resources would allow the Zephyr team to define a single logical resource that contains “child” resources. This would allow the Zephyr team to create a Microservice resource that incorporates the standard Kubernetes Namespace, Deployment, and Service objects. Deploying an instance of a Microservice resource deploys all the child objects defined in the component resource. This can simplify the Pulumi code through further use of DRY (Don’t Repeat Yourself).
Making use of Pulumi Deployments: Zephyr’s CI/CD pipelines are not a topic we’ve discussed in the series. Given that Zephyr uses GitHub, they could use GitHub Actions (GHA); Pulumi has an action for GHA workflows. Zephyr could also use Pulumi Deployments, which offers an extensive set of features—OIDC support for dynamic credentials, Review Stacks (per-PR stacks), support for GitHub Enterprise, and GitOps-style workflows, to name a few. Support for Pulumi Deployments is also baked into Automation API, meaning that Zephyr could extend their existing Automation API code to incorporate Pulumi Deployments.
Addressing secrets and configuration management: We’ve also not discussed how Zephyr could manage their secrets and configuration data. Pulumi ESC is a good option here, offering seamless integration with Pulumi for infrastructure as code while also supporting a variety of other DevOps tools and use cases.

Even though it’s working well, clearly there’s a lot of room for the Zephyr team to continue to enhance and improve their infrastructure as code implementation!

Summarizing Best Practices

Throughout the series, we’ve provided guidelines on how to best utilize Pulumi. While we recommend reading the entire series, in this section will “summarize our summaries” and provide some broad, overarching practices.

With regard to Pulumi projects, keep it as simple as possible, but no simpler. While tools like Automation API can help with orchestrating multiple projects, it remains true that in most cases a single project is simpler. Use multiple projects when necessary, but only when necessary.
Use stacks freely. Using multiple projects has a cost in complexity. Using multiple stacks, on the other hand, does not—so use them freely to model different environments, to provide per-developer test environments, to test changes before rolling them into production, or for any other use case where you need a separate, independent instance of the infrastructure defined in your Pulumi program.
Focus on the requirements. It can sometimes be too easy to focus on the technology instead of the requirements. There are a lot of very cool things you can do with Pulumi, but try not to get distracted from the reason you’re using infrastructure as code: repeatable, reliable, consistent deployments of your cloud infrastructure.
Don’t forget about security along the way. Make sure you are properly securing your infrastructure. Pulumi CrossGuard can help here by providing “policy as code” guardrails for cloud infrastructure. Pulumi Cloud’s RBAC functionality helps with ensuring the right users have the right permissions to the right sets of cloud infrastructure resources.

Reviewing the Code

You can view the code we’ve presented in this series in the following GitHub repositories:

The “zephyr-infra” repository contains the base infrastructure definitions.
The “zephyr-k8s” repository contains the Pulumi code to stand up the Kubernetes cluster onto which Zephyr deploys their online store application.
The “zephyr-data” repository manages Zephyr’s “data layer” for persistent data used by the online store application.
The “zephyr-app” repository is Zephyr’s online store. This repository has both the application code as well as the Pulumi code for deploying the application (see the “infra” directory).
In the “zephyr-automation” repository you’ll find Automation API programs used by Zephr to orchestrate deployments of the different projects involved in supporting the online store.

In each of these repositories, there are different branches that reflect the state of the code as of a particular blog post. The “main” branch reflects the final state of the code, as described in this wrap-up post.

Wrapping up

This series has used the story of a fictional company, Zephyr Archaeotech Emporium, to discuss some recommendations around how to best utilize Pulumi for solving their infrastructure as code needs. While this company is fictional, the benefits of Pulumi and infrastructure as code are real. We invite you to try for yourself—sign up for a free Pulumi Cloud account today!

You might also like:

Your Perfect Infrastructure May Not Be So Perfect

IaC Best Practices: Using Automation API

Scott Lowe — Wed, 26 Jul 2023 00:00:00 +0000

Welcome to the sixth post in our series of blog posts focused on Infrastructure as Code (IaC) best practices. So far in this series, you’ve seen how Zephyr Archaeotech Emporium—the fictional company at the center of this series—uses Pulumi to manage their online retail store. You read how Zephyr’s initial use of Pulumi changed to use short-lived per-developer stacks. Later, as Zephyr continued to grow, you saw how Zephyr restructured their Pulumi projects and stacks, incorporated Stack References, and used Pulumi Cloud’s role-based access control (RBAC) functionality to control access to their stacks. This post focuses on how Zephyr takes advantage of the Pulumi Automation API to bring an even greater level of orchestration to the stacks that represent their online store.

The ultimate goal of this series is to discuss best practices for using Pulumi to manage a fairly complex containerized application. These practices have unfolded organically, as a direct response to Zephyr’s evolving needs. The aim is to demonstrate that best practices are not set in stone, but are rather “point in time” recommendations that adapt as your company grows.

Managing Complexity in a Multi-Project Pulumi Setup

Over the course of this series, you’ve seen how Zephyr’s use of Pulumi has grown and changed. For the sake of completeness, here’s a quick recap:

Zephyr started with a single project in a single repository with two stacks, one for production and one for testing. (Details are available in understanding code organization and stacks post.)
To unlock developer velocity, Zephyr quickly added per-developer stacks. (You can read about per-developer stacks post in the series.)
As Zephyr grew and expanded, their single Pulumi project grew into three different Pulumi projects: one for base infrastructure, one for their Kubernetes platform, and one for the online store application. (The structuring Pulumi projects post describes the reasoning for adopting multiple projects.)
Accompanying the switch to multiple projects, Zephyr implemented stack references (to pass information between stacks in different projects) and applied role-based access control (RBAC) to the stacks in Pulumi Cloud. (Refer back to the stack references and RBAC & security posts in the series for more details on each of these areas.)

To see the state of Zephyr’s Pulumi implementation before the changes described in this article, please look at the blog/multi-project branch in each of the GitHub repositories (zephyr-infra, zephyr-k8s, and zephyr-app).

Since we last checked in on Zephyr, one significant change has emerged. As traffic to the Zephyr online store continued to increase, Zephyr’s operations team was finding it difficult to scale the relational databases running on Kubernetes (specifically, the orders and catalog databases). After evaluating their options, the team decided to migrate to Amazon Aurora with MySQL compatibility. Using the considerations outlined for structuring Pulumi projects, they decided to create a fourth Pulumi project, named “zephyr-data”, to manage the data infrastructure for the online store. Initially, this data infrastructure project manages only the Aurora clusters and database instances for the orders and catalog databases.

Because Zephyr was already using a modular approach to their overall application stack, inserting a new project for managing the online store’s data layer involved only minimal changes:

A new Pulumi program had to be written. This new program used stack references to leverage the existing infrastructure created by the base infrastructure stack and allowed traffic to the databases from the security group created by the Kubernetes platform stack. You’ll find this new program in the zephyr-data repository on GitHub (look in the blog/automation-api branch).
The Pulumi program for the Kubernetes platform stack only needed to export the security group created for the cluster nodes.
The Pulumi program for deploying the online store microservices referenced the database endpoints created by the new data infrastructure stack, and no longer spun up containerized instances of MariaDB for the orders and catalog databases.

The resulting four project architecture looks like this:

To see the state of the code after the changes described above—the addition of the zephyr-data project and the changes to use Amazon Aurora—please review the blog/automation-api branch in each of the Zephyr repositories.

Using a modular approach with multiple projects has its benefits. However, one of the trade-offs of splitting your IaC code into separate Pulumi projects is some additional relative complexity. With additional complexity comes additional room for error. To stand up a production-like environment, developers now had to stand up four different stacks (base infrastructure, Kubernetes platform, data infrastructure, and online store) in a specific order. If a user gets the order wrong, then the online store won’t come up properly.

With only four projects, this is not an insurmountable challenge by any stretch. However, what if it was 10 projects? Or 15 projects? Those numbers are not out of reach! Right now Zephyr has a single project to deploy all the various services involved for the online store. What if the application team decides to split up that project so that individual services can be deployed independently of each other? (After all, being able to deploy microservices independently of one another is one of the benefits of such an architecture.)

The Zephyr team started asking some questions: How can we make this process simpler? How can we remove room for error? How can we prepare our teams for greater scale?

Simplifying Advanced Deployments with Automation API

The challenges facing the Zephyr teams are not unique to Zephyr. Many IaC vendors recommend using a modular approach. Unfortunately, most IaC products rely on less-than-ideal approaches—like Bash scripts or Makefiles—to “stitch” together the different modules.

In some cases, organizations can use existing build automation tools and CI/CD pipelines to assist. What’s really needed, though, is a way to programmatically control the IaC tooling, and that’s what Pulumi’s Automation API is. The Pulumi Automation API allows Zephyr’s teams to use a general purpose programming language to automate Pulumi operations like performing an update, refreshing the stack state, or destroying a stack. Like Pulumi itself, the Pulumi Automation API supports multiple languages and supports multiple state backends, including Pulumi Cloud. Further, there is no requirement that Pulumi Automation API programs are written in the same language as the Pulumi programs they automate. In the case of Zephyr, which thus far has chosen to use TypeScript for their Pulumi programs, this means they are free to use any supported language for their Pulumi Automation API programs.

When the Zephyr team started evaluating options for simplifying the deployment of their online store and associated infrastructure, they decided to move forward with the Pulumi Automation API for these reasons:

Tight integration with Pulumi and Pulumi Cloud
Support for Pulumi Deployments (something we’ll examine in the next blog post)
Support for a broad range of programming languages, just like Pulumi itself
No requirement to use the same language as the Pulumi programs being automated (in fact, a Pulumi Automation API program could automate multiple programs written in different supported languages!)

Let’s take a look at Zephyr’s initial foray into using Pulumi Automation API.

Performing Stack Operations Efficiently

You can view the Pulumi Automation API program that Zephyr wrote in the local-source directory of the pulumi/zephyr-automation repository on GitHub, in the blog/automation-api branch.

Zephyr’s team decided to write their Pulumi Automation API program in Golang. One of the first things you’ll note about the Pulumi Automation API program is that it looks more like a “regular” Go program than a typical Pulumi Go program.

For example, a typical Pulumi Automation API program written in Go starts out like any other Go program:

func main() {
 // All your Go code is here
}

In a typical Pulumi Go program, on the other hand, the main() function starts off with a pulumi.Run call, like this:

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // All the Pulumi Go code is here
 })
}

The easiest way to understand this difference is to consider the way each program is invoked. A Pulumi Automation API program written in Go is executed by running go run main.go, like running any other Go program. In contrast, a Pulumi Go program is invoked by the pulumi CLI.

Zephyr’s Automation API program starts out with checking for a command-line argument to indicate the desired mode of the operation—should it be creating/updating resources, or destroying resources? In a standard Pulumi Go program, this would be determined by the invocation of the pulumi CLI (for example, whether you ran pulumi up or pulumi destroy). Here, though, that falls to the user to check:

// Determine mode of operation; default is to refresh/update
destroy := false
argsWithoutProg := os.Args[1:]
if len(argsWithoutProg) > 0 {
 if argsWithoutProg[0] == "destroy" {
 destroy = true
 }
}

Next, the program reads some configuration data from a YAML configuration file; this configuration data includes things like the relative file system location of the Pulumi programs that this Automation API program is going to execute:

// Get configuration data
data, err := os.ReadFile("config.yaml")
if err != nil {
 fmt.Printf("Error reading configuration file: %v\n", err)
 os.Exit(1)
}
var env Environment
err = yaml.Unmarshal(data, &env)
if err != nil {
 fmt.Printf("Error parsing configuration information: %v\n", err)
 os.Exit(1)
}

Separating configuration from code in this manner allows you to use the language’s existing ability to create binary executables that can be easily distributed to your teams. For example, Zephyr has the option of running go build to create a binary executable for distribution to internal teams.

From there, the Automation API program gets into setting up the stacks it will use:

// Set up base stack
baseStackName := auto.FullyQualifiedStackName(env.Organization, env.BaseProject.Name, env.StackName)
baseStack, err := auto.UpsertStackLocalSource(ctx, baseStackName, env.BaseProject.Location)
if err != nil {
 fmt.Printf("Failed to create or select %s stack: %v\n", env.BaseProject.Nickname, err)
 os.Exit(1)
}
fmt.Printf("Successfully created/selected %s stack\n", env.BaseProject.Nickname)

In this particular program, Zephyr decided to start with a local workspace. (You can read more about workspaces in the Pulumi Automation API docs.) This is reflected in the use of the UpsertStackLocalSource method shown above, which will create or select a stack for a Pulumi program found in a local filesystem location.

Users can also use UpsertStackInlineSource, which embeds the Pulumi program into the Automation API program as a function. There are trade-offs to each approach:

Using local source (i.e., the Pulumi program is stored outside of the Automation API program) has the benefit of allowing users to independently revise the Pulumi program or the Automation API program; this may be the ideal approach where organizations have separate teams that will create, manage, or maintain the Automation API programs. In Zephyr’s case, where there is a platform team, this made a lot of sense for them. This does also mean, as you might guess, that the Pulumi programs must be reachable by the Automation API program (this typically means users need to clone down the appropriate repositories to their system first).
Using inline source (where the Pulumi program is embedded in the Automation API program) may be a better fit when there is a single team managing or maintaining both the Pulumi program(s) as well as the Automation API program(s), or when—for whatever reason—these programs should not be revised independently of one another. This has the benefit of removing the need for users to clone or copy separate Pulumi programs onto their system before being able to use the Automation API program.

After all four stacks—base, platform, data, and app—have been set up for use within the Pulumi Automation API program, then comes the “real work”: performing operations on those stacks. To reduce duplicate code, the Zephyr program makes use of three functions:

The refreshStack function (starting on line 199 of main.go) performs the equivalent of a pulumi refresh against a stack.
The updateStack function (starting on line 218 of main.go) performs an update against the stack. This is like running pulumi up against a stack.
Finally, the deleteStack function (starting on line 236 of main.go) destroys the resources in a stack. This equates to running pulumi destroy against a stack.

Using these functions, lines 94 through 197 perform the key task of taking action to stand up or tear down the Zephyr online store and all associated infrastructure. If the user passes the parameter “destroy” on the command line (meaning the Pulumi Automation API program is invoked using go run main.go destroy), then it will tear down the stacks. Otherwise, the Pulumi Automation API program will perform a refresh and then an update—much the same way that the pulumi CLI itself behaves.

Along the way, you can see the Pulumi Automation API program passing configuration values to the stacks being orchestrated (this is lines 177 through 182, for example):

// Set config values for app stack
appStack.SetConfig(ctx, "platformOrgName", auto.ConfigValue{Value: env.Organization, Secret: false})
appStack.SetConfig(ctx, "dataOrgName", auto.ConfigValue{Value: env.Organization, Secret: false})
appStack.SetConfig(ctx, "platformProjName", auto.ConfigValue{Value: env.PlatformProject.Name, Secret: false})
appStack.SetConfig(ctx, "dataProjName", auto.ConfigValue{Value: env.DataProject.Name, Secret: false})
appStack.SetConfig(ctx, "platformStackName", auto.ConfigValue{Value: env.StackName, Secret: false})
appStack.SetConfig(ctx, "dataStackName", auto.ConfigValue{Value: env.StackName, Secret: false})

Among other things, this means you can take your existing Pulumi programs, and—without making any changes—orchestrate them using an Automation API program! Further, in reviewing the code shown above, you can see that the Pulumi Automation API ties into Pulumi’s secrets functionality.

In the end, using Pulumi’s Automation API enabled Zephyr’s platform team to simplify the workflow for standing up the online store application and all of its associated infrastructure into a simple and straightforward process:

Developers ensure they have cloned the appropriate source repositories onto their system.
Developers run go run main.go (or go run main.go destroy) to stand up or tear down the online store and all associated infrastructure.

Although Zephyr uses multiple GitHub repositories, the Pulumi Automation API has no requirements for multiple repositories. You could use a monorepo, if that better suits your particular requirements.

Summarizing Best Practices: Automation API

This post discussed the following best practices when using Pulumi Automation API:

Use the Automation API to orchestrate operations on multiple stacks. In cases where you’ve split Pulumi projects (per the recommendations in the post on structuring Pulumi projects), you’ve introduced a potential opportunity for using Automation API to orchestrate operations. This is a great counter-balance to offset the relative increase in complexity that comes from breaking large Pulumi projects into smaller, more focused Pulumi projects.
Separate configuration from code. This is already a generally-accepted recommendation in most programming circles, and it applies to Automation API programs as well. Doing so enables you, if you wish, to distribute binary artifacts to teams instead of requiring them to execute the source code of your Automation API programs.
Use local source to decouple Automation API programs from Pulumi programs when separate teams are involved. When separate teams are creating, managing, or maintaining the Automation API programs and the Pulumi programs, using local source instead of inline source enables the teams to operate somewhat independently.
When there is a single team managing both Automation API programs and Pulumi programs, use inline source. This is also true when, perhaps due to compliance or regulatory concerns, the Automation API program should be more tightly coupled to the Pulumi programs it is automating.

The Zephyr series wraps up in the next post, where we summarize all our recommendations so far. We’ll also take a look at areas of potential future growth for the team at Zephyr. Don’t miss it!

IaC Best Practices: Implementing RBAC and Security

Scott Lowe — Tue, 23 May 2023 00:00:00 +0000

This post continues our series of blog posts focused on IaC best practices. In earlier posts, we introduced Zephyr Archaeotech Emporium, the fictional company that sits at the center of this series, and discussed Zephyr’s primary use case for Pulumi: managing their online retail store. You read how Zephyr’s initial use of Pulumi changed to incorporate the use of short-lived per-developer stacks. Later, as Zephyr continued to grow, you saw how Zephyr restructured their Pulumi projects and stacks, and incorporated the use of Stack References. This post is a complement to the post on structuring Pulumi projects, concentrating on the use of role-based access control (RBAC) and security in Zephyr’s multi-project configuration.

The ultimate goal of this series is to discuss best practices for using Pulumi to manage a fairly complex containerized application. You’ve seen these practices emerge over the course of the series—not all immediately, and not all right away. Structuring the blog series in this way is a deliberate decision; many best practices are “point in time” recommendations: they are context-dependent and the recommendations for any given company, like Zephyr, may change as the company and its needs change.

Evolving Project Structure to Enhance Security and Scalability

So far, you’ve read about how Zephyr started out with a single project and a single Git repository. Adding short-lived per-developer stacks, as described in the post on best practices for developer stacks and Git branches, didn’t really impact this structure. However, as the company grew, Zephyr’s project and Git repository structure evolved into its current form:

The “zephyr-infra” project handles base infrastructure (VPC and related constructs).
The “zephyr-k8s” project automates the Kubernetes platform layer, building atop the base infrastructure components supplied by the “zephyr-infra” project.
Finally, the “zephyr-app” project handles the deployment of the online retail store application onto Kubernetes, layering on top of the platform provided by the “zephyr-k8s” project.

You can review the reasons for this structure in the post on best practices for structuring Pulumi projects. One of the reasons listed in the third Zephyr blog post was security:

…security is a factor that affects how projects and stacks should be structured. In Zephyr’s case, the ability for one team to affect another team’s resources was present when everything was a single project. With multiple projects, Zephyr can use stack permissions and role-based access control (RBAC) in Pulumi Cloud to appropriately control who can affect what.

The last part of that quote—the part about being able to “appropriately control who can affect what”—is another way of describing what’s known as the principle of least privilege. If you’re unfamiliar with this term, we recommend following the link and getting more information, because the principle of least privilege stands at the core of all the recommended practices you’ll read in this post.

Pulumi supports a number of self-managed backends, including Amazon S3 (and S3-compatible servers like Minio or Ceph), Azure Blob Storage, and Google Cloud Storage. The RBAC functionality described in this post applies only to the managed Pulumi Cloud backend.

Understanding the RBAC Building Blocks in Pulumi Cloud

To understand how to secure your Pulumi projects and stacks in Pulumi Cloud, it is important to first explain the basic building blocks you’ll use:

Organization: An organization represents a customer/company within Pulumi Cloud. Generally speaking, a single company like Zephyr is represented by a single organization.
Member: Each member represents a user account that is part of an organization. In Zephyr’s case, each Zephyr employee that needs to interact with Pulumi is a member of Zephyr’s organization in Pulumi Cloud.
Team: A team is a grouping of members, and is the primary component involved in assigning per-stack permissions in the Pulumi Cloud RBAC model. How you structure teams within your organization will depend on a great many different factors we’ll explore in this post.
Stack: A stack is a separate instance of the resources created by a Pulumi program within a project. Each stack has its own independent state and its own configuration values. With respect to RBAC, each stack is a separate entity to which one or more teams can be granted permission. In Zephyr’s case, there is a long-lived production stack for each of the separate projects, and a persistent (or long-lived) test stack for each of the separate projects. There are also the per-developer stacks, which are typically short-lived.

This diagram, also used in the post on best practices for stack references, shows the stacks and projects for Zephyr’s online store:

Now, how are these different components assembled together? At its most basic level, you would use teams to group members together, and then assign stack permissions to the team. This diagram helps illustrate the idea:

A couple of important points emerge from the above diagram:

The relationship between stacks and teams is many-to-many. That is, any given stack can have many teams assigned permissions, and any given team may be assigned permissions to many stacks.
Similarly, the relationship between teams and members is also many-to-many (any given team can—and almost always will—have multiple members, and any given member can be part of multiple teams at the same time).

Assembling Security for Stacks

The first, and perhaps the most important, best practice with regards to securing your stacks is to actually use Pulumi Cloud’s RBAC functionality. While this may seem obvious, there are practical implications to this statement that are useful to explore.

Pulumi Cloud has the ability to set a default stack permission for all members of an organization. In the screenshot below, you can see that the default permission has been set to Write:

In this configuration, you are effectively disabling Pulumi Cloud’s RBAC functionality. Permissions in Pulumi Cloud are additive; that is, permissions only state what you can do, not what you can’t do. Therefore, if the default stack permission is set to Write, then assigning a team read-only permissions to a stack has no effect—because members of that team can read from the stack (by virtue of their inclusion in a team) and can write to the stack (by virtue of the default stack permission). The team members’ read-only permission is added to the default stack permission for their effective permission.

For this reason, Pulumi recommends setting the default stack permission to Read or None. Setting it to Read means that all members of the organization can see and read from stacks that aren’t their own, but can’t change or modify the others stacks. Setting it to None means that members have permissions to their own stacks by default—gaining any form of access to stacks other than their own requires being added to a team that has permissions to those stacks.

When it comes to assigning a team access to a stack, there are three options:

Stack reader (read-only access; also called “Stack reader access” in some areas of the Pulumi Cloud UI)
Stack editor (read/write access, but can’t delete the stack; also called “Stack write access” in the Pulumi Cloud UI)
Stack admin (full access; you may also see it called “Stack admin access”)

When the “Allow organization members to create stacks and transfer stacks to this organization” is checked, like in the screenshot above, then the user (organization member) that creates a stack automatically becomes the stack admin for that stack. The user’s ability to delete the stack, however, is affected by the “Allow stack admins to delete stacks” setting.

Full details on what each of these permissions grants is detailed in the projects and stacks documentation (see the “Stack Permissions” section). In accordance with the principle of least privilege, Pulumi recommends you grant the minimum level of access necessary in order for a user (organization member) to meet their job responsibilities. You accomplish this by adding the member to a team, and then granting the team the minimum level of necessary access. You can’t assign stack permissions directly to users.

Because team membership is central to the use of stack permissions, Pulumi has several ways to make it easier to manage the relationship between teams and stacks:

You can use the Pulumi Cloud user interface to assign a team access to a stack.
You can use Pulumi Cloud’s REST API to control team access to stacks.
You can use the Pulumi Cloud provider for Pulumi to programmatically manage team access to a stack using Pulumi itself.
You can use the --teams flag to pulumi stack init to assign stack editor access to one or more teams when you create the stack.

In general, Pulumi recommends declaratively managing your Pulumi Cloud RBAC configuration with the Pulumi Cloud provider (formerly known as the Pulumi Service provider), unless you have specific requirements that would prevent it. However, be aware that the Pulumi Cloud provider doesn’t allow you to change a member’s role in an organization (every member is either a “member” or an “admin” within an organization); that would have to be done via the Pulumi Cloud user interface or REST API.

Pulumi also recommends using the --teams flag with pulumi stack init to ensure that new stacks are properly assigned team permissions upon creation. You can then go back and fine-tune those team permissions—which are set to “Stack editor” or “Stack write access” when you use the --teams flag—using a Pulumi program with the Pulumi Cloud provider. Alternately, teams can use a team access token when creating a new stack, and that will also grant the team permission to the newly-created stack.

Finally, Pulumi recommends allowing users to create their own stacks. Otherwise, only an organization admin would be allowed to create new stacks, and this is likely to cause a bottleneck in provisioning cloud infrastructure—one of the very issues that infrastructure as code tools like Pulumi are attempting to solve! (It may be necessary in some cases to restrict this functionality, such as in certain highly regulated industries or verticals.) Preventing users from creating their own stacks would also prevent the use of per-developer stacks like Zephyr uses, which as you’ve seen can be enormously helpful in accelerating development and deployment velocity.

Inspecting RBAC Implementation

Zephyr’s RBAC implementation in Pulumi Cloud is represented in part by a diagram that you also saw earlier in this post:

Zephyr set the organization default stack permission to Read. This enabled teams to read all stacks, and enabled the continued use of stack references without any additional configuration. Teams were created to grant “Stack editor” (also called “Stack write access”) to stacks based on the overall organizational structure of the company. Thus, the Platform team gained access to the non-production zephyr-infra and zephyr-k8s stacks, while the Application team was given access to the non-production zephyr-app stacks. However, only members of the Production team were given access to the production stacks in all three projects, ensuring that access to update those stacks—and thus the actual resources running Zephyr’s online store—remained relatively controlled.

Summarizing Best Practices: RBAC & Security

The post covered the following guidelines for securing your stacks in Pulumi Cloud:

Use Pulumi Cloud’s RBAC functionality by setting your organization’s default stack permissions to Read or None. This is generally the safest way to implement the principle of least privilege, unless you have a good use case for allowing all organization members full write access to all stacks.
When assigning team permissions, grant the minimum level of access necessary. If a user isn’t responsible for making updates to a stack, then give that user stack reader permission through a team membership (remember that you can’t assign stack permissions directly to organization members, you have to use a team). Keep in mind, though: if the organization level stack permission has been set to Read, then members already have the equivalent of stack reader permissions on all stacks, so creating a team to grant stack reader permission is unnecessary and superfluous.
Declaratively manage your Pulumi Cloud RBAC configuration where possible using the Pulumi Cloud provider from within a Pulumi program.
Use the --teams flag with pulumi stack init to ensure that teams have stack permissions upon stack creation. You can also use a team access token to accomplish the same result.
Allow users to create their own stacks. Generally, turning this functionality off (found in the Settings > Access Management screen of Pulumi Cloud) will end up creating a bottleneck, as only organization admins would be allowed to create new stacks.

In the next IaC Best Practices post, the focus is on Automation API. Stay tuned to see how Zephyr uses Automation API to further streamline and automate the deployment of their cloud resources!

IaC Best Practices: Applying Stack References

Scott Lowe — Fri, 31 Mar 2023 00:00:00 +0000

This is the fourth post in a series of blog posts focused on Zephyr Archaeotech Emporium—our fictional company—and their use of Pulumi to manage their online retail store. In the first three posts, you saw how Zephyr’s initial use of Pulumi changed as the company grew, and how the use of short-lived per-developer stacks helped Zephyr’s application development team meet the demands of a fast-growing company. This post is a complement to the earlier post on structuring Pulumi projects, discussing how Zephyr uses Stack References to link their projects together and sharing some IaC best practices around the use of Stack References.

As you may have read in earlier Zephyr posts, the ultimate goal of the Zephyr series is to share best practices on the use of Pulumi to manage your infrastructure and application resources (using a fictional company and a somewhat complex containerized application as the use case). The series exposes those practices over time—not all right away, and not without also discussing the context for the recommendations. This is deliberate, demonstrating how “point-in-time” recommendations change based on the needs of the company and its requirements.

Reviewing Zephyr’s Current Status

This post continues the previous post in the series. Zephyr’s infrastructure footprint and Pulumi codebase is the same; that is, Zephyr now has three Pulumi projects (zephyr-infra, zephyr-k8s, and zephyr-app). Each of these projects handles a different aspect of Zephyr’s online store:

The “zephyr-infra” project handles base infrastructure (VPC and related constructs).
The “zephyr-k8s” project handles the Kubernetes platform layer, building atop the infrastructure components provided by the “zephyr-infra” project.
The “zephyr-app” project handles the deployment of the online store services onto Kubernetes, layering on top of the platform provided by the “zephyr-k8s” project.

The reasons for this structure are explained in structuring Pulumi projects post. For Zephyr, it came down to supporting multiple teams and allowing the application to evolve independently of the infrastructure that supports it.

You can continue to use the blog/multi-project branch in the GitHub repositories to see the state of Zephyr’s code and projects for this blog post.

It’s clear that there are dependencies across projects: The zephyr-k8s project needs to know the VPC ID and the subnet IDs from the zephyr-infra project. Similarly, the zephyr-app project needs to have access to the Kubernetes cluster details from the zephyr-k8s project. As the Zephyr team went about building the code that’s now in use in their multi-project architecture, they needed to decide how best to handle these cross-project dependencies. The Zephyr team knew that hard-coding output values from one project as configuration values in another stack wasn’t the ideal way (and in fact Pulumi strongly recommends against hard-coding output values), but what was the best solution to use? The answer to that question is stack references.

Linking Stacks with Stack References

Stack references allow you to access output values from one stack in another stack. In other words, stack references offer a way to programmatically “link” stacks in different projects for the purpose of sharing information. You can find more details in the documentation on stack references.

Using stack references, the Zephyr team can make the necessary connections they need:

The code handling the Kubernetes cluster can reference the VPC ID and subnet IDs from the base infrastructure layer.
Similarly, the code that is responsible for deploying the online store services can reference the necessary Kubernetes connection details (the Kubeconfig) for the Kubernetes cluster.

You can see how the Zephyr team did this for their projects, including code samples, toward the end of this post (in the section titled “Examining Zephyr’s use of stack references”). This graphical representation also illustrates Zephyr’s current repository and project structure as well as the stack references:

It’s worthwhile to note that any information that needs to be accessible from another stack via a stack reference must be exported as a stack output in the source stack. If you don’t mark it as a stack output, then it can’t be used in a stack reference. Adding stack outputs after the fact requires little effort and has no impact on existing infrastructure.

Also, any value retrieved via a stack reference is treated as a Pulumi Output, and therefore may require some extra work to transform values (such as the use of Output.apply). The recent addition of OutputDetails support in Pulumi—you can read more about OutputDetails in this blog post announcing the functionality—helps considerably in this situation. Some SDKs also have language-specific mechanisms that can help; for example, using Go’s .AsStringArrayOutput() method on a StackReference.GetOutput statement makes referencing subnet IDs from other project much easier.

While stack references are conceptually straightforward and not difficult to implement, there are some best practices to be mindful of regarding the use of stack outputs and stack references:

When it comes to stack outputs, export what is needed. If it needs to be accessed from outside the stack, export it; otherwise, don’t. Why? Stack outputs form the application programming interface (API) by which other Pulumi programs interact with this Pulumi program. Like in any API implementation, expose what the user cares about or is likely to need, and leave the rest as an implementation detail. You can add more stack outputs after the fact with only minimal code changes, and as has been mentioned already this is done with a quick pulumi up that has no affect on existing infrastructure (you’re modifying the stack object itself).
Be judicious in your use of stack references. This is sort of the “flip side” of the previous bullet. If you need to reference an output from another stack, then do so; otherwise, don’t. If you find yourself needing lots of stack references, then the way in which you’ve structured your projects may need to be re-examined so as to minimize the coupling between projects. Additionally, there is a potential “speed of light” concern here; if where you are executing your Pulumi code is across a high-latency link from where you store your Pulumi state, stack references will be affected by that latency.
If you need to expose lots of outputs on a stack, consider exporting a structured data object. It’s possible to construct a JSON object (or dict or struct, depending on your language) to hold all the stack outputs, and then export that object. Then a stack reference can read that object, resulting in a single call to the backend. Be aware, though, that you’ll need to write the necessary code to understand/import/unmarshall that JSON object in the referring stack, so there is a small amount of extra work required in this situation.
In line with parameterizing as much of your code as possible, also be sure to parameterize your stack references. A stack reference is built using an organization name, a project name, and a stack name. Don’t hardcode these values; instead, use configuration values to allow the users to specify from which source stack(s) the values will be referenced. You’ll see an example of that in the next section.
Plan for handling cascading stack updates. If a stack has outputs being consumed via stack references, what happens when those stack outputs change? The Pulumi Service supports webhooks that can help, and Pulumi is currently exploring options for having Pulumi Deployments help with this (see Pulumi Cloud Requests GitHub issue). In practice this should be uncommon, and frequent occurrences may be a good indicator that your resource partitioning (how resources are split among multiple projects) needs to be adjusted.

One thing not listed above is security—and that’s because a later post in the series focuses on this topic. You’ll want to keep the security of your stacks in mind when planning for the use of stack references. The teams managing the stacks where the stack references exist will need at least read-only access to the stacks where the outputs exist. (For more on this front, refer to the IaC Best Practices post on RBAC and security.)

Examining Use of Stack References

With these best practices in mind, you can examine the Zephyr team’s implementation to see how they put these recommendations into action.

In the code for the base infrastructure stack, you can see that Zephyr exported the essential values needed by the Kubernetes platform stack:

// Export some values for use elsewhere
export const vpcId = eksVpc.vpcId;
export const privSubnetIds = eksVpc.privateSubnetIds;
export const pubSubnetIds = eksVpc.publicSubnetIds;

In the Kubernetes platform stack, the Zephyr team parameterized the values needed for the stack reference. This particular approach, by the way, is key to preserving the per-developer stacks to which the Zephyr team has grown accustomed (each developer needs to specify the correct organization, project, and stack name):

// Grab some configuration values
// Some code omitted here
const infraOrgName = config.require("infraOrgName");
const infraProjName = config.require("infraProjName");
const infraStackName = config.require("infraStackName");

// Create a StackReference to get information from base stack
const infraSr = new pulumi.StackReference(`${infraOrgName}/${infraProjName}/${infraStackName}`);

As with the base infrastructure stack, only the key value needed by other stacks is exported (in this case, the Kubeconfig required to access this cluster):
```
// Export some values for use elsewhere
export const kubeconfig = eksCluster.kubeconfig;
```

Finally, in the application stack, the stack reference values are again parameterized, and the Kubeconfig—necessary for the application stack to deploy onto the provisioned Kubernetes cluster—is referenced via a stack reference:

// Grab some configuration values
const config = new pulumi.Config();
const k8sOrgName = config.require("k8sOrgName");
const k8sProjName = config.require("k8sProjName");
const k8sStackName = config.require("k8sStackName");

// Create a StackReference to get Kubeconfig from base stack
const kubeSr = new pulumi.StackReference(`${k8sOrgName}/${k8sProjName}/${k8sStackName}`);

All of the GitHub links in the paragraphs above reference the blog/multi-project branch in each repository, which has the code as of this blog post (and the earlier blog posts).

Summarizing Best Practices: Stack References

This post covered the following guidelines for working with Pulumi:

Export the minimum of what’s needed from your stacks. It’s not difficult to add stack outputs later, and you can do it with no impact to existing infrastructure.
Be judicious in the use of stack references. Reducing stack references where possible helps keep projects from becoming overly coupled together, much in the same way as minimizing stack outputs.
Use a structured data object for large numbers of values that need to be referenced. This enables Pulumi to retrieve all the data with a single call, but be aware you’ll need to write extra code to understand the data structure in the referring stack.
Parameterize your stack references. Don’t hardcode organization, project, or stack name values. Instead, pass these in as configuration values (with default values applied, where applicable).
Plan to build mechanisms to handle cascading stack updates. Using the Pulumi Service’s webhook functionality can be useful here. Frequent instances of cascading updates might be an indicator that your project structure needs to be revisited for optimal resource partitioning.

The next post in the IaC best practices series focuses on security and role-based access control (RBAC)—a topic you won’t want to miss! Before then, the next visit to the “Zephyr universe” will be to take a look at an oft-overlooked use case for Pulumi: local testing. Come see how to use Pulumi to automate local testing for development!

IaC Best Practices: Structuring Pulumi Projects

Christian Nunciato — Fri, 17 Mar 2023 00:00:00 +0000

This is the third post in a series of blog posts focused on Zephyr Archaeotech Emporium—our fictional company—and their use of Pulumi to manage their online retail store. In the first post, you saw how Zephyr initially decided to go with a single Pulumi project for managing deployments of their online retail store application. In this post, you’ll see how Zephyr’s use of Pulumi changes as their company grows and evolves.

The first post about organizing code and stacks specifically mentions that the series will show “how Zephyr’s use of Pulumi evolves as the company grows and their online retail store application changes to accommodate their growth.” While the ultimate goal of this series is to discuss best practices for using Pulumi to manage a fairly complex containerized application, it’s important to note that many best practices are “point in time” recommendations: best practices are context-dependent and the recommended practices for a given organization will change as the company and its infrastructure needs also change. This blog post is an example of that evolution in action: the practices that worked well for Zephyr while they were smaller and had smaller infrastructure and less teams working on that infrastructure are no longer ideal as the company has grown.

Growing Development Teams

As a result of launching its online presence, Zephyr saw amazing success. The market for arcane artifacts and novel curiosities is booming, and Zephyr is becoming a leader in this market. Funded by this success, Zephyr has grown rapidly. Along with this growth, Zephyr’s software engineering and IT teams also grew and expanded. Fortunately, Zephyr’s use of per-developer stacks with short-lived ephemeral Git branches positioned the development teams well for this growth.

For a look at the state of Zephyr’s Pulumi code as it stood as of the end of the second post in the series, see the blog/dev-stacks branch of the pulumi/zephyr-app repository on GitHub.

Naturally this growth has resulted in some separation of duties. The IT team now has responsibility for the infrastructure that supports the online retail store. Within that team, some team members are focused on “core” infrastructure, while other team members are focused only on Kubernetes (a natural decision given Zephyr’s use of Kubernetes in their architecture). The application team continues to remain responsible for the online store application itself.

This left Zephyr again asking some questions about the best path forward:

Should they use a single project, or multiple projects? Zephyr had used a single Pulumi project up until now, but growth and organizational change meant they wanted to make sure they were using Pulumi in a way that best reflected their needs. What are the advantages and disadvantages of each approach?
If multiple projects, how many projects? What’s the best way to determine how many projects are needed? What factors play a role in this decision?

Structuring Pulumi Projects

One project, or multiple projects? And if multiple projects, how many projects? These are the core questions the Zephyr needed to resolve. In the end, there are a number of key factors that determine the answers to these questions:

Use case: Zephyr has only a single application (so far!), but in situations where there are multiple applications then keeping those applications (and sometimes their associated infrastructure, depending on other factors) in separate Pulumi projects is generally recommended. In some ways, “use case” is an aggregation of other factors—a new application with a distinct set of resources that do not share a lifecycle with existing resources and are potentially managed by a different team.
Company/department/team structure: How a company/department/team is organized should certainly be taken into account when structuring your Pulumi projects. In Zephyr’s early days with Pulumi, a single team managed everything so a single project was fine. Now with multiple teams, Zephyr will likely find using multiple projects to be more effective.
Security: Closely related to the previous bullet, security is a factor that affects how projects and stacks should be structured. In Zephyr’s case, the ability for one team to affect another team’s resources was present when everything was a single project. With multiple projects, Zephyr can use stack permissions and role-based access control (RBAC) in Pulumi Cloud to appropriately control who can affect what. (The IaC Best Practices blog post on RBAC and security discusses this topic in more detail.)
The relationship between resources: The relationships between resources—such as which resources are dependent upon other resources—is another factor that feeds into your Pulumi project structure. A “shared infrastructure layer” is often best handled in a separate project of its own, but may pull in resources that might otherwise have been kept with an application in its own project. (In some respects, this is a “counter-balance” to the first bullet point.) On the other hand, resources that belong exclusively to a specific application or function might be best bundled in the same Pulumi project.
Resource lifecycle: Do resources have a shared lifecycle, meaning they are always created, updated, or deleted together? If so, then those resources should be placed in the same project. If the resources do have a shared lifecycle, then placing them in the same project simplifies lifecycle management. A great example of resources with a shared lifecycle would be a Kubernetes Deployment and the associated Kubernetes Service.
Change rate of resources: Some resources may change very infrequently. Consider the example of an AWS VPC and associated objects. Once it is created, it may not change for extended periods of time, if ever. On the other hand, some resources may change frequently (in other words, these resources have a high change rate). A fast-moving software development team may revise the Kubernetes Deployment regularly to pull in a new container image with updated code. In a single project, combining resources that have a high change rate with resources that have a low change rate may be suboptimal.
Blast radius: Finally, there is the consideration of what happens in the event of human error, and limiting the “blast radius.” With a single project, all the resources in that project are within the blast radius—all of them could potentially be affected in the event of human error. Using separate Pulumi projects can limit the blast radius. It’s not something that should drive the decision all by itself, but it is a factor to be considered.

Now that you have an idea of some of the factors that affect how users structure their Pulumi projects, what are the advantages and disadvantages of each approach?

For single projects, the key advantage is simplicity. There’s no denying that a single project is simpler and less complex than working with multiple projects. This advantage, though, is also the key disadvantage. A single project does not afford the flexibility to let different teams manage different aspects of the infrastructure, for example, nor does it offer the ability to split resources that don’t have a shared lifecycle. And, as mentioned earlier, blast radius is a concern.

For multiple projects, the key advantage is flexibility. It’s true that the added flexibility comes at the cost of some added complexity (a future post will explore one aspect of that added complexity in the form of StackReferences). However, you gain the ability to separate infrastructure resources according to team ownership/responsibility, relationship to other resources (i.e., grouping resources in a “shared infrastructure layer”), or any of the other dimensions mentioned above.

In some ways, the decision points for single vs. multiple projects are analogous to the decision points for monolith vs. microservice: independent scaling and flexibility versus simplicity.

In general, most customers end up using multiple Pulumi projects per use case. As you’ve seen, though, there are a number of factors that influence that decision and will affect how many projects end up in use.

Moving Forward with Projects

Zephyr had a number of factors in play, not the least of which was the organizational change resulting from their growth and success. In the end, Zephyr decided to restructure their use of Pulumi for the online retail store into three separate projects:

An infrastructure project: This project handled the AWS VPC and related infrastructure resources.
A platform project: This project managed the Kubernetes cluster(s).
An application project: This project handled the actual online retail store itself.

You can use a combination of code refactoring and pulumi import to assist in the process of restructuring Pulumi projects, but it some cases it may be easier to create new infrastructure and migrate over to it.

In the code organization and stacks post, you read about some of the options for where to store Pulumi code relative to the applications it supports. At that time, Zephyr had decided to go with what’s called a monorepo approach, meaning that the Pulumi code was stored in the same repository as the application code. In conjunction with the project reorganization, Zephyr also decided to change how their code was organized:

Code for the infrastructure project moved to a separate “zephyr-infra” repository
A new “zephyr-k8s” repository housed the code for the platform project
The code for the online retail store application itself remained in the same “zephyr-app” repository

Note that relocating code (to a different GitHub repository or different filesystem location within a repository) generally has no impact on Pulumi. As long as the project name and stack name do not change, then changing the filesystem location won’t affect anything—keeping in mind, of course, that filesystem paths and such may need to be adjusted in code to account for any such changes.

After restructuring their projects, Zephyr’s arrangement of Pulumi projects and stacks, along with their associated GitHub repositories, looks something like this:

These repositories are available for you to review on GitHub (zephyr-infra, zephyr-k8s, and zephyr-app); use the branch/tag selector to find the blog/multi-project branch in each repository to see the state of the code as of this blog post. Be aware that the process for standing up the Zephyr online store is now a bit more complex; check the instructions in each repository for more details if you’d like to give these projects a spin.

Summarizing Best Practices: Pulumi Project Structures

This post covered the following recommended practices for working with Pulumi:

Use multiple Pulumi projects when you need the added flexibility that multiple Pulumi projects offer.
Align your Pulumi projects according to external factors like company/department/team structure, resource dependencies, lifecycle, and use case/application.
Change the filesystem location for Pulumi code as needed without impacting the resources managed by that code, as long as the Pulumi project name and stack name don’t change.

The next post will focus on some specific technical details related to Zephyr’s project reorganization; specifically, how they used StackReferences to share information between their Pulumi projects. Stay tuned!

IaC Best Practices: Enabling Developer Stacks & Git Branches

Christian Nunciato — Fri, 10 Mar 2023 00:00:00 +0000

In the first post about code organization and stacks, we introduced Zephyr, a fictional company that uses Pulumi to manage its online retail store. Following on from that post, which discusses code organization and stacks, this post explores two more questions users frequently ask when working with Pulumi in teams — namely, How can I best enable multiple developers to collaborate on a Pulumi project? And how can I use Git and Git branching to support this kind of collaboration? In this post, we’ll provide some guidance and best practices around these topics, using Zephyr and its online store as the use case.

The ultimate goal of this series is to discuss best practices for using Pulumi to manage a fairly complex containerized application. However, it’s important to note that these practices will emerge over the course of the series — not all immediately, and not all in the beginning. This is a deliberate decision to allow you to see how Zephyr’s use of Pulumi evolves as the company grows and its retail application changes to accommodate its growth.

Existing IaC Workflow

When we last met up with the Zephyr team, they were off and running, managing their newly refactored online store, Zephyr Archaeotech Emporium, with a single Pulumi project and two Pulumi stacks — one for development (dev) and another for production (prod). The team had chosen to use one Git repository (a monorepo) to manage the code for the online store and its infrastructure after refactoring the store into a set of containerized microservices deployed with Kubernetes on Amazon EKS.

For a snapshot of the code as it was at the end of the first post in the series, see the blog/original branch of the pulumi/zepyhr-app repository on GitHub.

Up to now, the two-stack approach has worked well for the team. It’s allowed them to focus on building new features, integrating those features into the shared dev environment by developing and testing locally (which we’ll cover in a future post) and merging their feature branches regularly into the base branch, main, using standard GitHub pull-request workflows. Around once a day, as the team’s changes are merged into main, the team lead (and original author of the Pulumi program) deploys them into the dev environment with Pulumi with the following commands:

git pull origin main
cd infra
pulumi stack select zephyr/dev
pulumi up

This builds new container images for any services whose code has changed since the last stack update, pushes those images to Amazon ECR, and updates the EKS cluster to deploy the services automatically.

Then, around once a week, as new features are ready for customers, the team lead deploys the main branch of the repository into production similarly:

git pull origin main
cd infra
pulumi stack select zephyr/prod
pulumi up

The deployment process is still a bit more manual than the team would like — but since updates typically take only a few minutes, the team lead is generally satisfied with the status quo.

A few more details worth noting out about Team Zephyr’s current setup:

Both of the project’s stack-configuration files, Pulumi.dev.yaml and Pulumi.prod.yaml, are checked into version control. This is safe and considered good practice (even with secrets) because it allows the team to review or change the configuration of a given stack easily by viewing or editing the code in each file.
The main branch is the only long-lived branch in the repository; all others (e.g., developers’ feature branches) are deleted once they’re merged into main.
Thanks to this workflow, the application deployed into the dev environment tracks closely with the latest code in the main branch. As such, the dev stack serves as a kind of pre-prod or integration environment that not only allows the team to dogfood new features as they’re added, but also to catch bugs or other regressions before they escape into production. When things do go wrong, the team can review the list of recent stack updates to identify the source of the defect and get it fixed.
Zephyr is using the Pulumi Service backend with a single organization (that everyone on the team belongs to) and separate AWS accounts for the dev and prod environments. Everyone on the team has read/write access to the AWS account used for development, but only the team lead has access to the AWS account used for production.

Growing Pains

Life for Team Zephyr was good — for a while.

Then one day during a particularly active sprint, a developer who’d started working on an improvement to the checkout process realized it might benefit from an updated version of Redis, which appeared to be presently pinned at 6.0 in the Pulumi program:

// ...
const checkoutRedisDeployment = new k8s.apps.v1.Deployment("checkout-redis-deployment", {
 spec: {
 template: {
 spec: {
 containers: [{
 image: "redis:6.0-alpine",
 // ...

A quick glance at the dev and prod stacks in the Pulumi Service confirmed this was the case.

After a quick spike to validate the enhancement locally, the developer felt confident that a Redis upgrade was the way to go — but since the deployed version of Redis would ultimately dictate the code that’d have to be written to finish the feature for real, they couldn’t really be sure without testing the theory with a full deployment. What they really wanted was to be able to deploy a Redis upgrade, along with the associated code changes, into an AWS environment as much like production as possible — the dev environment, say — without having to go through the process of making a pull request, getting it merged, and waiting for the team lead to deploy the dev stack only to realize later that the change didn’t actually work as expected, and be forced to revert.

In situations like these, it can be tempting to suggest that the developer simply pick up and use the dev stack — especially on small, fast-moving teams like Zephyr’s where everyone already has access to the Pulumi stack and the shared cloud environment. Indeed, Pulumi makes this sort of thing easy: to deploy their local changes into the dev environment, our developer need only check out their feature branch, select the dev stack, run pulumi up, and then wait a few moments for the changes to be deployed:

cd infra
pulumi stack select zephyr/dev
pulumi up

This approach definitely works — but it presents at least a few problems:

The team would lose all of the practical benefits of having the dev stack track closely with the latest content in main.
The team lead’s deployment of main into dev later that day would overwrite the developer’s own feature-branch deployment. There’s a good chance the lead would notice the diff of the deployment preview and deduce that someone else on the team had possibly deployed into dev out of band:
```
 kubernetes:apps/v1:Deployment (create):
 spec: {
 template: {
 spec: {
 containers: [
 [0]: {
- image: "redis:6.0-alpine"
+ image: "redis:7.0-alpine"
 ...
 }
 ]
 }
 }
 }
 ...
```
But if they didn’t, the developer’s ad-hoc deployment would be lost. At best, it’d force both team members to drop what they were doing and get in touch to figure out what to do next. At worst, it could lead to confusion and the potential for shipping untested code.
In addition to raising the risk of developers overwriting each other’s work, active collaboration the same stack increases the likelihood of deployment conflicts. On the one hand, this is a good thing in that Pulumi prevents same-stack updates from running concurrently:
```
$ pulumi up
error: [409] Conflict: Another update is currently in progress.
```
But on the other, it’s a not-so-good thing in that it’s a reduction in development velocity: The update that can’t run because another update is in progress is an update that will have to wait until sometime later. The more often this happens, the longer it ultimately takes for the team to get the validation it needs to keep moving forward.

So facing these issues, what should the Zephyr team do? The dev stack had originally been created for rapid development — and that worked well, when there was only one developer working on infrastructure. With several, however, things clearly get complicated. Should the team be using a dev stack at all? If so, how? And if not, what should they be doing instead?

Stepping Back to Reassess

With what we’ve learned so far, we can return to the questions we raised at the beginning of this post and begin to sketch out a few recommendations for Team Zephyr. Those questions, again, were:

How can I best enable multiple developers to collaborate on a Pulumi project? For Zephyr, an ideal answer to this question would give every member of the team the ability to iterate on both application and infrastructure code concurrently, deploying as often as needed without having to worry about disrupting the work or the velocity of anyone else on the team.
How can I use Git and Git branching to support this collaboration? So far, the feature-branch/pull-request workflow has been serving the Zephyr team well. Should they continue on the same path, merging frequently to main and deploying to dev and prod from there? Or is there some alternative workflow that might serve the team better, either now or at some time in the future?

Let’s answer these questions now by taking a look at how the Zephyr team handled them.

Using Developer Stacks

We’ve seen a few ways that having multiple developers share a single stack for ad-hoc deployment can lead to various problems. But we’ve also seen at least one situation in which being able to run ad-hoc deployments — without the risk of colliding with existing environments — can be quite valuable. In fact there are many reasons you might want to do this — and fortunately Pulumi makes it quite easy to do so.

The best practice is to allow developers to create and manage their own Pulumi stacks. In this workflow, developers working on feature branches are able to iterate freely on both application and infrastructure, deploying as often as necessary into a shared cloud environment (typically a sandbox environment, isolated from production) and destroying cloud infrastructure when it’s no longer needed.

For the developer, the typical workflow looks something like this:

Check out a new feature branch
Make changes to application and/or infrastructure code
Select your personal developer stack with pulumi stack select
Deploy your stack into your team’s shared cloud environment with pulumi up
Commit your code and submit a pull request, optionally including a link either to the Pulumi update (e.g., when using the Pulumi Service) or to certain key resources belonging to the stack — for example, an application’s primary HTTP endpoint

The benefits of a workflow like this one are many:

No more resource-naming collisions. Because Pulumi names cloud resources uniquely by default, no two stacks will ever produce cloud resources with the same names.
No more deployment conflicts. When every developer has a stack of their own, they can deploy to the sandboxed environment as often as necessary, without having to wait for another deployment to finish first.
Everyone on the team has their own production-like environment to use for end-to-end testing.

For Zephyr, the quickest path to adopting this workflow was to use the dev stack as a foundation for new developer stacks. To do that, team members simply ran the following Pulumi commands:

pulumi stack init zephyr/developer-name --copy-config-from zephyr/dev
pulumi up

Developer stacks are considered ephemeral, so they’re torn down with pulumi destroy when not in use in order to cut down on cost. Stack configuration files (e.g., Pulumi.developer-name.yaml) are checked into version control and stored alongside other stacks like Pulumi.dev.yaml and Pulumi.prod.yaml.

Keeping Feature-Branch Workflows

We’ve also seen that the Zephyr team’s existing Git workflow (short-lived feature branches merged frequently into a long-lived base branch) has worked well for them — and it turns out this workflow also works well when combined with the use of developer stacks.

For example, after creating and deploying a new stack to validate and test the proposed Redis upgrade, our Zephyr developer simply included a link to the application (deployed with their dev stack) in the associated GitHub pull request. Reviewers were then able to follow the link to the online store to see the effects of the shopping-cart change firsthand, which meant they were able to approve and get the change merged quickly and confidently knowing it’d work when deployed into production.

Naming Stacks Clearly

Lastly, having adopted the use of developer stacks, the Zephyr team realized the name “dev” probably didn’t work quite as well as it once had for the dev stack. Since the stack and its long-running deployment had essentially been serving (and continued to serve) as an environment for testing changes before releasing them into production, the team decided to rename the dev stack test instead. To do that, they simply ran:

pulumi stack select zephyr/dev
pulumi stack rename zephyr/test

Viewing the Next Iteration of Code

You can view the second iteration of Zephyr’s Pulumi and application code — the iteration that corresponds to the decisions described in this blog post — by navigating to the Zephyr App GitHub repository. From the branch/tag dropdown, switch from the main branch to the blog/dev-stacks branch.

From that GitHub repository, you can also choose to deploy the Pulumi code yourself. Full instructions for deploying the code are found in the repository.

Summarizing Best Practices: Developer Stacks and Git Branches

In this post, we covered several new guidelines for working collaboratively with Pulumi, all aimed at optimizing for high team and developer velocity:

Use short-lived feature branches and merge them frequently into a single base branch.
Deploy the base branch regularly into a long-running pre-prod environment to continuously test the integration of the components of the system.
Enable developers to create and use dev stacks of their own as sandboxes for development, testing, and collaboration.
Name your stacks clearly and in alignment with the environments they deploy into — test, staging, production, etc.

The next Zephyr blog post will dive into how the Zephyr team structures their Pulumi projects. Stay tuned!

IaC Best Practices: Understanding Code Organization & Stacks

Aaron Kao — Mon, 20 Feb 2023 00:00:00 +0000

This is the first in a series of blog posts that explores how a fictional company—Zephyr Archaeotech Emporium—uses Pulumi to manage their online retail store. This post explores a couple of common questions that users ask when working with Pulumi; specifically, where should I store my Pulumi code? And how do I support multiple environments with Pulumi? This post will provide some guidance and Infrastructure as Code best practices around these topics, using Zephyr and their online store as the use case.

The ultimate goal of this series is to discuss recommended practices for using Pulumi to manage a fairly complex containerized application. However, it’s important to note that these practices will emerge over the course of the series—not all immediately, and not all in the beginning. This is a deliberate decision to allow you to see how Zephyr’s use of Pulumi evolves as the company grows and their online retail store application changes to accommodate their growth.

Setting Up the Scenario

Zephyr is short for Zephyr Archaeotech Emporium, the fictional company in our scenario. Zephyr is an online retailer that specializes in the sale of “rare arcane artifacts and replicas.” Over the past few years, the company has experienced a significant increase in its online presence, making it easier for customers to purchase unique and mysterious items. Zephyr’s collection includes a variety of rare and unusual objects that are difficult to find elsewhere, making it a popular destination for collectors, enthusiasts, and adventurers.

Moving to a Containerized Architecture

Zephyr’s application is the second generation of their online store. Their first generation was a monolith that they manually deployed. It was sufficient in Zephyr’s early days, but as word about Zephyr spread and the volume of traffic to their online store increased, Zephyr found it increasingly difficult to deploy small, incremental changes to the store’s code. As a result, the velocity of their development efforts slowed. In preparation for the next phase of their growth—driven by robust sales of arcane artifacts—Zephyr’s team evaluated a number of different architectures that would enable them to increase development velocity and give them greater flexibility in scaling different aspects of their online store. In the end, they settled on a containerized architecture deployed to Kubernetes because some of the existing team was already familiar with these technologies.

Here’s a diagram of their second-generation application architecture:

As you can see in the diagram, the application has been broken down into microservices. The UI service communicates with other backend services via HTTP APIs; the various backend services communicate with various databases like MySQL and Redis for data persistence and/or caching. The backend services are responsible for implementing the functionality of a typical e-commerce site like orders, checking out, and managing the inventory catalog. While the choice of Kubernetes does afford Zephyr some much-needed functionality—like the ability to easily scale different services within the application—it also brings with it an added level of complexity that did not exist with their previous monolithic architecture. The Kubernetes manifest to deploy their application tops out at just over 1,000 lines of YAML!

Astute readers may note that Zephyr’s application looks very similar to this application; in truth, this series uses a fork of the AWS Containers retail store example as the application in this fictional scenario. (Many thanks to the AWS Containers DA team for their hard work here!) This is an admittedly over-engineered application, but it does afford the opportunity to address a range of customer scenarios in this blog series.

Managing Infrastructure and Application Deployments

As part of the switch to their new architecture, Zephyr decided they wanted to use Pulumi to manage both the infrastructure and the application deployment. Why Pulumi? Zephyr’s team recognized that moving to microservices on Kubernetes was going to make it more difficult to create reproducible, consistent implementations of their application. Zephyr’s team also knew that adopting infrastructure as code would help them with fast and repeatable deployments with little additional complexity. Another bonus was being able to use programming languages they already knew.

As they prepared to embark on using Pulumi, a couple of questions came up for the Zephyr team:

Where should they store their Pulumi code? The ability for Zephyr’s developers to use a full programming language to manage the infrastructure and deployment of their online store is a huge plus, but should this code go in the same repository as their application code? Or a different repository?
How do they address the need for multiple environments? It’s clear to the Zephyr development team that they’ll need multiple instances of the online store (for things like testing, QA, or development). What’s the best way to handle this when using Pulumi?

Let’s examine each of these issues in a bit more detail, and see how Zephyr chose to proceed.

Storing Pulumi Code

When it comes to answering the question of where to store Pulumi code relative to the application(s) it supports, there are two basic options:

In the same repository as your application code: This approach has the advantage of being simpler, but may not offer the necessary flexibility when dealing with multiple teams or varying levels of access control.
In a separate repository from your application code: Using a separate repository adds some complexity, but does provide greater flexibility to more easily address a variety of organizational requirements and varying levels of access control.

So which approach is best? That will depend on a number of different factors, many of which are outlined in the organizing projects & stacks documentation. Some other factors that users need to take into consideration include:

Who is responsible for maintaining the code? If the application code and the Pulumi code are managed by different teams, then using separate repositories may be the best approach.
What sort of access controls are needed? If the organization needs different access controls for the application code and the Pulumi code, then separate repositories are generally needed.
What is the relationship between the infrastructure resources managed by Pulumi and the applications? If the infrastructure is “shared,” meaning it is used by multiple applications, then keeping the Pulumi code in a separate repository may be preferred. If the infrastructure is dedicated to a specific application, then keeping the application code and the Pulumi code together may be preferred.
What about CI/CD? If an organization will use a CI/CD pipeline, that may affect how you organize your code. (This is something we’ll delve into more deeply later in this series.)

It’s worth stressing that the answers to these questions may change over time as organizations grow and their applications evolve. Addressing that change and its effects on an organization’s Pulumi projects and stacks is something we’ll be discussing later in this series.

The discussion above is working from the assumption that Pulumi users will store their Pulumi code in a version control system, like Git. Strictly speaking, using a version control system isn’t required for Pulumi to function (there is one exception we’ll touch on later in this series when we discuss Pulumi Deployments), but we very strongly recommend that all users use Pulumi in conjunction with version control.

In the case of Zephyr, their team felt like a monorepo approach (storing Pulumi code and application code together) was most appropriate for their specific requirements. Zephyr is a relatively small organization with a single team of developers who are responsible for managing both the cloud infrastructure as well as the development and deployment of the online store application. Zephyr’s team also decided to use a single Pulumi project—for now. (Be sure to follow this blog series to see how that evolves, and why!)

Addressing the Need for Multiple Environments

The second question Zephyr encountered is how to handle the need for multiple instances of their application. In addition to a production instance—which is the instance behind Zephyr’s public-facing online store—Zephyr also felt they needed an environment for the developers to use in testing changes to the online store.

This use case—needing to have multiple, separate instances of the infrastructure and applications created by a single Pulumi program—is exactly what Pulumi stacks were designed to address. Each stack is a separate instance of the resources created by a Pulumi program within a project. Further, each stack has its own independent state, and each stack has its own configuration values. Stacks can be short-lived (meaning the associated resources are also short-lived), or stacks can be long-lived. Aside from the cloud resources created by the Pulumi program, stacks are lightweight and simple to create or delete.

Zephyr decided to initially start with two stacks: a production stack (named “prod”) and a development stack (named “dev”). As you’ll observe throughout this series, this is a decision that is easily adjusted over time as your organization’s requirements change.

Organize your infrastructure with stacks

Model separate development, staging, and production environments as Pulumi stacks in a single project, each with its own state and configuration.

Get started

Viewing the First Iteration of Code

You can view the first iteration of Zephyr’s Pulumi and application code—the iteration that corresponds to the decisions described in this blog post—by navigating to the Zephyr App GitHub repository. From the branch/tag dropdown, switch from the main branch to viewing the blog/original branch.

From that GitHub repository, you can also choose to deploy the Pulumi code yourself. Full instructions for deploying the code are found in the repository.

Summarizing Best Practices: Organizing Code and Stacks

This article introduced you to Zephyr, the fictional company for this blog series, and reviewed the application architecture for Zephyr’s online store (a containerized application running on Kubernetes). Throughout the blog series, this application serves as the use case around which various recommended practices are discussed.

In this post the following guidelines were covered:

Use version control for your Pulumi code. Although it isn’t strictly required in many cases, a number of the benefits of adopting infrastructure as code are lost without the use of version control.
Place your Pulumi code into version control repositories in a way that aligns with organizational requirements. There is no “right” or “wrong” approach; choose the approach that enables you to satisfy your organization’s requirements around access control, code/repo ownership, and other organizational requirements.
Leverage Pulumi stacks to create multiple instances of cloud resources with a single Pulumi program. Pulumi stacks were created to address the need for organizations to create multiple instances of cloud resources, each with its own state and its own set of configuration data, from a single Pulumi program.

For now, Zephyr has decided to go with a monorepo approach—a single repository that contains both their application code and the Pulumi code to manage the infrastructure resources. All of the resources are defined in a single Pulumi project, with multiple stacks that correspond to development and production environments. Over the course of this series, we’ll see how Zephyr’s use of Pulumi changes as Zephyr grows and their application evolves.

The next Zephyr blog post will examine how the Zephyr team adjusts their use of stacks and Git branches to support developer productivity. Stay tuned!