Pulumi Blog: Platform Engineering Pillars Series

Governance as an Enabler: Scaling Safely and Confidently

Adam Gordon Bell — Tue, 17 Jun 2025 00:00:00 +0000

In previous articles in this series, we’ve explored how platform engineering transforms infrastructure chaos into consistent provisioning, empowers engineering teams through self-service infrastructure, optimizes workflows, embeds security directly into your platform, and provides observability as a superpower. Each pillar builds upon the previous ones, creating a cohesive foundation that accelerates innovation and productivity.

But as your platform scales, new challenges inevitably emerge. You’ve empowered engineering teams with self-service infrastructure, streamlined workflows, and embedded security directly into your platform. But as your platform scales, new challenges emerge: How do you ensure consistency, compliance, and cost control without slowing your teams down?

In this article, we’ll explore how Platform Engineering transforms governance from a manual, bureaucratic process into an automated, built-in enabler, helping your organization scale safely and confidently. By embedding governance directly into your platform, you can maintain control, ensure compliance, and manage costs effectively, all while preserving the autonomy and speed your engineering teams have come to expect.

The Problem: Governance as a Manual Bottleneck

Dealing with manual compliance checks and red tape

With increased team autonomy and self-service capabilities, how do you ensure consistency, compliance, and cost control across your entire organization?

Governance often feels like a necessary evil: manual, bureaucratic, and slow. Application teams see it as red tape, while operations teams struggle to maintain control. Manual compliance checks, lengthy audits, and unclear or inconsistent policies create friction and frustration. Teams may bypass governance processes entirely, leading to shadow IT, inconsistent resource configurations, and hidden risks.

The consequences of manual, bureaucratic governance are clear:

Increased compliance risks and audit failures: Without automated enforcement, compliance becomes reactive and error-prone, increasing the likelihood of regulatory violations and audit findings.
Unpredictable cloud costs and budget overruns: Without clear guardrails, self-service infrastructure can lead to resource sprawl, wasted resources, and unexpected cloud bills.
Reduced team autonomy and slower innovation: Manual governance processes reintroduce bottlenecks, slowing down deployments and undermining the agility your platform was designed to achieve.

The Solution: Embedding Governance into Your Platform

Successfully scaling safely with embedded governance

Governance should live inside your platform, not off to the side as a separate process. To make that happen, build these four capabilities into your IDP:

Policy-as-Code for Automated Compliance: Declare rules (like approved regions or required tags) as code. The platform enforces them whenever infrastructure is created or updated, so compliance happens automatically.
Platform-Level RBAC for Permission Boundaries: Decide who can act on projects, stacks, and templates before any cloud credentials run. This early check prevents unauthorized requests from ever reaching cloud provider.
Audit Logs and Drift Detection for Real-Time Visibility: Record every deployment, who ran it, and what changed. Continuously compare live infrastructure to the desired state in code. If someone bypasses approved processes, the platform flags it and alerts the team.
Resource Lifecycle and Deployment Controls: Automatically retire idle environments after a set time (Resource TTLs) so forgotten test clusters don’t rack up bills. Also if needed, gate production changes behind lightweight approval workflows. Routine dev or staging updates roll out instantly, but high impact production changes wait until a reviewer signs off.

Let’s dive into each of these.

A. Policy-as-Code for Compliance and Operational Standards: Automating Trust and Consistency

An engineering team is ready to launch a new service. They’ve tested it and everything looks good, until the deployment fails. Not because of a bug. Because it’s targeting an unapproved cloud region.

Now they’re stuck. A compliance review kicks off. Slack threads fly. A ticket gets filed. What should’ve been a smooth release turns into a delay, all because of a policy someone missed.

Policy-as-code prevents this.

When teams deploy something that breaks the rules (like using an unapproved region), the platform blocks it automatically. The error shows up right away, with a clear message. Nothing gets provisioned, and nobody has to file a ticket.

If you’re already using intent-based components (“I need a Java service with Kafka and PostgreSQL”), most details are handled for you: tags, regions, naming. But people still override things. That’s why policy-as-code matters.

Think of it as a safety net. A menu of components handle the defaults. Policies catch anything that slips through. Together, they keep your platform consistent without slowing anyone down.

B. Role-Based Access Control (RBAC): Balancing Autonomy and Control

Stopped by lengthy approval processes

As your platform grows, managing permissions manually gets messy. If an engineer needs to fix a production issue but doesn’t have access, they file a ticket and wait, sometimes for days. Give developers too many rights, and they might change production by accident. Both options slow teams down and increase risk.

The fix is an RBAC model built into your platform. First, the platform decides who can deploy, who can publish components, and who can manage templates. This check runs before any cloud credentials are used, so invalid requests get blocked early. Second, the cloud IAM layer controls which API calls are allowed, like creating an EC2 instance or updating a database.

This pairs well with a two-level intent-based approach. Teams describe what they need (“I want a Python Lambda with an SQS queue”), and the platform enforces access only for users with the right scopes. Everyone gets just enough access to do their job, no more, no less.

A Platform with RBAC makes permissions clear, reduces mistakes, and keeps everything auditable. Devs move fast, spinning up resources as needed, while strong guardrails stay in place. The result is a scalable, least-privilege model that balances autonomy and control, so your organization can grow safely.

C. Auditability, Traceability, and Drift Detection: Ensuring Visibility and Trust

Anxious about audit failures and compliance risks

An ops engineer spots a production database misbehaving. A quick check shows someone changed its configuration outside the approved workflow. Without an audit trail or drift detection, the team scrambles to figure out who made the change and when. Meanwhile, the incorrect setting stays active, posing a security and compliance risk. No one can fix it without guessing.

A platform with audit logs records every action: who deployed, when, and what changed. Drift detection watches live infrastructure and compares it to the desired state in code. If someone bypasses the workflow (say, editing a database setting in the console), the platform flags it and alerts: “User Alice changed max_connections on prod-db-01 at 3:42 PM, which no longer matches the expected state.” Now the team can pinpoint the change, talk to the right person, and revert or update the code, restoring consistency in minutes, not hours.

Together, audit logs and drift detection give you real-time visibility into every change. You stop playing detective. You see who did what, when, and how it deviated from code, all in one place. That transparency speeds audits, catches unauthorized changes fast, and builds trust across teams. With automatic traces of every change, your platform scales without surprises.

D. Resource Lifecycle and Deployment Controls: Scaling Responsibly and Safely

An engineer spins up a test environment, then walks away. Another pushes a change straight to production without review. The abandoned test cluster runs up cloud costs; the unreviewed prod tweak risks an outage. Without automation, both lead to wasted spend and stressful cleanups.

A modern platform handles this with ephemeral environments where possible and approval gates where it matters.

In dev and staging, engineers can move quickly. They can create test or preview environments, often tied to users or pull requests, that shut down automatically after a set time. TTL rules keep things tidy without manual cleanup.

Production, by contrast, is gated. High-impact changes, like modifying a database schema or adjusting a load balancer, require approval. Before anything is provisioned, sign-off is required. Every approval (or denial) is logged: who, when, and why.

This setup keeps development fast and flexible, while making production changes deliberate and auditable. Your platform stays clean, cost-effective, and safe without getting in the way.

Drowning in resource sprawl and unexpected costs

Real-World Example: Governance Enablement in Action

An engineering team opens a pull request for a new customer-facing service. The platform spins up a preview environment using a template with secure defaults, pre-approved modules, and policy checks. CI runs tests and policy checks in preview (names, regions, encryption, compliance) so the team catches issues early. If a rule fails (say, an unencrypted database), the PR fails before it reaches main. After the PR merges to main, it deploys to production, confident all policy validations have passed.

Idle QA environments shut down after 48 hours, so forgotten clusters don’t rack up bills. Sensitive production changes, like updating a load balancer or altering a critical schema, are carefully reviewed via pull request. Once approved, the platform deploys automatically and logs every action. Drift detection flags console edits, letting the team revert or update code in minutes.

Result: Governance becomes an invisible safety net. Engineers move fast, knowing policy-as-code, RBAC, TTL cleanup, approval gates, and change tracking catch mistakes. Ops stays in control without firefighting or chasing orphan resources. The platform scales safely, balancing freedom with built-in guardrails.

Metrics: Measuring Governance Enablement

To ensure your governance practices truly empower your organization, it’s essential to track clear, actionable metrics. These metrics help you understand the effectiveness of your governance processes, identify areas for improvement, and ensure governance remains frictionless and enabling:

Time Spent on Manual Compliance Checks and Audits: Measure how much time your teams spend manually verifying compliance or performing audits. Effective governance automation should significantly reduce this overhead, freeing teams to focus on higher-value tasks.
Number of Compliance Violations or Audit Findings: Track how frequently compliance violations or audit issues occur. Effective governance should reduce these incidents, demonstrating that automated policies and guardrails are working as intended.
Cloud Cost Predictability and Budget Adherence: Monitor how accurately your cloud spending aligns with forecasts and budgets. Good governance practices, such as automated tagging, resource lifecycle management, and cost controls, should improve predictability and reduce unexpected cost overruns.
Engineering Team Satisfaction with Governance Processes: Regularly survey engineering teams to gauge their satisfaction with governance processes. High satisfaction indicates that governance is enabling rather than hindering their workflows.

Tracking these metrics helps you continuously improve your platform’s governance practices, ensuring they remain effective and frictionless.

Working within clear, automated guardrails

Pulumi and Governance Enablement

Pulumi provides built-in governance features that help you scale safely and confidently, embedding compliance, consistency, and control directly into your platform:

CrossGuard (Policy as Code) Define and enforce compliance and operational policies automatically. CrossGuard checks every resource against your organization’s standards before deployment, preventing non-compliant resources and reducing manual audits.
Role-Based Access Control (RBAC) and Teams Manage permissions with precision. Pulumi’s RBAC ensures teams get exactly the access they need, no more, no less, so developers can move quickly within clear boundaries and ops can reduce risk.
Audit Logs and Drift Detection Capture a full history of every change and compare live infrastructure to the desired state in code. Audit logs simplify compliance reviews, drift detection spots unauthorized edits, and teams can fix issues in minutes.
Time-to-Live (TTL) Stacks / Ephemeral Environments Spin up short-lived environments for testing or previews. You can assign a TTL to any stack so it shuts down automatically after a set period. That keeps forgotten test resources from racking up costs and ensures your platform stays clean.

By leveraging Pulumi’s governance features, including CrossGuard, RBAC, audit logs, and TTL stacks, your platform becomes a powerful enabler. You automate compliance, maintain consistency, and empower engineering teams to innovate quickly and safely.

Conclusion: Governance as a Platform Feature

Having autonomy with built-in controls

Governance doesn’t have to slow you down. By embedding governance directly into your platform, you empower engineering teams to innovate quickly while ensuring compliance, consistency, and control. Instead of manual checks, governance becomes automatic, transparent, and frictionless, enabling your organization to scale safely and confidently.

Your engineering teams gain autonomy and speed, your operations teams gain visibility and control, and your organization gains the confidence to innovate at scale.

You’ve now seen all six pillars of a modern internal developer platform—provisioning, self-service, developer experience, security, observability, and governance. If you’d like to see how Pulumi makes building and running a platform like this simpler, check out Pulumi IDP.

Observability as a Developer Superpower

Adam Gordon Bell — Tue, 10 Jun 2025 00:00:00 +0000

Frustratedly trying to figure out what's actually happening

In previous articles in this series, we’ve shown how platform engineering turns infrastructure chaos into consistency, gives teams self-service tools, smooths developer workflows, and bakes security into the platform. Each pillar builds on the last. Together, they create an internal developer platform that cuts friction and speeds innovation.

Even so, teams still face a big challenge: seeing what’s really happening. Whether things go wrong or run smoothly, engineering teams need clear, actionable insights into their systems. Without observability, you end up guessing, reacting slowly, and hunting through scattered data.

This article shows how observability can be a superpower, giving teams the visibility, insights, and confidence to build better software. Embedding observability into your platform lets teams spot, understand, and fix problems fast, turning reactive firefighting into proactive innovation.

The Problem: Data Overload Without Insights

Teams drown in metrics, logs, and traces but lack useful insights. In practice, this shows up in three common friction points:

Tool sprawl: Teams use separate tools for metrics, logs, and traces. They waste time flipping between dashboards and stitching data together.
Alert fatigue: Teams get hit with noisy, context-free alerts. With no clear priority or context, key alerts get lost, causing missed issues or slow responses.
Reactive debugging: Troubleshooting turns into a late-night fire drill. Teams spend hours digging through logs and metrics after users have already noticed the problem.

When observability is limited to post-mortems and fragmented dashboards, your team wastes time reacting instead of preventing problems, and innovation grinds to a halt.

The Solution: Observability as an Engineering Superpower

Exhausted from hours of reactive debugging when the problem could have been caught earlier

The solution isn’t just about bolting on more monitoring tools. it’s about baking visibility, context, and guidance into your platform. To do this, embrace three key principles:

Centralized Service Dashboards & Service List Surface every running service (or database, function, etc.) in one “Services” portal, complete with health badges (CPU, error rate), on-call owner info, and one-click links to that service’s metrics, logs, and traces. By unifying all telemetry behind a single service card, you eliminate context-switching and help engineers find exactly what they need in seconds, not minutes.
Actionable Alerts and Insights Replace vague, noisy notifications with context-rich, prioritized alerts that include severity, correlated root-cause data, and recommended next steps (“Database latency jumped 200% since last deploy: rollback or scale up replicas”). Group and surface only the most critical issues first to reduce alert fatigue and speed up resolution.
Embedding Observability into Engineering Workflows Ship every new microservice with built-in logging, metrics, and tracing by including those hooks in your platform’s service templates and CI/CD pipelines. When instrumentation is automatic, “oops, I forgot to add a span” moments disappear, and teams gain immediate visibility into performance and errors from day one.

When observability becomes a superpower, engineering teams gain the visibility and insights they need to confidently build, deploy, and operate software. Instead of drowning in data, they proactively identify and resolve issues, optimize performance, and innovate with confidence.

A. Centralized Service Dashboards & Service List

Reading the Service Catalog

Imagine you’re paged at 2 AM because “OrderService” is failing, but you don’t know where to look. Metrics live in Grafana, logs are in Elasticsearch, traces in Jaeger, and you still have to hunt down who’s on call. You spend precious minutes clicking through multiple UIs and Slack channels just to figure out who owns the service and where its telemetry lives.

A centralized service list solves this by surfacing every running microservice (or database, or function) in one place. In your platform’s web portal, you land on a “Services” page that shows OrderService alongside CPU and error‐rate badges, an on-call owner, and links to its real-time dashboard, filtered logs, and trace waterfall. No matter which team spun it up, you know exactly where to click: metrics, logs, traces, deployment history, and contact info all live behind a single service card.

By embedding a service list into your platform, you eliminate context switching and reduce onboarding friction. If a service isn’t listed, it isn’t properly instrumented, so gaps stand out immediately. In practice, this “one‐pane‐of‐glass” approach means engineers spend seconds finding the right dashboard and the right person, instead of minutes piecing together fragments across disconnected tools.

B. Actionable Alerts and Insights: Reducing Noise and Accelerating Response

Digging through alert noise

It’s 4 AM and your phone buzzes with three simultaneous alerts, each with vague descriptions like “High CPU usage detected” or “Error rate increased.” Without clear context or recommended actions, you must manually investigate each alert, digging through logs and metrics to determine severity and root cause. This manual triage process is slow, frustrating, and error-prone, increasing the risk of missing critical issues or delaying resolution.

Engineering teams often face a constant barrage of alerts, many of which lack clear context, severity, or actionable next steps. This flood of noisy, ambiguous notifications creates alert fatigue, causing your team to overlook critical issues or waste valuable time investigating false positives.

A platform approach should focus on actionable metrics. Instead of vague notifications, clearly state the issue (“Database latency increased by 200%”), provide relevant context (“Latency spike correlated with recent deployment”), and suggest immediate actions (“Rollback recent deployment or scale database resources”). Additionally, alerts are automatically grouped and prioritized based on severity and impact, ensuring you focus on the most critical issues first.

By holistically approaching alerts and insights, you significantly reduce alert fatigue and noise, accelerate incident response and resolution, and empower engineering teams with greater confidence and autonomy. Your team spends less time manually triaging alerts and more time proactively resolving issues, improving reliability, productivity, and overall team satisfaction.

C. Embedding Observability into Engineering Workflows: Visibility from Day One

At 2 AM, where are the logs for this service.

You’ve just deployed a brand-new microservice to production, only to discover performance issues or unexpected behavior. Sure, you should remember to add tracing, logging, and metrics by hand, but in practice, things slip through the cracks. It isn’t until real-world traffic hits that you realize you forgot to instrument X or Y, and now you’re scrambling to retroactively add code, redeploy, and wait for data to appear, delaying resolution and frustrating your team.

If your platform’s service templates already include all the necessary logging, metrics gathering, and tracing out of the box, it makes life a lot easier. Embedding observability into those templates and engineering workflows ensures every new microservice ships with built-in instrumentation.

This proactive approach reduces “oops, I forgot” moments, accelerates troubleshooting, and increases team productivity and satisfaction, ultimately improving the reliability and quality of your software.

Real-World Example: Observability Superpower in Action

Applying an Actional Metric

At 3:15 AM, your pagerduty alert goes off: “CheckoutService latency spiked 150%.” You log into your platform’s Services portal and immediately see CheckoutService highlighted with a red latency badge and Todd Rivera listed as the on-call owner. Rather than scouring multiple dashboards, you click its service card to jump straight to the metrics, logs, and trace views.

The alert itself is remarkably precise: “CheckoutService latency rose 150% at 3:10 AM following the v2.3.1 deployment. PaymentGatewayService upstream error rate jumped from 0.2% to 2.3%. Recommendation: rollback v2.3.1 or scale PaymentGateway pods. Contact Todd Rivera.” Instantly, you know where the problem lies, which upstream service is impacted, and what the next step should be. In the trace waterfall, you spot a 200 ms delay on CheckoutService calls to PaymentGateway. The logs, automatically instrumented by your service template, filter to TimeoutException entries all timestamped at 3:10 AM. Opening the “Ask Platform” AI widget, you type, “Why did CheckoutService latency spike at 3:10 AM?” The AI responds: “Likely cause: v2.3.1 added index idx_created_at to PaymentGateway’s transactions table, causing an 80 ms delay per request. Rollback v2.3.1 or patch queries to remove the new index.”

Armed with this precise diagnosis, you open a rollback pull request and, after Todd’s ok, deploy it within minutes.

CheckoutService latency and PaymentGateway errors immediately return to baseline. By moving from alert to resolution entirely within the platform (thanks to built-in instrumentation, actionable alerts, and AI-driven analysis, you’ve squashed a major incident before most users ever noticed.

Metrics: Measuring Observability Enablement

To ensure your observability practices truly empower engineering teams, it’s essential to track clear, actionable metrics. These metrics help you understand the effectiveness of your observability tools and processes, identify areas for improvement, and demonstrate the tangible impact observability has on your organization.

Key metrics to measure observability enablement include:

Mean Time to Detection (MTTD):
How quickly are issues identified after they occur? Effective observability should significantly reduce the time it takes to detect problems, enabling faster responses and minimizing user impact.
Mean Time to Resolution (MTTR):
How quickly are issues resolved once detected? With clear, actionable insights and unified observability, your teams should resolve issues faster, reducing downtime and improving reliability.
Engineering Team Satisfaction with Observability Tools:
Regularly survey your teams to gauge their satisfaction with observability tools and workflows. Higher satisfaction indicates that observability is effectively embedded into engineering workflows, reducing friction and increasing productivity.
Adoption Rate of Observability Tools and Dashboards:
Track how widely observability tools and dashboards are adopted across teams. Increased adoption indicates that your teams find these tools valuable, intuitive, and helpful in their daily work.
Reduction in Alert Noise and False Positives:
Measure the volume and accuracy of alerts over time. Effective observability should reduce noisy, irrelevant alerts, ensuring your teams focus on meaningful, actionable notifications.

Tracking these metrics helps you continuously improve your observability practices, ensuring they remain effective and empowering. By regularly reviewing and acting on these insights, you can proactively enhance team productivity, reliability, and overall satisfaction.

Pulumi and Observability Enablement

Platform insights

Pulumi’s platform features let you explore observability without bolting on extra tools. Key Pulumi features that enable observability include:

Pulumi Insights:
Provides unified visibility and powerful search across all your cloud resources. Your teams can quickly discover, explore, and understand their infrastructure, eliminating manual searches and reducing cognitive load.
Centralized Service List: Pulumi IDP’s Services portal gives you a single place to register each microservice, database, or cloud resource and link to its dashboards, logs, and traces.
Pulumi Co-Pilot:
Delivers AI-powered troubleshooting and insights directly within your workflows. Your teams can ask natural-language questions about their infrastructure (such as “What infrastructure changed yesterday?”) and receive immediate, actionable answers.
Built-In Instrumentation via IDP Components:
When you author components and templates in Pulumi IDP, you can bake in standard logging, metrics, and tracing hooks. Every service spun up from those templates ships with consistent instrumentation on day one.

With Pulumi, observability can become an integrated part of your platform, accelerating innovation, improving reliability, and empowering engineering teams to confidently build, deploy, and operate software.

Conclusion: Observability as a Platform Feature

Happy resolving an incident in minutes.

Observability isn’t just about plugging in more tools. It’s about baking-in consistent instrumentation, measurement, and context so every engineer (platform, DevOps/SRE, or application) knows exactly where to look and how to act.

Service-Templates with Built-In Telemetry By providing service templates that already include logging, metrics gathering, and tracing, you eliminate “Oops, I forgot to instrument X” moments. Every new microservice inherits a standard setup, so you never have to retroactively add code or scramble when traffic first hits production.
Consistent Service Dashboards & Centralized Service List Instead of hunting across eight different dashboards, engineers always start from a single “Service List” page. From there, one click takes them to that service’s metrics overview, log stream, or trace waterfall. This unified entry point reduces cognitive load and cuts straight to “Where’s the problem?”
Measuring Alert Quality and Actionability A truly mature platform doesn’t just send alerts. It tracks whether those alerts are helpful or noise. By measuring “ratio of actionable alerts vs. false positives,” you continuously fine-tune thresholds and eliminate alert fatigue. The result? Engineers trust their notifications and respond faster to real incidents.
AI-Driven Context and Natural-Language Troubleshooting On top of unified telemetry and alert quality metrics, AI can instantly correlate recent deployments, configuration changes, and error spikes. Engineers can ask, “Why did latency jump at 3 AM?” or “What changed in production last night?” in plain English, and the platform provides a clear, context-enriched answer. This additional layer turns reactive firefighting into proactive problem prevention.

When you combine these elements (components and templates, a single service dashboard, alert quality measurement, and AI/natural-language querying), you transform observability into a genuine superpower. Issues are spotted, triaged, and fixed before customers even notice.

Next time, we’ll dive into the final pillar, Platform Governance, showing how to enforce policy, manage costs, and keep your platform secure and compliant as it scales.

Security as an Enabler: Building Trust into Your Platform

Adam Gordon Bell — Fri, 11 Apr 2025 00:00:00 +0000

In previous articles, we looked at how platform engineering fixes infrastructure chaos, enables self-service, and improves developer workflows. These pillars work together to boost both developer productivity and organizational speed.

But there’s still one critical element that can make or break all this progress: security.

Traditional security efforts — even “shift-left” initiatives — often create friction instead of clearing the way for innovation. Embedding security directly into your platform changes that. By weaving in policy-as-code, centralized secrets management, and identity-based authentication, you turn security from a blocker into an enabler. And with the right metrics, you can measure how well your platform balances protection and speed.

When security is baked into your internal developer platform, it stops being a gatekeeper. Developers move faster — and with more confidence.

Done right, your teams gain autonomy without compromising safety. Security people become partners instead of blockers. Your organization builds trust that speeds up innovation rather than slowing it down.

The Problem: Security as a Gatekeeper

Traditional security practices often create friction rather than enable innovation. Developers run into several pain points: last-minute security reviews that delay deployments, unclear policies that feel arbitrary, and security teams that act more like blockers than partners.

This friction has real consequences. When security becomes a bottleneck, developers find ways around it. They take shortcuts. All of this increases risk through inconsistent security practices.

The end result is that innovation slows down. Developer autonomy shrinks. And the agility your platform was supposed to deliver disappears.

Many organizations try to fix this by “shifting security left” - moving security checks earlier in development. But just dumping security responsibilities onto developers isn’t the answer. It overwhelms them and can still create inconsistencies.

The real solution isn’t just about changing when security happens. It’s about changing how it works.

Key Security Functions within a Platform

Effective platform security involves several key functions, regardless of how they are distributed across teams or individuals in your organization:

Defining Security Strategy & Policy:
- Developing and maintaining security policies, standards, and architectural best practices tailored to the organization.
- Providing security guidance, education, and support across engineering teams.
- Performing threat modeling, security architecture reviews, and proactive security assessments.
Monitoring & Responding to Threats:
- Actively monitoring systems to detect security events and potential threats in real-time.
- Managing security tooling (like SIEMs, vulnerability scanners) and orchestrating incident response procedures.
- Investigating incidents, coordinating remediation efforts, and managing security automation.
Integrating Security into Development & Infrastructure:
- Embedding security controls directly into the development lifecycle and infrastructure provisioning processes.
- Implementing secure-by-default configurations, automated security checks (SAST, DAST, SCA), and policy-as-code guardrails.
- Ensuring security considerations are built into CI/CD pipelines, service templates, and infrastructure modules from the start.

Collaboration between these functions is crucial. A successful platform ensures that policy definition, operational monitoring, and practical implementation work together seamlessly to create a secure environment without hindering velocity.

The Solution: Embedding Security into Developer Workflows

The key to solving security friction isn’t just adding more checkpoints earlier. It’s weaving security directly into your platform and your developers’ daily work.

Adding automated scans to your CI/CD pipeline is a good start. Tools like Dependabot, Snyk, and Trivy catch problems early and alert developers quickly. But these “shift-left” checks alone aren’t enough. Developers still need to interpret findings and fix issues, often without adequate support.

Platform engineering takes a different approach. It builds security in from the start, rather than treating it as a final hurdle or dumping complex responsibilities on developers. It uses secure-by-default components and automated guardrails that work continuously.

Let’s see what this looks like in practice.

Policy-as-Code for Security: Automating Trust

If you’ve implemented the intent-based approach we covered earlier, your developers already use secure-by-default modules. They simply request what they need (“I need a Java service with Kafka and PostgreSQL”), and the platform handles security details like IAM roles, encryption, and permissions.

But secure defaults aren’t enough. You still need automated guardrails. Policy-as-Code adds this enforcement layer, keeping security standards consistent—even when someone tries to bypass or change the modules. It’s a backup system: secure modules provide your first defense, and Policy-as-Code catches anything that might slip through.

For Example:

IAM Roles with Least Privilege:
- Secure Default (Module): The platform team’s IAM module automatically generates roles with least-privilege permissions. Developers don’t specify IAM roles directly.
- Policy-as-Code (Additional Guardrail): A policy explicitly checks all IAM roles provisioned by the platform, ensuring no role grants overly broad permissions (e.g., no wildcard permissions). If a module is accidentally modified or misconfigured, the policy catches it immediately.
Mandatory Encryption at Rest:
- Secure Default (Module): The platform team’s database module automatically provisions databases with encryption at rest enabled. Developers don’t specify encryption settings directly.
- Policy-as-Code (Additional Guardrail): A policy explicitly checks all storage resources to ensure encryption at rest is always enabled. If a developer or platform engineer accidentally modifies the module or bypasses it, the policy immediately flags the issue.

When you add Policy-as-Code to your platform, security becomes automatic and clear. Developers get instant feedback they can trust, and security teams maintain control without slowing things down. Your teams can move fast without taking shortcuts that create risk.

Protecting Sensitive Credentials by Default

Policy-as-Code secures your infrastructure, but you still need to protect how applications access sensitive resources like databases and APIs.

Here’s what often happens: A developer copies database passwords into a local .env file. They accidentally commit this file to Git, exposing the credentials to anyone with repository access. Or they inadvertently bake these secrets into a Docker image that others can download. Worse yet, if their laptop gets hacked, attackers can use these credentials to break into production systems.

When these breaches happen, everything stops. Teams scramble to rotate credentials, investigate the damage, and fix systems—creating delays and increasing risk.

Centralized secrets management provides a robust solution to credential security. It stores sensitive information in a secure vault and offers controlled access methods at runtime. Developers work with the secrets management solution instead of handling credentials directly, which significantly reduces exposure risk.

Platform teams build these secure patterns into their service templates. They create clear pathways for accessing credentials that guide developers toward secure practices in both development and production environments.

With this approach, your deployments become both faster and safer. Developers follow simpler, more consistent security procedures, while security teams gain confidence that credentials are properly protected throughout your systems.

Identity and Access Management: Frictionless Security through Identity

Your platform must also solve another security problem: how people and systems safely access cloud resources.

Developers hate juggling multiple credentials—cloud provider keys, passwords, or tokens. These credentials can expire, get lost, or leak, causing headaches and security risks.

Consider a common scenario: A developer trying to fix a broken production system finds their credentials have expired. Now they’re stuck hunting for new credentials instead of fixing the problem.

Identity-based authentication fixes this. Developers just use their regular login. When developers log in normally, they automatically get secure access to cloud resources through OIDC or SAML. No more managing credentials. Access stays secure and current.

This approach simplifies access and removes barriers. Developers get secure access to resources when they need them, speeding up work without weakening security.

Identity-based authentication, combined with policy-as-code guardrails and centralized secrets management, creates a comprehensive security system that protects without impeding work. To see how these practices function together, let’s look at a real example.

Real-World Example: Security Enablement from Build to Incident Response

A developer deploys a new microservice. They select a secure template from the service catalog. Policy-as-code instantly checks that encryption and permissions are set correctly, and credentials are setup in a centralized secrets management environment.

As the service builds, automatic scans check dependencies and container images for vulnerabilities. The system creates a dependency list (SBOM), tracking every component used. Build pipelines verify and sign code, so only trusted code reaches production.

Once deployed, security monitoring begins automatically, showing the service’s security status in real time. Later, someone discovers a vulnerability in a library used by the microservice. Thanks to the dependency list, the security team immediately knows which services are vulnerable. Monitoring spots suspicious activity and sends clear alerts to the team’s dashboards.

Clear incident response plans help teams contain and fix the problem. The team patches the vulnerable component, using the same secure pipelines to quickly roll out the fix.

In this scenario, security helps rather than blocks at every stage. From building to responding to incidents, your platform helps teams build secure software quickly.

But how do you know if your security practices are truly enabling your teams? To answer that, you need clear, actionable metrics.

Metrics: Measuring Security Enablement

To check if your security approach works, track two things:

Security Incident Rate:
Count how many security problems reach production. Fewer incidents means your security is working.
Developer Security Friction Score:
Include security questions in your regular developer surveys. Ask how much security processes slow their work or cause frustration. Improving scores show that security fits naturally into their workflow.

These two measures combined tell you your security works and at what cost, so your team can build good software fast.

Pulumi and Security Enablement

Achieving these goals, a decreasing incident rate coupled with a low developer friction score, requires a platform built with the right foundations. Pulumi helps you construct such a platform by providing built-in capabilities that directly embed security into your infrastructure workflows:

Policy as Code (CrossGuard): Automatically enforce security and compliance standards.
Secure Secrets Management (Pulumi ESC): Centralize and securely inject secrets without manual handling.
Identity-Based Authentication: Simplify secure access to cloud resources using existing identities.

With Pulumi, security becomes an integrated, frictionless part of your platform—accelerating innovation while building trust.

Security as a Platform Feature

Security can help, not hinder. By building security into your platform, you help developers work faster and safer. Security becomes a feature, not an obstacle.

Security completes the platform foundation we’ve built in previous articles on Provisioning, Self-Service, and Developer Experience, creating a platform that speeds up development.

But developers still need to see what’s happening in their systems. Next time, we’ll explore how Observability helps developers spot and fix problems quickly.

Improve Developer Experience: Increase Dev Productivity with Internal Developer Platforms

Adam Gordon Bell — Thu, 13 Mar 2025 00:00:00 +0000

In the last article in this Platform Engineering Pillars series, we explored how self-service infrastructure frees developers from bottlenecks and dependency gates. By providing reusable infrastructure modules and intent-based configurations, platform teams dramatically reduce infrastructure friction. This self-service model powers faster deployments, increased autonomy, and fewer delays.

However, infrastructure provisioning alone isn’t enough to improve developer experience. Even with efficient provisioning, developers can still face inconsistent local setups, sluggish CI/CD pipelines, poor documentation, and fragmented tooling. These obstacles quietly reduce developer productivity, slow developer velocity, and increase operational overhead.

Your platform’s ability to support daily workflows directly impacts core success metrics like time to first commit, time to production, and developer satisfaction.

When improving developer experience as part of a platform engineering initiative, measure:

Time to First Commit – How quickly a developer can become productive in a new environment or project. Pay particular attention to new hires.
Time to Production – How fast code moves from commit to successful production release.
Developer Satisfaction – Regularly survey friction points and prioritize fixing the top issues.

Friction logs are also a powerful way to identify roadblocks, recording every obstacle encountered during common workflows. This reveals hidden friction and provides a roadmap for improvement.

A great developer experience accelerates iteration, improves problem-solving, and drives better products. An internal developer platform (IDP) designed for DevEx is a fundamental competitive advantage.

The Service Catalog: Foundation for Developer Experience

Developer friction often starts with something simple: not knowing what’s already available or who owns it. Without a single, reliable source of truth, developers reinvent the wheel, duplicate work, and waste time chasing information.

A well-structured service catalog in your internal developer platform addresses this by providing a central repository for all services and applications. At its most basic, it’s just a web page with each service’s README, service metadata, and operational information.

But, properly structured, this delivers some large benefits:

Context at a Glance: Every catalog entry follows a standardized template, capturing core information such as purpose, interfaces (REST, gRPC, events, etc.), language/runtime, and environment requirements. New developers can quickly grasp a service’s role and easily jump into productive work.
Discoverability and Reuse: Without a catalog, teams often recreate services they don’t realize already exist. Duplicate services not only waste resources but lead to fragmented knowledge and inconsistencies. With a catalog, developers quickly find and leverage existing solutions, increasing reusability and reducing churn.
Ownership Transparency: Each entry in the catalog explicitly names team ownership, CODEOWNERS files, and pertinent contact details like Slack channels or on-call rotations. Clear ownership streamlines communication, issue resolution, and collaboration, saving precious time and avoiding frustrations.

A service catalog supports autonomy, reduces wasted cycles, and increases developer velocity by making essential information instantly accessible.

But a Service Catalog alone isn’t enough. To fully streamline and enhance your development workflow, you must pair it with consistent local environments and reliable CI/CD processes. Both of which can be serviced by standardized service templates.

Service Templates: Golden Paths for Improving Developer Productivity

Earlier in this series, we introduced service templates as a way to scaffold consistent, ready-to-go services. In an internal developer platform, templates aren’t just for infrastructure, they’re the golden paths that define how developers start and succeed.

If 90% of new services are Go GRPC services, then a carefully crafted, opinionated go GRPC service blueprint will remove a lot of friction. It will give developers everything they need to begin delivering real business value immediately. A few select curated options of common starting points will help guide developers to generate projects complete with standardized structure, built-in quality checks, documentation, and clear guidelines for contribution.

A great service template provides:

Project Bootstrapping

The template quickly scaffolds an entire service—including all necessary files and directories. Everything essential for your organization’s standards should be included, from CI/CD configurations and dependency management to directory structures and environment-specific configuration files.

Built-in Quality Tools

Templates embed guardrails that maintain consistency and high-quality standards across teams:

Linting rules and formatters: Tools like ESLint, flake8, or Checkstyle that ensure code style and consistency across your codebase.
Testing frameworks: Unit tests, integration tests, and contract testing tools set up and pre-configured (JUnit, pytest, Jest, Pact), helping teams maintain high quality right out of the gate.

Documentation and Contribution Guides

Many platform teams overlook how clear, consistent documentation is crucial for initial and ongoing developer productivity. Every new service should arrive out-of-the-box with:

README: Describing at a glance what the project does, why it exists, and how to run it.
Documentation: Service template establishes a place for service-specific documentation that will be surfaced in the Service catalog. Markdown explanations, example input/output, and real-world use-cases.
Contribution guidelines and coding standards: Clearly documenting the expectations, conventions, and standards every developer can follow with confidence.

Embedding this documentation into templates ensures that even brand-new services are discoverable, understandable, and maintainable — right from their first commit.

A great template means the service catalog becomes more than just an organizational listing. These thoughtfully designed templates can directly power it. Every new service entering your catalog aligns neatly with your organization’s best practices in code quality, documentation, and process standards.

When paired with your service catalog, templates ensure every new service aligns with best practices — boosting developer productivity and developer velocity from day one.

Streamlining Local Development to Reduce Friction

One of the fastest ways to improve time to first commit is to standardize local development workflows through your internal developer platform.

Your service templates provide the ideal mechanism for simplifying how quickly developers can get projects running locally. By embedding standardized local-development tooling into these templates, you ensure they’re available consistently across your service catalog. Good templates typically contain:

A preconfigured, containerized environments: Using Dev Containers, Docker Compose, or similar tools lets developers launch an environment that closely resembles production with minimal friction, completely avoiding dependency drift or local environment inconsistencies.
Simple, standardized command runner: Makefiles, just scripts, package.json scripts—whatever works as they’re consistently documented and easy to run commands such as make build, make test, or make lint. Teams might diverge on preferred tooling over time; if so, simply record clearly in the project’s readme (and thus in your service catalog) exactly what’s needed.
Built-in documentation: Each service template includes clear instructions in a README outlining step-by-step processes for setup, running tests, and other everyday developer tasks.

With these pieces fully integrated into your catalog and templates, you significantly reduce friction and speed up development cycles, moving you meaningfully closer to improving your core developer-experience metrics.

The next critical area we need to address is how smoothly and quickly code moves from a local commit to a production release.

CI/CD Integration: Fast, Reliable Pipelines for Experienced Developers

Experienced developers cite unreliable or slow pipelines as major productivity killers. Their frustrations often revolve around flaky builds, unreliable tests, and cumbersome environment setups. Slow or unpredictable pipelines break developer flow, causing frustration, delays, and lower productivity.

A strong internal developer platform integrates fast, stable, and predictable CI/CD processes:

Stable, Fast Feedback:
Every new service comes pre-configured with build acceleration strategies like intelligent caching (Gradle, Bazel, Docker layers) and automatic parallel test execution. Stable and speedy pipelines help teams iterate quickly and confidently.
Ephemeral, Self-service Environments:
Instead of battling shared staging environments – which get blocked by other teams— developers spin up short-lived testing environments directly from pull requests. Need to test your payment-service changes against the latest user-authentication service? Create a dedicated, temporary environment on-the-fly, validate interactions, then shut it down automatically when finished.
Test Reliability:
Strictly enforce policies to quarantine flaky tests, quickly escalate notifications to responsible developers, and provide clear paths for fixing instabilities. Proactive flakiness management ensures credibility and reliability for test pipelines over time.

Reliable CI/CD pipelines protect developer focus, reduce cognitive load, maintain developer velocity, and prevent wasted time, increasing developer productivity.

Developer experience is a vast, nuanced topic, and there’s much more a comprehensive platform can include than we’ve been able to explore fully here. Even just within your Service Catalog and developer workflows, valuable additions might include:

Onboarding Guides: Step-by-step guides rapidly bringing new developers up to speed on teams, tools, and processes.
Frictionless Product Backlog Integration: Quick access from codebases to the product backlog to connect code directly with the business context.
Code Coverage & Quality Dashboards: Easy-to-use visualizations offering fast insights into quality metrics and test coverage.
Deprecation Notices & Change Management: Automated notifications about changes, upcoming maintenance, or decommission plans to keep developers informed and prepared.
Pre-built Common Components: Shared libraries and components solving everyday needs like logging, authentication, validation, and configuration.

And that’s truly just scratching the surface. But remember: Platform engineering is an iterative journey, you don’t need to tackle every single area all at once. If you start with friction logs and surveys to regularly identify and address pain points affecting daily workflows, you’ll already be heading confidently in the right direction.

Conclusion: IDPs as a DevEx Multiplier

A great developer experience is not optional — it’s a force multiplier for developer productivity and developer velocity. An internal developer platform with service catalogs, golden path templates, streamlined local dev, and integrated CI/CD removes friction at every step of the developer journey.

The result? Faster onboarding, quicker delivery, and happier teams.

With Pulumi, platform teams can automate consistent environments, implement golden paths, and enable self-service infrastructure that scales with your organization.

Organization Templates enables standardized service creation and consistent developer experiences
Stack References for managing dependencies between environments and services
Pulumi Deployments to streamline CI/CD workflows and enable self-service infrastructure
Review Stacks for creating ephemeral, on-demand testing environments
Automation API for programmatically managing infrastructure and implementing platform workflows

By prioritizing developer experience in your platform engineering strategy, you empower teams to move faster, build better, and maintain momentum, making your internal developer platform one of the most valuable assets in your organization.

Self-Service Infrastructure: From Tickets to Tools

Adam Gordon Bell — Thu, 06 Mar 2025 00:00:00 +0000

Previous articles in this series explored platform engineering principles and how Infrastructure as Code creates a solid foundation. But there’s still an important challenge to address: the infrastructure provisioning process itself. Without proper modularity and a clear separation between intent and infrastructure details, things get messy—leading to friction, delays, and unnecessary complexity.

The Missing Layer: Abstraction Through Modularity

The root issue here is how teams define and work with infrastructure. Without good abstraction layers, infrastructure code typically becomes unwieldy and complex.

You end up with

Highly detailed specifications that require deep domain knowledge
Environment-specific configurations with subtle variations across resources
Complex interdependencies that are difficult to trace and verify

This approach burdens developers with low-level details they don’t need to worry about, while platform teams become gatekeepers, reviewing intricate configurations. Alternatively, a traditional ticketing model takes over, discarding the benefits of modern practices.

Solution: A Platform Engineering Approach

What’s missing is a proper abstraction layer between raw infrastructure code and developer needs. That layer is the platform. A Platform for self-service infrastructure. Without modular, reusable infrastructure patterns, every new application deployment becomes an exercise in starting from scratch and defining every detail instead of just stating what the application requires and deferring details to the platform.

Common Anti-Patterns Without Proper Abstraction

Without this modular platform approach to cloud infrastructure, organizations typically fall into one of two problematic patterns.

The Copy-Paste Anti-Pattern

Developers copy infrastructure code from previous projects. They find something similar, duplicate large chunks of configuration, make a few changes, and hope everything is right, and submit it for review. This leads to a brittle system with siloed knowledge. Security fixes must be applied manually across every instance – no one has a full picture of the infrastructure – and Day 2 Operations become error prone.

And this isn’t limited to infrastructure: CI scripts get copied and modified, Dockerfiles and Kubernetes manifests get copied and modified.

It feels like control, but it’s not. Developers still rely on platform specialists to validate changes and catch mistakes.

The Ticketing Anti-Pattern

Alternatively, some organizations rely on rigid ticketing systems for infrastructure changes. A platform specialist applies writes and applies all changes, ensuring consistency — but also creating bottlenecks. Developers might wait days or weeks for simple modifications, leaving the process entirely unfinished.

This guarantees expert oversight but strips developers of autonomy, making the platform team a bottleneck for every deployment.

Neither anti-pattern strikes a balance between standardization and autonomy, leading to either sloppy infrastructure (copy-paste) or slow deployments (ticketing).

The Resulting Problems

Day 1 Bottleneck: Slow, Manual Infrastructure = Developer Gridlock

When provisioning becomes a bottleneck, developers with finished code wait days for infrastructure. This disrupts focus and forces frequent context switching. They juggle tasks to stay productive and then have to rebuild focus when the requested infrastructure is ready.

These delays slow the entire delivery pipeline. Teams add buffer time for infrastructure delays, projects slip, and the business sees technology as slow-moving. Meanwhile, the real culprit is slow provisioning that kills momentum.

Day 2 And Beyond Chaos: Sprawling, Unpredictable Infrastructure = Operational Nightmare

Day 1 bottlenecks frustrate developers, but the real long-term cost is patching hell. Copy-paste infrastructure creates hundreds of snowflakes, each needing manual attention for every security fix or version bump. One-off configurations and custom environments pile up, creating technical debt no one fully understands.

Operations teams are overwhelmed by the basic, critical task of simply keeping the lights on across a sprawling, inconsistent landscape.

The Solution: A Platform of Options and Self Service

A Self-service platform is the solution to these problems. The platform is this curated list of modules, providing application developers with a set of pre-built options that let them work at a higher level of abstraction. By choosing from these modules, developers express their intent for infrastructure, leaving the platform to handle the implementation details. Let me explain.

A strong self-service approach divides responsibilities into two clear layers:

Level 1: Platform Team (Module Definition)

Builds and updates reusable infrastructure modules
Bakes in security, compliance, and best practices
Hides complexity so developers don’t need to worry about copy and pasting low-level details
Sets guardrails for scalability, security, and compliance
Updates modules as technologies and needs change

Level 2: Application Developers (Module Consumption)

Browse a catalog of ready-made modules
Compose pre-built modules to set up application infrastructure
Spend time on code, not cloud configurations
Operate within built-in guardrails
Deploy applications faster using standardized components

This setup keeps things fast and flexible. Developers get the autonomy to build without reinventing infrastructure, while platform teams maintain control where it matters.

The Power of Intent-Based Specification

Layer of Abstraction to the Rescue

The two-level approach works by replacing over-specification with intent-based design.

Over-specification happens when developers specify too many infrastructure details that:

Require expertise they might not have
Tightly coupled infrastructure to specific implementations
Make future changes hard or even impossible

Intent-based specification means developers state what their app needs, not how to implement it, meaning:

Platform teams can update underlying infrastructure without breaking applications
New capabilities can be added without disrupting existing applications

Focusing on what developers need, not how it’s built, creates a system that’s easier to maintain and evolve for both platform teams and developers.

Let’s walk through an example.

Practical Example: Java Application with Kafka, PostgreSQL, and Redis

The platform team provides reusable modules:

Java Application Module

This module handles everything needed to run a Java app. It covers the Java runtime, memory settings, JVM options, scaling rules, health checks, monitoring, and Kubernetes namespacing. It also manages containers, security scans, resource limits, and logging integration—so developers don’t have to.

Kafka Integration Module

Similarly, this module simplifies Kafka integration, handling broker connections, authentication, topics, and security. It takes care of SASL/SSL authentication, replication factors, ACLs, and encryption. Developers just specify topics and schemas.

And so on for common infrastructure components used in the org.

Level 2: Developer Implementation

Developers use a simpler interface. To deploy a new Java service with Kafka, PostgreSQL, and Redis, they only need to specify the required input parameters in the platform-defined modules:

# Developer's intent-based configuration
application:
 name: order-processing-service
 module: java-application
 version: 17
 memory: 2GB
 scaling:
 min: 2
 max: 10

integrations:
 - module: kafka
 topics:
 - name: incoming-orders
 partitions: 5
 - name: processed-orders
 partitions: 3
 consumer_group: order-processors

 - module: postgresql
 database: orders_db
 size: medium
 backup: enabled

 - module: redis
 size: small
 ttl: 3600

( Note: YAML not required. This pseudo-code could be UI driven, or it could be written in your programming language of choice. The point is it specifies intent and defers specifics to the platform. )

Solving Day 1 and Day 2 Challenges

The intent-based specification solves immediate and long-term problems:

The Self-Service Transformation in Action:

Developer Experience	Platform Response
Submits simple intent-based configuration	Expands into complete infrastructure
Focuses on application needs, not infrastructure details	Provisions resources with appropriate sizing and security controls
Specifies “what” is needed, not “how” to configure it	Handles networking, monitoring, backups, and credentials
Limited expertise required in Kubernetes, networking, or security	Implements best practices automatically across all deployments
	Maintains and updates the underlying implementation over time

This approach transforms infrastructure from a developer burden into an organizational asset that becomes more valuable over time, rather than decaying into technical debt.

Developer Interaction Models: Choosing Your Platform Interface

Hopefully, now you are bought in on the value of separate layers of abstraction and building a platform to expose those.

The next key decision is how developers will actually use this platform. The ideal interface balances ease of use with the necessary flexibility. Organizations and requirements vary, so a spectrum of interaction models exists, each with its own trade-offs. Tree common approaches are: CLI-Driven Infrastructure as Code, the Self-Service Portal, and a Hybrid model.

Aspect	CLI-Driven	Self-Service Portal	Hybrid Approach
Developer Experience	Code-based, using IaC modules and components directly	Click-based UI with forms and catalogs	Portal for common tasks, code for customization
Code Visibility	Full visibility and control	Limited or no visibility	Visible but with varying levels of access
Governance	Trust-based or PR reviews	Enforced through portal constraints	Guided paths with managed exceptions
Learning Curve	Higher (requires IaC knowledge)	Lower (minimal infrastructure knowledge)	Moderate (basic UI with optional advanced use)
Best For	Experienced teams needing flexibility	Teams prioritizing speed and standardization	Organizations with diverse skill levels

1. CLI-Driven Infrastructure as Code: Empowering Developers with Familiar Tools

In this approach, platform teams build reusable infrastructure modules while developers consume them directly through code. Developers use familiar IaC tools like Pulumi CLI to instantiate these modules, with changes flowing through version control like any other code.

The key strength is transparency and familiarity. Developers maintain visibility and ownership of their intent-based definitions. This code-centric workflow integrates naturally with existing development practices.

However, this flexibility creates governance challenges and requires either strong trust or approval processes that might reintroduce bottlenecks. The approach also assumes developers have some IaC knowledge, making it best suited for organizations with mature DevOps practices and experienced teams.

2. Self-Service Portal: Click-Ops Simplicity for Rapid Provisioning

The self-service portal approach creates a web interface where developers provision infrastructure without writing code. They simply browse a catalog, fill out forms, and click buttons to deploy standardized resources—all pre-configured by the platform team.

This approach keeps infrastructure code completely centralized and hidden from developers. The portal translates simple selections (“Add Database: Medium Size”) into the appropriate underlying configurations, ensuring perfect compliance with organizational standards.

The result is dramatically reduced friction and learning curve, enabling even infrastructure novices to deploy resources in minutes. However, this simplicity comes at the cost of flexibility. Customizations and unique needs typically require platform team involvement, making this model ideal for organizations with predictable workloads or where standardization outweighs customization needs.

3. Hybrid Approach: Code-Backed Self-Service

The hybrid approach generates and modifies infrastructure code via pull requests from UI interactions. When developers configure resources through the portal, the system commits code to repositories automatically, creating a transparent workflow where changes are visible, reviewable, and auditable through standard Git processes.

This model provides both simplicity and flexibility. Routine tasks remain point-and-click while complex scenarios allow direct code editing within the same PR workflow. The approach maintains a complete audit trail through commit history while giving platform teams visibility into all changes.

A significant tradeoff is implementation complexity: building a system that generates commits and PRs based on version control integration requires substantial investment. Organizations must weigh this upfront cost against the long-term benefits of combining UI simplicity with code transparency.

Lifecycle Management

More effective lifecycle management is an advantage of a platform approach. Consider migrating from Kafka to AWS Kinesis. Traditionally, this would involve significant application developer effort, including updating connection details, authentication, and application logic. However, with the two-level abstraction and intent-based approach described earlier, this migration becomes simpler. Platform teams introduce a new Kinesis module matching the existing Kafka module’s intent-based interface. Data is mirrored, and developers can plan a switchover. Migration will not be free, but efforts and coordination will be reduced.

Life-Cycle management is a huge topic in Day 2 operations and outside the scope of what can be covered here, but introducing abstractions and a platform of modules will pay off long term.

Looking Ahead: Operational Excellence and Beyond

The two-level abstraction model does not just simplify provisioning. It also lays a strong foundation for operational excellence. By standardizing infrastructure through reusable modules and intent-based specifications, we create consistency and visibility that significantly simplify critical operational concerns such as:

Drift Detection and Reconciliation: Standardized modules make it easier to detect and correct deviations from desired state.
Continuous Compliance and Security: Centralized module definitions allow security and compliance controls to be consistently enforced and audited.
Cost Management and FinOps: Intent-based specifications enable clearer cost attribution and optimization opportunities.
Observability and Incident Response: Consistent infrastructure patterns simplify monitoring, alerting, and troubleshooting.

Later articles in this series will explore each of these areas in greater depth, demonstrating how a platform approach provides a powerful foundation for addressing them effectively.

Self-Service with Pulumi

The two-level architecture of IDP self-service creates a powerful interface between platform teams and application developers. By embracing intent-based specification, this approach enables developers to clearly express what they need with minimal complexity while giving platform teams the flexibility to implement and evolve the underlying infrastructure.

As you build out your self-service approach, consider how Pulumi might fit in. With Pulumi, you get:

Component resources for creating reusable infrastructure modules that encapsulate best practices.
Organization templates for building a catalog of infrastructure templates.
Packages in your language of choice for a code-first self-service workflow.
A Platform-in-the-box experience for those who want a portal-first experience for self-service infrastructure.
A Backstage plugin for those who prefer to use Backstage for an IDP.

A great self-service workflow and abstraction layer are key to making platform engineering work. Next in this series, we’ll tackle developer experience directly.

Provisioning: From Chaos to Control

Adam Gordon Bell — Thu, 27 Feb 2025 00:00:00 +0000

Provisioning is the first pillar of platform engineering. Without consistent infrastructure provisioning – the automated creation and management of the underlying cloud resources – the rest of the platform suffers. Self-service, governance, and streamlined developer workflows all depend on it. Ultimately, a self-service layer on top of your cloud infrastructure is the goal, enabling developers to quickly and safely provision the resources they need, while adhering to organizational best practices and policies. But before self-service, the foundation of a good IDP is a robust and reliable provisioning system.

By defining cloud resources as code and automating deployments, platform engineering teams ensure every environment – development, staging, and production – stays consistent and maintainable. This cuts down on configuration drift, reduces manual work, and supports auditable, collaborative workflows for every change.

Let’s explore how platform engineering teams can achieve this by version-controlling infrastructure, automating deployments, separating environments properly, and limiting console interventions. By applying these principles, teams can create a platform where developers can move fast without breaking things, and where infrastructure supports innovation rather than slowing it down.

Top 5 IaC anti-patterns

Console-Only Changes: Making “quick fixes” in production without updating IaC leads to drift, confusion, and hidden risks.
Hard-Coded Secrets: Sensitive values in code or config files are a security liability. Use encrypted state or a secrets manager.
Copy-Paste Config: Reusing half-baked snippets across projects creates cruft. Break out shared modules or templates.
No Clear File Structure: Throwing all configs into a single file or folder obscures dependencies. Organize by environment, service, or module.
Skipping Peer Reviews: Infrastructure mistakes can be costly. A simple pull request process can catch errors before deployment.

💡 Tip: Read our list of IaC Best Practices 💡

Building a reliable IaC foundation

It’s not enough to simply define your resources in code. A reliable IaC foundation should ensure that infrastructure changes move smoothly from idea to production, without surprising developers or straining operations. Here’s how to structure these workflows so they’re both transparent and resilient:

Version control as the single source of truth

Whether you’re using GitHub, GitLab, or something else, every infrastructure definition needs to live in version control. This is just the basics – and hopefully, you are already doing this, as you have been for years. Version-controlled changes allow you to track history, compare revisions, and peer review. A pull request might catch a misconfigured subnet or a database that was accidentally left publicly accessible, mistakes that could be catastrophic if they slip into production. This also means anyone can propose improvements or bug fixes, expanding the pool of expertise beyond a single “infrastructure guru.”

Automating changes with CI/CD

Next, consider how to apply infrastructure modifications after they’re merged into main. A CI pipeline can automatically run a “plan” or “preview,” then only proceed with the actual update if no blockers appear. Policy checks can also be added, enforcing rules such as “no public S3 buckets” or “use encrypted volumes only.” By codifying these conditions, you avoid manual verification steps and reduce the chance of risky configurations slipping past manual review.

Trunk-based infrastructure workflow

Protected main Branch: No direct commits
Pull Request Merges: Integrate changes via Pull Requests
Automated PR Checks: Previews and Policy-as-Code run on every PR
Merge: Merge to main triggers automated deployments

Distinguish each environment

Not all environments have the same requirements or budgets, so it’s wise to separate dev, staging, and production configurations. You might share core modules across these environments, but keep each environment’s parameters and secrets in independent config files or stack definitions. This deliberate separation ensures that dev changes are tested in a realistic but contained environment before they can impact production. As a result, teams can experiment more confidently without risking live workloads.

Modularity and reusability

A key principle of effective IaC, and a cornerstone of any good Internal Developer Platform, is reusability. You shouldn’t be copying and pasting large chunks of infrastructure code. Instead, you should build reusable components (or modules, or constructs, depending on the tool’s terminology) that encapsulate best practices and can be easily consumed by developers. This achieves several critical goals:

Consistency: Ensures that similar resources are provisioned in the same way, reducing configuration drift.
Abstraction: Hides complexity from developers. They don’t need to know how a secure, compliant database is created, just that it is.
Maintainability: Changes to the underlying implementation of a component can be rolled out to all consumers of that component.
Speed: Developers can provision infrastructure faster by using pre-built components.
Reduced Errors: Standard components means that there will be less opportunities to make mistakes.

Think of these reusable components as the “building blocks” of your IDP. Your platform team creates and maintains them, and developers use them to assemble their applications.

Regardless of the specific IaC tool you choose, reusable components generally share these characteristics:

Parameterized Inputs: Components accept inputs (parameters, properties, configuration – the term varies) that allow users to customize their behavior. For example, a “database” component might accept a database_size parameter.
Defined Outputs: Components expose outputs that other parts of the infrastructure can consume. For example, a “database” component might output the database’s connection string.
Encapsulation: The internal implementation details of the component are hidden from the consumer. The consumer interacts with a well-defined interface (inputs and outputs).
Versioned: Components should be versioned (e.g., using semantic versioning) to allow for controlled updates and rollbacks.
Discoverable: Components should be easily discoverable by developers, often through a central catalog or registry.

Example: An IAM roles component

IAM roles are one of the most commonly misconfigured AWS resources, making them a perfect candidate for abstraction into a reusable component. Developers frequently make mistakes when setting up roles, leading to overly permissive access, broken permissions, or operational headaches.

Common IAM role mistakes

Overly Broad Permissions – Developers often grant * permissions instead of following the principle of least privilege.
Missing Assume Role Policies – Without a proper trust policy, services may not be able to use the role at all.
Directly Attaching Policies to Users – Instead of assigning roles to groups, users might manually attach policies, leading to hard-to-audit permission structures.
Lack of Naming Conventions & Tagging – Unstructured role names and missing tags make governance difficult.

How a reusable IAM module fixes this

A platform team can provide a self-service IAM role module that enforces best practices while still allowing customization. This module would:

✅ Require a Least Privilege Policy – The module only allows explicitly defined permissions, preventing wildcards.
✅ Auto-Generate Trust Policies – Ensures correct settings, reducing misconfigurations.
✅ Enforce Naming Standards & Tagging – Ensures roles are easily identifiable and trackable.

By using a reusable IAM role module, developers can self-service their permissions while the platform team ensures security, consistency, and maintainability. This removes a major source of security risk and operational overhead.

No console interventions

No matter how carefully IaC is managed, emergencies may occasionally necessitate direct console changes. However, such interventions are strictly forbidden except under an official “break glass” procedure. If a quick fix is applied in the cloud console, it’s critical that the change is immediately documented and returned to code. Regular drift detection can help with this: once a day or once a week, automated scans can compare actual cloud resources to your declared infrastructure. When it detects a mismatch, you decide whether the code or the running resource is the correct version. Either way, human “hotfixes” never linger as hidden surprises for long.

By combining version control, automation pipelines, environment-based configs, and drift checks, you build a solid provisioning foundation. Once you have that in place, the next step is making these patterns easy to use, so teams can start new services without reinventing the wheel.

But first, let’s cover some metrics and roadblocks.

Metrics that matter

The classic DORA metrics focus on software delivery. Adapting and modifying them for infrastructure provisioning can help measure how effectively your platform delivers reliable, self-service cloud resources:

Lead Time for Infrastructure Changes: How fast does a merged PR turn into running infrastructure? Shorter indicates efficient automation and fewer manual steps.
Infrastructure Drift Frequency: Are developers constantly making console tweaks, or does your infrastructure stay in sync with code? A low drift rate means your code and actual resources stay aligned.
Infrastructure Change Failure Rate: What percentage of infrastructure changes result in incidents or rollbacks? A low failure rate indicates a robust, well-tested infrastructure pipeline, reducing wasted effort and potential outages.
Platform Self-Service Rate: What percentage of provisioning changes are implemented by the requesting team (without needing to involve a central operations team)? A high self-service rate shows that your platform empowers developers and reduces bottlenecks.

Common roadblocks

Adopting Infrastructure as Code helps teams automate provisioning, but common pitfalls can slow progress before they see results. Left unaddressed, problems hidden configuration tweaks, scattered code, or reliance on a single “infrastructure hero” make it hard to keep environments consistent and secure.

By spotting these roadblocks early, teams can build a stronger foundation for a repeatable, collaborative IaC workflow.

Knowledge silos: Relying on just one person (“Go ask Sarah”) for all infrastructure tasks creates a risky bottleneck. The broader team can’t replicate or improve the process, slowing new initiatives and leaving you vulnerable if that individual is out of office or leaves the company. Moving to a collaborative, version-controlled IaC model ensures that no single engineer is a gatekeeper for new environments or configuration changes.
Overly complex or unstructured IaC: Bloated or disorganized infrastructure code leads to confusion, duplication, and frequent errors. Engineers might copy-paste configurations from previous projects or cram all resources into a single file. By refactoring IaC into smaller modules, adopting a clear folder structure, and removing dead code, teams gain consistency and accelerate ramp-up times for new services.
Learning curves: Teams new to IaC often encounter a steep learning curve. There is initially a lot to learn, and resistance from people used to quickly working through cloud UI can slow down full adoption. But time invested in learning tools and setting cultural norms pay off in agility and a platform strategy with guidance and golden paths can help reduce the burden.
Cultural inertia: The adoption of a platform engineering approach is a shift in how organizations manage infrastructure, and like all, change is not merely technical but also organizational. Resistance to change and organizational inertia can make it difficult to implement new ways of getting work done. Expecting, understanding, and working through this resistance is key to overcoming it.
Lack of comprehensive approach: IaC tools are great for provisioning, but without a full platform approach, covering monitoring, logging, and the full life cycle of cloud resources, provisioning efforts may fall short. We will see in this series how the pillars of a comprehensive approach strengthen themselves.

These roadblocks highlight why a thoughtful, well-structured approach to provisioning is essential. But that’s not enough because the next step is to make these consistent patterns easily accessible so teams can spin up entire services without starting from scratch. And that’s where “golden path” templates come in.

Platform team provisioning responsibilities

Setting up the initial IaC foundation
Creating and maintaining templates
Automating workflows
Monitoring usage and compliance
Internal Platform Evangelism

Service templates

Even with a solid IaC foundation and reusable modules, spinning up a brand-new service can still require developers to understand how to pull in the underlying infrastructure components and how to use them. That’s where templates come in.

These are pre-vetted, ready-to-use blueprints that capture your organization’s best practices for a complete service or application, including infrastructure, application, and security components. They are the key to enabling true self-service.

Think of service templates as the next step up from the reusable components we discussed earlier. While a component might represent a single “building block” (like a secure S3 bucket or a database), a service template represents an entire structure built from those blocks (like a complete web application with a database, load balancer, and containerized application). They combine those building blocks with sensible default configurations and best practices for things like logging, Kubernetes setup, and security baked in.

Beyond bundling your organization’s best practices into a single repository, these service templates can also include a preconfigured CI/CD pipeline.

This means that when a team spins up a new service from the template, they inherit not just the infrastructure modules and best-practice configurations, but a fully operational CI/CD workflow. No extra setup is required. Developers get immediate feedback on their changes, and the merge to main triggers automatic deployments that keep every environment consistent.

Instead of cobbling together snippets from old repos, teams can scaffold out a service by simply selecting the template that fits their need – whether it’s a containerized web app, a serverless function, or a data-processing pipeline – and providing a few key parameters.

These curated paths cut onboarding time, reduce configuration missteps, and ensure that the IaC modules are used in new services. If developers don’t have to remember which tags to apply or how to route logs, or even where to import infrastructure modules from, they can focus on delivering features rather than wrestling with infrastructure details. And as your platform matures, and new versions of your modules are introduced, you can update these templates, automatically sharing improvements with every new project.

Bringing it all together: The impact of effective provisioning

With reusable modules and service templates in place, and a deployment workflow that takes changes from git commits all the way to your cloud account, your platform engineering team creates a foundation that transforms how developers interact with infrastructure. This transformation yields measurable benefits:

Lower Infrastructure Change Failure Rate: Automated validations and peer reviews catch issues early, reducing the number of failed deployments.
Increased Platform Self-Service Efficiency: Developers can provision resources in minutes instead of days, boosting productivity and reducing bottlenecks.
Reduced Lead Time for Infrastructure Changes: Streamlined automation shortens the time from code commit to deployed infrastructure.
Minimized Infrastructure Drift: Fewer manual console interventions mean your actual infrastructure stays aligned with what’s declared in code.

(In the upcoming article, we’ll dig deeper into how governance guards these golden paths, ensuring that new projects comply with security and corporate policies right from day one.)

Managing infrastructure provisioning with Pulumi

As you build out your platform engineering plan, consider how Pulumi might fit in. With Pulumi, you get:

Infrastructure as Code in the language of your choice.
Component resources for creating reusable infrastructure modules that encapsulate best practices.
Pulumi Deployments for a fully managed infrastructure deployment solution.
And a Platform in the box experience.

Reliable infrastructure provisioning is the cornerstone that supports your entire platform engineering initiative. By implementing the practices outlined in this article – version-controlled IaC, automated CI/CD workflows, environment separation, and golden-path templates – you establish a foundation that enables consistency, speed, and trust.

But provisioning alone isn’t enough. In upcoming articles, we’ll explore how policy-as-code and broader governance tie into this foundation, ensuring that all those newly created services meet security and compliance standards from the moment they’re deployed. But first, we’ll look at expanding the provisioning foundation covered here into a true self-service platform, so teams can quickly spin up services without risking the organizational chaos that often accompanies speed.

Provisioning is the foundation – next in this series we’ll build on it.

Platform Pillars: Build Platforms, Not Infrastructure

Adam Gordon Bell — Wed, 26 Feb 2025 00:00:00 +0000

Software drives innovation. Development teams face pressure to ship features faster. But speed collides with infrastructure complexity. Developers struggle with cloud setups, juggle scattered tools, and wait on operations teams for resources. The result is friction and slower innovation.

This is where Platform Engineering comes in. It helps developers move faster by creating tools that actually work. A good internal platform lets teams self-serve infrastructure, find documentation, follow best practices, and focus on what they do best: writing useful software.

Building a platform isn’t about finding one perfect tool. It’s about assembling the right pieces, or pillars. These pillars define what every successful internal developer platform needs.

This series explores these key pillars of Platform Engineering, offering a practical guide to building platforms that remove barriers to developer speed. Each pillar addresses a specific challenge organizations face when scaling developer productivity. The first challenge is overcoming infrastructure chaos.

Problem: Infrastructure chaos and bottlenecks

Developers face many pain points with manual provisioning, which is why Infrastructure as Code has become so essential:

Inconsistent environments: Dev, staging, and production drift apart. “Works on my machine” becomes everyone’s excuse.
Slow processes: Console clicking and waiting on ops creates bottlenecks. Developers wait days for basic infrastructure.
Configuration drift: Manual configs lead to errors. Quick production fixes become undocumented mysteries that cause problems later.
Lack of repeatability: Spinning up new environments or replicating existing ones is a complex, time-consuming, and often unreliable process.

That’s why provisioning is the first pillar to master if you want to build a successful developer platform.

Problem: Dependency bottlenecks and limited developer autonomy

Even with automated provisioning, another challenge emerges - developers still struggle when tools are provided without considering usability.

Hard to find what you need: Developers waste time hunting for the right infrastructure modules. They end up searching docs, bothering colleagues, or rebuilding something that already exists.
Too much complexity: Most developers aren’t infrastructure experts. Learning all the options and parameters creates hesitation and mistakes.
Lack of real autonomy: Developers have the tools but not the confidence to use them. They still need approvals or fear breaking production.
Inconsistent workflows: Teams create their own ways of using infrastructure, making collaboration harder.

By implementing self-service infrastructure provisioning, you transform your platform from a centrally controlled system to a developer-centric environment. You unlock developer autonomy, accelerate development cycles, and free up operations teams to focus on higher-value activities.

Problem: Inefficient and fragmented development processes

Infrastructure is just one piece of the puzzle. Developers still face daily friction moving code to production.

A typical developer’s day is full of frustrations:

Inconsistent local environments: Variations in local setups, dependency conflicts, and operating system differences lead to debugging headaches and wasted time getting started.
Slow CI/CD pipelines: Long builds, flaky tests, and mysterious pipeline configs block progress. Developers stare at progress bars instead of writing code.
Cumbersome testing and staging setups: Provisioning environments for testing and staging can be a manual process. Environments may not accurately mirror production, leading to surprises and integration issues late in the development cycle.
Lack of discoverability and reusability: Nobody knows what code already exists, so everyone rebuilds it. Duplication becomes the norm.
Developer toil and context switching: Developers spend more time on environment setup and troubleshooting than on writing features.

These inefficiencies add up, creating significant friction in the developer workflow. Developers spend more time wrestling with processes and less time focused on building and innovating. A Platform approach can help solve these challenges.

Problem: Security as a bottleneck and source of friction

Speed and efficiency matter, but security can’t be compromised. Traditionally, security acts as a gate – a necessary hurdle before releasing software. Platform Engineering aims to transform security from a bottleneck into an enabler.

For many developers, security feels like a roadblock, leading to common friction points:

Last-minute security: Checks happen at the end of development, causing costly rework when issues are found.
Restrictive policies: Security rules feel arbitrary and unclear, slowing down deployments.
The “No” department: Security teams are seen as blockers who enforce policies without offering solutions.
Speed vs. security mindset: Everyone believes you must choose between going fast and being secure.
Security is “Someone else’s job”: Developers think security is only the security team’s responsibility.

This “security as a gate” approach not only slows things down but leads to workarounds as developers try to bypass security to meet deadlines.

By integrating security into your platform, you transform security from a gatekeeper into a built-in feature of the development process. Security becomes an enabler of trust and speed, letting developers innovate with confidence.

Problem: Data overload without insights

Developers in modern cloud environments often find themselves in a paradox: they are drowning in data, yet starving for actionable information. Monitoring systems pump out metrics, logs, and traces, but finding what matters feels impossible. An ideal internal developer platform shifts the focus from data collection to actionable insights.

Here’s what developers typically experience:

Tool sprawl: Teams use different tools for metrics, logs, and traces. Developers waste time jumping between dashboards and manually connecting the dots.
Alert noise: Developers get bombarded with alerts lacking context or clear next steps. Important signals get buried in the noise.
Dashboard overload: Existing dashboards overwhelm developers with granular metrics that are hard to connect to application behavior.
Reactive debugging: Troubleshooting becomes a time-consuming fire drill. Developers spend hours sifting through logs to pinpoint issues.
Missing proactive insights: Observability is only used for firefighting. Developers lack insights to optimize performance or prevent issues before they affect users.
Disconnected from production: Many developers feel blind to how their code actually runs in production, limiting their ability to improve it.

This data flood without clear insights leaves developers reactive rather than proactive, hampering their ability to build reliable applications.

By improving observability in your platform, you transform it from an operations function into a developer superpower. Developers gain the insights they need to build better applications with greater confidence, using AI to focus on what truly matters.

Problem: Balancing autonomy with control and cost efficiency

Empowering developers with self-service is essential, but unchecked autonomy leads to chaos. As platform adoption grows, organizations need to maintain control without creating bottlenecks. This is where Governance as Code becomes critical, establishing guardrails while preserving speed.

Self-service creates new challenges:

Configuration drift: In decentralized environments, infrastructure setups diverge from standards, creating security risks.
Cloud cost explosion: Without proper controls, self-service can lead to wasted resources and budget overruns. Teams over-provision or forget to clean up unused environments.
Compliance headaches: Meeting regulatory requirements becomes harder when infrastructure is created in an uncontrolled way.
Audit gaps: Without clear governance, tracking who created what and when becomes nearly impossible.
Manual governance slows everything: If governance relies on approvals and tickets, it defeats the purpose of self-service.

These challenges highlight the need for automated governance that scales with your platform, ensuring control without slowing developers down.

By implementing Governance as Code, you ensure that self-service doesn’t conflict with security, compliance, and cost efficiency. Governance becomes an automated part of your platform, enabling responsible scaling.

Conclusion

These six challenges represent the core obstacles standing between your organization and true developer velocity. When properly addressed, each transforms into a capability that empowers developers and accelerates innovation.

The upcoming articles in this series will explore each area in depth, with practical strategies and real-world examples to help you build an effective internal developer platform.

Ready to implement these principles in your organization? Check out “Platform Engineering: Recommended Practices for Infrastructure as Code”. It provides detailed guidance on building scalable internal developer platforms.

How to structure infrastructure code for scalability
Managing development environments with developer stacks and Git branches
Designing modular and reusable Pulumi projects
Implementing cross-stack dependencies with stack references
Setting up RBAC, security, and governance at scale
Automating infrastructure management with stack orchestration

Get the Free Platform Engineering Ebook →

And don’t miss the next article in this series, where we’ll explore provisioning as the foundation of your developer platform journey.