Pulumi Blog: Backstage

Backstage vs Pulumi IDP: Why Infrastructure-First Wins!

Engin Diri — Wed, 13 Aug 2025 00:00:00 +0000

Developers are losing days every month to infrastructure bottlenecks, compliance hurdles, and inconsistent environments. Platform engineering promised to fix that, yet too many platforms fail before they deliver real impact.

In this comparison of Backstage vs Pulumi IDP, we’ll explore why choosing the right architectural approach matters more than the tool itself.

Quick comparison: Backstage vs Pulumi IDP

Backstage is an open-source developer portal framework from Spotify that focuses on service catalogs and documentation. It requires significant setup time (12-18 months) and ongoing maintenance (3-5 FTEs).

Pulumi IDP is an infrastructure-first internal developer platform that embeds governance, cost control, and security from day one. It can be deployed in hours with minimal ongoing maintenance.

The problem isn’t whether you choose Backstage or Pulumi IDP. The real question is: Will your platform treat infrastructure as a first-class concern, or as an afterthought?

According to Gartner, 80% of large enterprises will deploy internal developer platforms within the next two years. But without a strong infrastructure foundation, those platforms risk becoming expensive, hard-to-maintain bottlenecks instead of accelerators.

Why platform engineering efforts fail: Common mistakes

Platform teams consistently make four critical mistakes:

Neglecting Infrastructure Foundations
Over-focusing on the developer portal experience while letting infrastructure governance and scalability lag behind.
Ignoring Cost Efficiency
Prioritizing productivity at the expense of cloud cost management.
Example: One company considered giving every developer their own cluster, a scaling nightmare both technically and financially.
Treating Security & Governance as Add-Ons
Bolting on compliance after launch leads to constant firefighting and security debt.
Building for Yesterday’s Architecture
Designing general-purpose platforms that can’t keep up with Kubernetes-native workloads.

Portal-first vs infrastructure-first platform engineering

There are two dominant approaches:

Portal-First typified by Backstage, which starts with developer experience and service catalogs.
Infrastructure-First exemplified by Pulumi IDP, which builds the platform foundation around scalable, policy-driven infrastructure.

Platform Engineering Feature	Backstage (Portal-First)	Pulumi IDP (Infrastructure-First)
Primary Focus	Developer portal & catalog	Scalable, policy-driven infrastructure
Setup Time	12-18 months	Hours
Maintenance	3-5 FTE ongoing	Minimal ongoing overhead
Infra Provisioning	Requires separate tooling	Built-in
Cost Awareness	Depends on infra layer	Native
Security & Governance	Add-on	Built-in from day one

Backstage: The portal-first platform engineering approach

Backstage is an open-source framework for building developer portals. Originally built by Spotify, it excels at unifying tools, services, and documentation.

Strengths

Mature Ecosystem with over 1,200 integrations and a large community.
Excellent Service Catalog providing centralized metadata, ownership, and discovery.
Docs-as-Code that keeps documentation close to the codebase.
Self-Service Templates that standardize scaffolding for new projects.

Challenges

High Maintenance Overhead requiring 3-5 FTEs just to keep it running in large orgs.
Complex Setup with Yarn version headaches, TypeScript complexity, plugin upkeep.
Infra as an Afterthought that lacks built-in provisioning, often pushing infra concerns down the road.
Hidden Costs resulting in multi-million dollar TCO over three years for a 300-dev team when factoring in staffing and infra complexity.

Pulumi IDP: the infrastructure-first platform engineering approach

Pulumi IDP brings Infrastructure Platform Engineering (IPE) to the forefront by embedding scalable, secure, cost-aware infrastructure into the platform from day one.

Strengths

Infrastructure as a First-Class Concern with built-in multitenancy, isolation, and governance.
Efficiency & Cost Awareness through golden paths and reusable components optimized from the start.
Policy-Driven by Design that enforces security, compliance, and cost policies automatically.
Cloud-Native Ready ideal for ephemeral workloads, Kubernetes, and multi-cloud.

Considerations

Newer Platform with a smaller plugin ecosystem than Backstage’s.
Infra-Heavy Focus that’s best fit for infra-intensive orgs; overpowered if you only need a catalog and docs.

Real-world platform engineering outcomes: Backstage vs Pulumi IDP

Pulumi IDP Adoption: CLEAR leveraged Pulumi IDP’s prebuilt capabilities instead of building infra automation from scratch, speeding up delivery while keeping governance tight.

Backstage vs Pulumi IDP decision guide

When to choose Backstage

Choose Backstage for platform engineering if you:

Have a dedicated platform team with strong React/TypeScript skills
Primarily need service cataloging and documentation
Already have robust infrastructure provisioning and governance tools
Have 18+ months and budget for ongoing maintenance (3-5 FTEs)

When to choose Pulumi IDP

Choose Pulumi IDP for platform engineering if you:

Need infrastructure governance from day one
Want cost management and policy automation built in
Must scale efficiently across Kubernetes and multi-cloud
Prefer to avoid building everything from scratch
Need to deploy quickly (hours vs months)

Build your platform on infrastructure

Pulumi IDP embeds governance, cost control, and security into your platform from day one, giving your teams self-service golden paths with policy enforced automatically.

Get started

The shift to infrastructure platform engineering (IPE)

Platform engineering isn’t failing because the idea is flawed. It’s failing because too many efforts start with the portal and leave infrastructure for later.

Infrastructure Platform Engineering flips that model, ensuring the platform’s foundation is:

Hybrid approaches are also possible. Pulumi offers a Backstage plugin so teams can pair Backstage’s portal features with Pulumi IDP’s infrastructure power.

Bottom line: infrastructure-first wins for platform engineering

If your platform is slow, costly, or hard to govern, it’s not a developer problem. It’s an infrastructure problem. Treating infrastructure as a first-class concern from day one isn’t optional. It’s the difference between a platform that scales and one that stalls.

Frequently asked questions

What is the main difference between Backstage and Pulumi IDP?

Backstage is a portal-first developer platform that focuses on service catalogs and documentation, requiring separate tools for infrastructure provisioning. Pulumi IDP is an infrastructure-first platform that embeds governance, cost control, and infrastructure automation from the start.

How long does it take to implement Backstage vs Pulumi IDP?

Backstage typically requires 12-18 months for full implementation with ongoing maintenance by 3-5 FTEs. Pulumi IDP can be deployed in hours with minimal ongoing maintenance overhead.

Can I use Backstage and Pulumi IDP together?

Yes, Pulumi offers a Backstage plugin that allows teams to combine Backstage’s portal features with Pulumi IDP’s infrastructure capabilities for a hybrid approach.

What are the costs of running Backstage vs Pulumi IDP?

Backstage’s TCO for a 300-developer team can reach multiple millions over three years when factoring in staffing and infrastructure complexity. Pulumi IDP has lower operational costs due to built-in automation and minimal maintenance requirements.

Which platform is better for Kubernetes environments?

Pulumi IDP is purpose-built for cloud-native and Kubernetes environments with built-in support for ephemeral workloads and multi-cloud deployments. Backstage requires additional tooling for Kubernetes infrastructure management.

Learn more about Pulumi IDP and how infrastructure-first platform engineering avoids common failure patterns.

Pulumi in a Cloud Native World

Engin Diri — Fri, 02 Aug 2024 00:00:00 +0000

In today’s complex digital landscape, organizations are increasingly turning to platform engineering to optimize their software delivery processes and maximize efficiency. The growing complexity of modern applications, coupled with the need for rapid, secure, and scalable deployments, has created a pressing demand for robust Internal Developer Platforms (IDPs).

IDPs are a key component of modern platform engineering strategies. An IDP is a self-service layer that sits on top of an organization’s infrastructure, abstracting away complexity and providing developers with the tools and environments they need to build, test, and deploy applications efficiently.

Key features of an IDP:

Developer Control Plane. Curated experiences that empower developers by meeting them at their level of expertise, whether it’s an abstracted developer portal, custom CLI, or shared IaC templates.
Integration & Delivery. Automations to version control, test, trace, and deploy all infrastructure from resources, configurations, environments, and secrets as well as orchestration automations to manage provisioning workflows.
Monitoring & Logging. Components to log, monitor, and observe all infrastructure for greater operational control as well as optimize against unnecessary costs.
Security & Identity. Security and compliance guardrails that regulate every piece of infrastructure from policies to fine-grained access controls to secrets.
Resources. Providers that support modern cloud architectures such as Kubernetes, containers, serverless, generative AI, machine learning, data lakes, hybrid cloud/on-premises, and more.

By implementing an IDP, organizations can significantly reduce cognitive load on developers, accelerate time-to-market, improve security and compliance, and foster a culture of innovation. IDPs act as a force multiplier, allowing development teams to focus on creating business value while the platform handles the underlying complexity of cloud-native infrastructure.

The Cloud Native Computing Foundation (CNCF) landscape offers a large ecosystem of tools and technologies that can be leveraged to build these platforms from scratch. However, the sheer number of options can be overwhelming, making it difficult for organizations to determine the best combination of tools for their specific needs.

At Pulumi’s Customer Experience Team, we’ve observed that adopting infrastructure as code often serves as a catalyst for organizations to reassess their entire software delivery process. This reassessment frequently leads to the implementation of more streamlined, automated, and secure methodologies – the core principles of platform engineering mentioned above.

To provide guidance for our customers, we have created a reference architecture called the KEBAP stack, this stack offers a structured approach to integrate these technologies into an effective platform engineering solution using Pulumi.

The KEBAP Stack Reference Architecture

The KEBAP stack consists of the following (most commonly used) CNCF projects:

Kubernetes (with Kyverno as a bonus)
External Secrets Operator
Backstage
Argo CD
Pulumi

Have a look on the high-level architectural diagram of the KEBAP stack to get a first idea of what we are going to talk about in the following sections:

Breaking Down the KEBAP Stack

Let’s take a closer look at each component of the KEBAP stack and understand what their role. The KEBAP is layered in such a way that each component builds on the capabilities of the previous one. The five layers of the KEBAP stack are designed to address different capabilities of our internal developer platform:

Kubernetes

In modern cloud computing, the hub-and-spoke architecture pattern is widely used to manage multiple Kubernetes clusters. In this pattern, a central control plane (hub) manages a set of clusters (spokes) that run applications and services. The hub provides a unified view of the entire infrastructure, enabling centralized management, monitoring, and security.

The control plane cluster in the KEBAP stack is the hub cluster and contains the control plane logic for the entire IDP.

Pulumi

Pulumi manages the lifecycle of all cloud infrastructure, providing developers with self-service capabilities. It allows you to use familiar programming languages to define your infrastructure as code, offering more flexibility and power than traditional YAML or domain-specific languages.

The Pulumi Kubernetes Operator enables you to manage your infrastructure directly from within Kubernetes, seamlessly integrating with your existing Kubernetes workflows.

Since we use GitOps as our deployment method, we define the infrastructure in a Git repository and then let Argo CD deploy it to the hub cluster. As soon as the infrastructure definition is deployed, the Pulumi Kubernetes Operator will take over and create the defined resources in the selected cloud provider.

Here is an example Pulumi definition of a new Kubernetes cluster using DigitalOcean:

apiVersion: pulumi.com/v1
kind: Program
metadata:
 name: ${{values.name}}
program:
 variables:
 clusterName: "do-cluster"
 clusterRegion: "nyc1"
 nodePoolName: "default"
 nodeCount: 1
 version: 1.30.1-do.0

 resources:
 do_cluster:
 type: digitalocean:KubernetesCluster
 properties:
 name: "${clusterName}"
 region: "${clusterRegion}"
 version: "${version}"
 destroyAllAssociatedResources: true
 nodePool:
 name: "${nodePoolName}"
 size: "s-2vcpu-2gb"
 nodeCount: "${nodeCount}"
 # cut for brevity
---
apiVersion: pulumi.com/v1
kind: Stack
metadata:
 name: ${{values.name}}
spec:
 stack: ${{values.stack}}
 envRefs:
 DIGITALOCEAN_TOKEN:
 type: Env
 env:
 name: DIGITALOCEAN_TOKEN
 programRef:
 name: ${{values.name}}
 destroyOnFinalize: false

Argo CD

Argo CD handles the deployment layer, ensuring that the current deployment matches what is declaratively defined in the source. It’s emerging as the de facto standard due to its rich front-end and powerful features. Argo CD implements the GitOps model, continuously monitoring your Git repositories and automatically updating the deployed applications to match the desired state.

In the KEBAP stack, we use Argo CD to deploy the Pulumi infrastructure defined in the Git repository to our hub cluster.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
 name: ${{values.name}}-${{values.stage}}-${{ (values.owner | parseEntityRef).name }}
 namespace: argocd
spec:
 destination:
 namespace: pulumi-operator
 server: https://kubernetes.default.svc
 project: ${{ (values.owner | parseEntityRef).name }}
 source:
 path: "gitops/teams/clusters/${{values.name}}-${{values.stage}}-cluster"
 directory:
 recurse: false
 exclude: '{catalog-info.yaml}'
 targetRevision: main
 repoURL: https://github.com/my-backstage-demo/backstage-infrastructure-provisioning-templates-workshop.git
 syncPolicy:
 automated:
 prune: true
 selfHeal: true
 syncOptions:
 - ServerSideApply=true
 - CreateNamespace=true

External Secrets Operator (ESO)

The External Secrets Operator (ESO) addresses the challenge of securely managing secrets in Kubernetes. It integrates with various secret stores (such as Pulumi ESC, AWS Secrets Manager, HashiCorp Vault, Google Secrets Manager, Azure Key Vault). This allows you to configure access to the secret store once for all your required secrets, enhancing security and simplifying management.

We even take it a step further by using ESO to manage the secrets from Pulumi ESC (Environments, Secrets, and Configurations) with the Pulumi Provider for ESO. This way, we use the Pulumi not only for the infrastructure but also for the secrets management reducing the number of tools we need to manage.

To give the Pulumi Kubernetes Operator access to the secrets, we create following ClusterSecretStore object:

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
 name: secret-store
spec:
 provider:
 pulumi:
 organization: <org>
 environment: <env>
 accessToken:
 secretRef:
 name: pulumi-access-token
 key: PULUMI_ACCESS_TOKEN
 namespace: argocd

And then we can define the ExternalSecret with all the secrets that we want to manage and pass to our Operator as environment variables extraEnv:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
 name: pulumi-operator-secrets
spec:
 refreshInterval: 1h
 secretStoreRef:
 kind: ClusterSecretStore
 name: secret-store
 data:
 - secretKey: pulumi-access-token
 remoteRef:
 key: "backstage.pulumi-pat"
 - secretKey: do-token
 remoteRef:
 key: "backstage.do"

Backstage

Backstage is an open platform for building developer portals. It provides a centralized place for managing software catalogs, documentation, and tooling. This layer helps in organizing microservices and infrastructure, streamlining the process for developers to create, manage, and explore services.

In our KEBAB stack, we installed the Pulumi plugin for Backstage, which allows us to manage our infrastructure as code directly from the Backstage UI by cookie-cutting the Pulumi project and making the result available in our source control repository. From there, Argo CD takes over and deploys the definition to the hub cluster as described above.

apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
 name: kubernetes-template
 title: Kubernetes Cluster
 description: |
 A template for creating a new Kubernetes Cluster.
 tags:
 - pulumi
 - kubernetes
spec:
 steps:
 - id: pulumi-new-component
 name: Cookie cut the component Pulumi project
 action: pulumi:new
 input:
 name: "${{ parameters.component_id }}-infrastructure"
 description: ${{ parameters.description | dump }}
 organization: ediri
 stack: ${{ parameters.stack }}
 template: "https://github.com/my-silly-organisation/microservice-civo/tree/main/infrastructure-${{ parameters.cloud }}-${{ parameters.language }}"
 config:
 "node:node_count": "${{ parameters.nodeCount }}"
 folder: .

Bonus: Kyverno

Kyverno can be used to enforce policies and automate security and operational best practices in Kubernetes clusters. Kyverno is a policy engine designed for Kubernetes that allows you to define policies as code and enforce them at runtime. This ensures that your Kubernetes clusters are secure and compliant with your organization’s policies.

In the KEBEP stack, we use Kyverno to create a ClusterPolicy object to automatically add vCluster clusters to the Argo CD as spoke clusters. This will generate a Secret object in the argocd namespace for each vCluster cluster the dev teams create.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
 name: sync-secret-vcluster
 annotations:
 argocd.argoproj.io/sync-wave: "5"
spec:
 generateExistingOnPolicyUpdate: true
 rules:
 - name: sync-secret-vcluster-nonprod
 match:
 any:
 - resources:
 names:
 - "*-dev-*"
 - "*-staging-*"
 - "*-prod-*"
 kinds:
 - Secret
 exclude:
 any:
 - resources:
 namespaces:
 - kube-system
 - default
 - kube-public
 - kyverno
 context:
 - name: namespace
 variable:
 value: "{{ request.object.metadata.namespace }}"
 - name: name
 variable:
 value: "{{ request.object.metadata.name }}"
 - name: ca
 variable:
 value: "{{ request.object.data.\"certificate-authority\" }}"
 - name: cert
 variable:
 value: "{{ request.object.data.\"client-certificate\" }}"
 - name: key
 variable:
 value: "{{ request.object.data.\"client-key\" }}"
 - name: vclusterName
 variable:
 value: "{{ replace_all(name, 'vc-', '') }}"
 jmesPath: 'to_string(@)'
 - name: stage
 variable:
 value: "{{ split(vclusterName, '-')[2] }}"
 jmesPath: 'to_string(@)'
 - name: team
 variable:
 value: "{{ split(vclusterName, '-')[3] }}"
 jmesPath: 'to_string(@)'
 generate:
 kind: Secret
 apiVersion: v1
 name: "{{ vclusterName }}"
 namespace: argocd
 synchronize: true
 data:
 kind: Secret
 metadata:
 labels:
 argocd.argoproj.io/secret-type: cluster
 clusterType: "vcluster"
 env: "{{ stage }}"
 team: "{{ team }}"
 stringData:
 name: "{{ vclusterName }}"
 server: "https://{{ vclusterName }}.{{ namespace }}:443"
 config: |
 {
 "tlsClientConfig": {
 "insecure": false,
 "caData": "{{ ca }}",
 "certData": "{{ cert }}",
 "keyData": "{{ key }}"
 }
 }

Putting all the Pieces Together

As we have seen, each component of the KEBAP stack plays a crucial role in the overall architecture. Let’s how the different workflows look like when we put all the pieces together. We go through the following scenarios:

Ignite the Control Plane Cluster With Pulumi
Ordering a New Development Kubernetes Cluster
Ordering a New Production Kubernetes Cluster
Ordering Other Infrastructure

Ignite the Control Plane Cluster With Pulumi

We use Pulumi code to create the control plane cluster in our cloud provider of choice. After the creation of the control plane, we install the Argo CD in a very bare-bones way:

const argocd = new ArgoCD("argocd", {
 initialObjects: initialObjects
}, {
 providers: {
 kubernetes: k8sProvider,
 },
});

The idea is to let then Argo CD fetch the rest of the configuration from the Git repository and apply it to the cluster. And here comes the power of GitOps: It will also manage the Argo CD itself. This way, we separated the workload running on the cluster from the creation of the cluster itself.

Ordering a New Development Kubernetes Cluster

Starting point is Backstage, where we can order a new development Kubernetes cluster. As we want to keep the costs low for development cluster, we opted here for the vCluster project. The vCluster project allows us to create a Kubernetes cluster on demand, which runs as containers on the control plane cluster. The moment the cluster is created, we add it to the Argo CD as spoke cluster. This way, we can roll out applications to the development cluster in the same way as we would do for production clusters.

See the definition of the ClusterPolicy object in the Kyverno section.

Ordering a New Production Kubernetes Cluster

The Backstage catalog is again the starting point to order a new production Kubernetes cluster. This time, we use the Pulumi code to create the cluster in the cloud provider of choice. After the creation of the cluster, we add it to the Argo CD as a production cluster. This way, we can roll out applications to the production cluster in the same way as we would do for development clusters.

Ordering Other Infrastructure

Any other infrastructure components such as databases, message queues, and storage will be also ordered via Backstage. And similar to the ordering of a new production Kubernetes cluster, we use Pulumi to create the cloud infrastructure.

Conclusion

A lot of organizations are building their internal developer platforms using CNCF projects to fulfill the needs of their internal customers. The KEBAP stack provides a structured approach to integrate these technologies into an effective platform engineering solution. We also saw that Pulumi plays a crucial role in this stack by taking care of the management of the cloud infrastructure.

This KEBAP stack offers several benefits, including:

A secure, automated, and auditable environment
Reproducible and programmable infrastructure-as-code
Unified management of cloud resources, Kubernetes, and application deployments
Faster onboarding for new team members
Easier compliance with regulatory requirements due to increased auditability

But there is also a downside: The KEBAP stack requires a significant investment in time and resources to set up and maintain as you are in full charge of the integration of the components as well as operating them. All the projects under the umbrella of CNCF are not automatically implying that they are compatible with each other out of the box. Any platform engineering team will need to invest continuously time and resources to keep the stack up-to-date and running. This is a not negligible investment to consider for any organization.

For this reason, we have another reference architecture that uses components from all of the Pulumi Platform for an end to end fully-managed solution. This solution takes care of the integration of the different components of an internal developer portal and provides a carefree experience of managing your infrastructure at scale.

Continue to upskill and learn about platform engineering:

Read Pulumi Patterns and Practices (P3): A Pulumi-based reference architecture for large-scale organizations
Learn how to build Developer Portals with Pulumi.
Register for one of our Platform Engineering or DevOps workshops.

Build The Best Developer Portal with Pulumi Templates

Meagan Cojocar — Thu, 12 Oct 2023 00:00:00 +0000

At Pulumi, we work with organizations that range from a few platform team members to entire departments for managing infrastructure. Many organizations, like Mercedes-Benz, have built internal developer platforms on top of Pulumi to enable developers to self-serve infrastructure templates, and partners like AWS Proton and Port have built integrations with Pulumi to enable self-service scenarios. We are thrilled to have announced yesterday the launch of our new suite of tools to build internal developer portals with Pulumi, a result of working directly with our customers to understand their problems and how Pulumi can solve it.

Pulumi launched Pulumi Templates last year, out of the box templates based for common architectures, such as serverless templates, container service templates and Kubernetes cluster templates. The introduction of these templates allows Pulumi customers to go from 0 to 1 much quicker, but as they scale their IaC complexity a need for custom templates arises. To address this, we are expending into managing custom templates as a self-service provisioning experience.

Introduction to Developer Portals with Pulumi

Many of our customers are building their own developer portals on top of Pulumi Automation API and Pulumi Deployments, but we also wanted to offer options for teams that cannot (or do not want to) build a custom solution. The suite of tools for building internal developer portals include:

New Project Wizard: Showcased in the GIF above, Pulumi Cloud now enables developers to provision infrastructure entirely via point-and-click in Pulumi Cloud with the new New Project Wizard experience. This offers a great out-of-the box developer portal experience for any team using Pulumi Cloud.
Organization Templates: Pulumi Cloud now supports registering your own custom templates for your organization which will be made available in the New Project Wizard. This enables platform teams to standardize infrastructure creation and distribute those best practices where developers are already spending their time.
Pulumi Backstage Plugin: For teams using Backstage, they can now integrate Pulumi directly within their Backstage instance, via support for a new Pulumi plugin as well as new Pulumi scaffolder actions.
pulumi new support for private templates: By default, pulumi new provides a number of templates provided by Pulumi, but now your organization can also use private templates.

Key Benefits

Ease of Deployment: With this new functionality, you can quickly find and deploy the services you need, reducing the time and complexity of managing your cloud infrastructure. As a developer, if you are looking for a platform team sanctioned way to deploy a Kubernetes cluster, instead of chasing it down you can head to Pulumi Cloud, find the golden template and deploy it.

Enhanced Distribution: As a platform team member, share templates and configurations with your developers to ensure consistency and collaboration across your organization. It’s as simple as adding the template to a repo and selecting that repo in Pulumi Cloud. Now all of your developers have access to it when they go to create new infrastructure.

Combination with Other Features: The new template experience seamlessly integrates with your existing Pulumi workflows, providing a unified experience across your infrastructure management. Increase your developer productivity by using this suite of features in combination with Pulumi ESC. Attach an Environment to your template to have it come with configuration and secret values. Once the templates are deployed they will show in the Pulumi Cloud console use Policies over it, set up Review Stacks on the new stack to get ephemeral environments tied to PRs, find the resources that were deployed with Resource Search, and all the other Pulumi Cloud features.

Extensibility: You can leverage the out of the box New Project Wizard Experience or build developer portals with Pulumi yourself using any combination of: the Pulumi Backstage Plugin, pulumi new public and private templates, Automation API, Pulumi Deployments, or the Pulumi Cloud REST API.

Getting Started

New Project Wizard

The New Project Wizard allows anyone in your organization to pick a template they want to install, create the repo or directory for it, and walk through configuring the deployment of that template. We’ve added support for configuring Pulumi Deployments, meaning any template can be deployed without needing the Pulumi CLI locally, or any other CI/CD configuration. Follow these steps to be up and running with your infrastructure in a few clicks:

You can either 1/ use the UI or 2/ use Deploy with Pulumi buttons. If using the UI, navigate to the “New Project” tab. Select “Use a template” if you’d like to create a fully featured project, or select “Use a starter” if you want to create a bare-bones project with only the minimal necessary boilerplate file structure and Pulumi code. If you are using Deploy with Pulumi buttons, you can embed a button on your templates inside GitHub to enable point-and-click deployment of those templates.
In order to use templates you will need to authorize Pulumi with GitHub so that it can clone private repositories as template sources and commit new code for your projects. Click the “Authorize GitHub” button and accept the required permissions if you would like to use templates.
On the next screen select your deployment method. You may need to install the Pulumi GitHub app if you haven’t already to use Pulumi Deployments.
You’ll now be prompted to enter some information about the project you’re about to create: the project and stack name, the configuration details specific to that template, the repo to use when committing your new project code, and optionally the Environment you want to use. Environments enable you to use a bundle of pre-configured secrets and configuration values without needing to re-specify all of them during project creation.
After you’ve configured your project settings, you will be able to configure the behavior of Deployments, including when to trigger new Deployments and environment variables to include with your updates.

And just like that you have created a new project from a template that is deployed, without the use of the Pulumi CLI.

Organization Templates

Pulumi now allows customers on the Pulumi Cloud Enterprise and Business Critical editions to define Organization Templates to help get projects off the ground faster. A Pulumi organization admin can specify an Organization Template Source (or Sources) which will then be populated in the New Project Wizard experience described above.

Navigate to the “Integrations” tab to define your Organization Templates Source. Enter sources as github.com/<owner>/<repo>/<optional subdirectory>. A source can be a directory containing either a Pulumi.yaml, or other subdirectories with their own Pulumi.yaml files. A single repository can provide multiple templates from various subdirectories. For example, these are both valid sources: github.com/pulumi/templates (all public Pulumi templates) and github.com/pulumi/templates/aws-typescript (a specific public template).

The Pulumi.yaml file can optionally contain a template section, which typically includes a config section for specifying required config values for the project. Each config value can have a description and a default value which can be overridden. Config values can be marked as secret, which ensures values in templated projects will be stored with secure encryption.

name: my-aws-project
runtime: nodejs
description: My AWS project description
template:
 config:
 aws:region:
 description: The AWS region to deploy into
 default: us-west-2
 myAccessToken:
 description: My access token
 secret: true

The above snippet includes an aws:region configuration value with a default value of us-west-2, as well as a myAccessToken secret without a default value.

Pulumi & Backstage

We’ve seen many developer portal technologies rapidly growing in popularity over the last few years. In particular, we’ve seen Pulumi users adopting Backstage and as a result we built the Pulumi Backstage Plugin to address the needs of organizations using Backstage and Pulumi together. There are now two ways to use Pulumi from within Backstage, you can leverage both or just one or the other:

Pulumi Scaffolder Backend Module
Pulumi Backstage Plugin

Pulumi Scaffolder Backend Module: The Pulumi Scaffolder Backend Module extends the scaffolding with two new actions: Pulumi New and Pulumi Up. The Pulumi New action is a custom action that allows you to create a new Pulumi project from a template using the pulumi new Pulumi CLI command. The Pulumi Up Action is a custom action that allows you to run the pulumi up command, which creates or updates resources in a stack.

apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
 name: kubernetes-template
 title: Kubernetes Cluster
 description: |
 A template for creating a new Kubernetes Cluster.
 tags:
 - pulumi
 - kubernetes
spec:
 steps:
 - id: pulumi-new-component
 name: Cookie cut the component Pulumi project
 action: pulumi:new
 input:
 name: "${{ parameters.component_id }}-infrastructure"
 description: ${{ parameters.description | dump }}
 organization: ediri
 stack: ${{ parameters.stack }}
 template: "https://github.com/my-silly-organisation/microservice-civo/tree/main/infrastructure-${{ parameters.cloud }}-${{ parameters.language }}"
 config:
 "node:node_count": "${{ parameters.nodeCount }}"
 folder: .

With a few clicks, developers can create new software projects provisioned with best practices, with code managed in the right repositories, and with the right configurations, all from a self-service UI.

Pulumi Backstage Plugin: Backstage Plugins are extensions or add-ons to the core Backstage application that allow you to integrate additional tools, services, and functionalities into the platform, enhancing its capabilities and customizing it to meet the specific needs of your development environment.

The Pulumi Backstage Plugin extends Backstage by displaying essential details like the Pulumi stack, organization, project name, and description, as well as providing an activity view with aggregated information from the Pulumi Cloud. See a screenshot of the new Pulumi tab within Backstage below:

Check out the Pulumi Backstage Plugin in the Backstage Plugin directory or the Roadie Pulumi Backstage Plugin guide to get started today.

Wrapping it up

The launch of this new functionality for internal developer portals marks a significant step forward in our commitment to providing powerful, turn-key tools for platform teams. We believe these features will enhance your experience with Pulumi, whether you’re a seasoned cloud expert leading a team of platform engineers or just getting started on your IaC journey.

As always, we welcome your feedback in the Pulumi Cloud Requests repository and are here to support you every step of the way on your platform engineering journey.

Pulumi for Platform Teams: New Features for Developer Portals, Policy and Deployments

Luke Hoban — Wed, 11 Oct 2023 04:00:00 -0700

Over the last two years, we’ve seen a huge surge in adoption of Pulumi by Platform Teams – centralized teams within a business responsible for building out core cloud infrastructure and providing tools to the rest of the organization to maximize the productivity, cost efficacy, compliance and velocity of application and service delivery throughout the organization. These teams use Pulumi to manage their own cloud infrastructure complexity, to offer best practices components to their organizations, to enforce organizational policy, and to drive infrastructure delivery automation.

Today, we’re excited to announce a broad range of new features that extend Pulumi Cloud’s support for Platform Teams, across the entire lifecycle of cloud delivery.

Developer Portals: Every platform team we work with is looking to expose reusable cloud infrastructure components and templates via a developer portal of some kind. Many are building their own custom developer portals on top of Pulumi Automation API and Pulumi Deployments, but we also wanted to offer options for teams that cannot (or do not want to) build a custom solution. Today, we are releasing three new options for Platform Teams:

🆕 Organization Templates and New Project Wizard: Pulumi Cloud now supports registering your own custom templates for your organization, and enabling developers in your organization to provision them entirely via point-and-click in Pulumi Cloud with the new New Project Wizard. This offers a great out-of-the box developer portal experience for any team using Pulumi Cloud.
🆕 Pulumi Backstage Plugin: For teams using Backstage, they can now integrate Pulumi directly within their Backstage instance, via support for a new Pulumi plugin as well as new Pulumi scaffolder actions.
🆕 pulumi new support for private templates: By default, pulumi new provides a number of templates provided by Pulumi, but it can also use your own private templates.

Policy and Compliance: Platform teams need robust and flexible solutions for establishing policy and compliance guardrails for their cloud infrastructure. Pulumi’s existing CrossGuard policy-as-code framework provides the richest option for defining and enforcing policy across all infrastructure in an organization, using the full capabilities of your favorite languages. Today, we are extending that support with two important new features for platform teams:

🆕 Compliance Ready Policies: New pre-built policy packs for for enforcing common security and compliance policies (PCI DSS, ISO 27001 and CIS) across a broad range of cloud providers (AWS, Azure, Google Cloud and Kubernetes), and dozens of services within each platform.
🆕 Remediation Policies: Pulumi Policy as Code now supports authoring policies which don’t just warn or error, but actually change the infrastructure directly to remediate compliance gaps. Remediation policies can be configured and applied across any subset of stacks within an organization.

Deployments and Delivery: Managing the delivery of cloud infrastructure changes across a wide variety of environments is an essential component of a cloud platform, and we released Pulumi Deployments in preview last year to provide a rich solution for infrastructure delivery directly within Pulumi Cloud. Today, we are making that support Generally Available (GA) along with a few key enhancements:

🆕 Deployments GA: Pulumi Deployments is now generally available, with pricing and support available across all Pulumi Cloud editions.
🆕 Deployments Support for GitHub Enterprise: Pulumi Deployments now supports integration with GitHub Enterprise as well as GitHub.com.
🆕 Org-Wide Deployments Overview Page: A new page where you can see all deployments across the organization, giving you a single pane of glass to manage company wide deployments.
🔜 Deployments Customer Managed Agents: Customers will soon be able to run deployments inside agents they run within their own infrastructure, for cases where this is required for unique compliance or security reasons.

Together with Pulumi ESC (Environments, Secrets and Configuration) which we announced yesterday, and the core Pulumi IaC infrastructure as code platform, these new features provide a rich collection of building blocks and out of the box experiences for empowering platform teams via Pulumi Cloud. At every layer of the Platform Engineering stack, Pulumi and Pulumi Cloud offer key technologies to accelerate your Platform Engineering projects.

Organization Templates and New Project Wizard

We’ve introduced a new wizard experience to create Pulumi projects into Pulumi Cloud, making it easier than ever to get started deploying new infrastructure with Pulumi.

The new project wizard allows anyone in your organization to pick a template they want to install, and walk through configuring the deployment of that template. We’ve added support for configuring Pulumi Deployments automatically, so that the template can be deployed without needing the Pulumi CLI locally, or any other CI/CD configuration. Just a few clicks and you have deployed your infrastructure.

We’ve also added integration with the newly released Pulumi ESC so that you can set up environments (and the configuration and credentials needed to deploy to them) that users within your organization can deploy into, determine who can use which environments using RBAC, and then allow those users to deploy templates directly into the environments they can access – no need to copy secrets, credentials or configuration around!

We’ve also added the ability for organizations to provide their own custom set of Organization Templates — templates specific to their organization, which can live in either public or private Git repositories. The Organization Template Source provided will populate the templates that are displayed to members in the New Project wizard experience.

Together, Organization Templates and the new New Project Wizard offer a foundational developer portal experience directly within the Pulumi Cloud console. Users can come to Pulumi Cloud, select one of the pre-defined templates set up by their Platform Team, and then in a few clicks have it deployed into a predefined cloud environment.

The New Project wizard is available now for all Pulumi users, and Organization Templates are available to Pulumi Enterprise and Business Critical users. Check out the documentation to get started.

Backstage Plugin

We’ve seen many developer portal technologies rapidly growing in popularity over the last few years. In particular, we’ve seen many Pulumi users adopting Backstage. Last week we launched a new Pulumi Backstage Plugin to address the needs of organizations using Backstage and Pulumi together.

The new Pulumi tab gives you direct access to all Pulumi stack activity associated with your backstage projects that include Pulumi stacks.

The Pulumi Backstage Plugin also supports two new scaffolding actions, pulumi:new and pulumi:up, which can be used to template out new Pulumi projects and to trigger updates to Pulumi stacks, fully integrated into the Backstage scaffolding system.

apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
 name: kubernetes-template
 title: Kubernetes Cluster
 description: |
 A template for creating a new Kubernetes Cluster.
 tags:
 - pulumi
 - kubernetes
spec:
 steps:
 - id: pulumi-new-component
 name: Cookie cut the component Pulumi project
 action: pulumi:new
 input:
 name: "${{ parameters.component_id }}-infrastructure"
 description: ${{ parameters.description | dump }}
 organization: ediri
 stack: ${{ parameters.stack }}
 template: "https://github.com/my-silly-organisation/microservice-civo/tree/main/infrastructure-${{ parameters.cloud }}-${{ parameters.language }}"
 config:
 "node:node_count": "${{ parameters.nodeCount }}"
 folder: .

Check out the Pulumi Backstage Plugin in the Backstage Plugin directory or the Roadie Pulumi Backstage Plugin guide to get started today.

Compliance Ready Policies

We first released our CrossGuard policy-as-code framework nearly 4 years ago, and have seen many large organizations leverage its rich expressiveness, flexibility and early prevention features to enforce key compliance criteria for their organization. Support in Pulumi Cloud for automatically enforcing these policies across all Pulumi updates within an organization allows platform teams to put compliance guardrails in place for users within their organization.

However, traditionally each organization has needed to build their own policies, resulting in some activation energy needed to start benefiting from CrossGuard. As we’ve worked with many customers to build out their own internal compliance policies using CrossGuard, we’ve identified some common compliance standards and policies that we can offer directly as part of CrossGuard.

Today, we are making a large collection of new policies available for CrossGuard users. These policies support many common compliance standards (PCI DSS, ISO 27001 and CIS) across a broad range of cloud providers (AWS, Azure, Google Cloud and Kubernetes), and dozens of services within each platform.

These policies are offered as a library, from which Platform Teams can enable the subset they want to apply. For example:

import { PolicyPack } from "@pulumi/policy";
import { policyManager } from "@pulumi/compliance-policy-manager";

new PolicyPack("aws-iso27001-compliance-ready-policies-typescript", {
 policies:[
 ...policyManager.selectPolicies({
 vendors: ["aws"],
 services: ["ec2", "eks"],
 severities: ["critical", "high"],
 frameworks: ["iso27001"]
 }, "advisory"),
 ],
});

The new Policy Pack wizard also makes it easier than ever to create a new policy pack from the compliance-ready policies, and enable it within your organization.

Check out the open source repository to get started with Compliance-Ready Policies today.

Remediation Policies

Existing CrossGuard policies have always had the ability to be set to Advisory to emit a warning when they fail, or to Mandatory to prevent the Pulumi deployment from completing when they fail. This enables strict prevention of non-compliant infrastructure deployments before they even get into production.

However, in many cases, it’s clear what change needs to be made to the infrastructure to make it compliant. Today, we’ve launched new support in CrossGuard for Remediation Policies, a new kind of policy which actually fixes up the infrastructure to be compliant instead of just warning or erroring!

This is a magical experience for both policy authors and infrastructure engineers.

const s3NoPublicRead: ResourceValidationPolicy = {
 name: "s3-no-public-read",
 description: "Prohibits setting the publicRead or publicReadWrite permission on AWS S3 buckets.",
 enforcementLevel: "remediate",
 remediateResource: remediateResourceOfType(aws.s3.Bucket, (bucket, args) => {
 if (bucket.acl === "public-read" || bucket.acl === "public-read-write") {
 // Modify the ACL and return the new bucket state to use instead.
 bucket.acl = "private";
 return bucket;
 }
 }),
};

def s3_no_public_read_remediator(args: ResourceValidationArgs):
 if args.resource_type == "aws:s3/bucket:Bucket" and "acl" in args.props:
 acl = args.props["acl"]
 if acl == "public-read" or acl == "public-read-write":
 # Modify the ACL and return the new bucket state to use instead.
 args.props["acl"] = "private
 return args.props

s3_no_public_read = ResourceValidationPolicy(
 name="s3-no-public-read",
 description="Prohibits setting the publicRead or publicReadWrite permission on AWS S3 buckets.",
 enforcement_level=EnforcementLevel.REMEDIATE,
 remediate=s3_no_public_read_remediator,
)

Check out the documentation to get started adding remediation policies to your policy-as-code configuration today.

Deployments GA

In the last year since we launched Pulumi Deployments in preview, we’ve seen incredible growth in usage and scale of Pulumi Deployments across the Pulumi user base. Customers use Pulumi Deployments for a single built-for-purpose CI/CD solution for their Pulumi infrastructure delivery, but also leverage rich API-driven deployments, Review Stacks, Click-to-Deploy within the Pulumi Cloud Console, and much more.

As part of General Availability, we are announcing pricing for deployments at $0.01 per deployment minute. The new pricing will be effective November 1st. Individual Edition users get 500 minutes free per month, and Team Edition users get 3,000 minutes free per month. Full details of the pricing can be found on the Pricing Page.

In addition to pricing, we’re making available several new features in the Deployments platform.

The new Deployments pane in the Pulumi Cloud console provides a unified view across all deployments in your organization. This includes in progress and queued deployments so that you can easily access any currently in flight deployments, as well as a full history of all deployments performed across all stacks you have access to within your organization.

And finally, we’re extending Pulumi Deployments to support additional key enterprise needs. We recently added support for connecting Pulumi Deployments to GitHub Enterprise, and we will be delivering support for self-hosted Pulumi Deployments agents which can be run within your own infrastructure later this month. You can fill out this form to register for the private preview of self-hosted agents today.

Conclusion

Pulumi Cloud offers a rich and broad platform to support any platform team today. Across the entire lifecycle of cloud infrastructure and application delivery, Pulumi Cloud supports platform teams in building cloud platforms for their organizations which empower developers via rich IaC components and developer portals, enable seamless delivery through Deployments and Automation API, provide robust compliance guardrails through Compliance-Ready Policies and Remediation Policies, and securely manage secrets and configuration across a variety of environments with Pulumi ESC.

It’s never been easier to build your internal developer platform with the help of Pulumi Cloud.

Pulumi Backstage Plugin: Developer Portal Integration

Engin Diri — Fri, 06 Oct 2023 15:00:48 -0400

Backstage is an open source framework for building developer portals, created at Spotify and donated to the CNCF. It allows engineers to create their own development portal internally. Backstage makes it easy for teams to create a unified portal for their infrastructure tooling, software templates, services, documentation and plugins for external tools.

What are Backstage plugins?

Backstage plugins typically include functions from another application or platform and place them in context in Backstage. This adds functionality within the development environment and helps centralize critical developer functions. Backstage provides a marketplace where developers can review plugins and implement them into their own applications.

Pulumi Joins the Stage

Today we are announcing a new Pulumi Plugin for Backstage. Using the Pulumi Backstage Plugin, you can create a new Pulumi project from an organization template stored in Pulumi, you can run previews and updates, and you can run and view Deployment.

How to get started with the Pulumi Backstage Integration

The Pulumi Backstage integration has two independent components:

Pulumi Scaffolder Backend Module
Pulumi Backstage Plugin

What is Scaffolding in Backstage?

The Backstage Scaffolder is a core part of the Backstage developer portal. Using standardized templates, developers can create new components (services, libraries, websites, or other software). With a few clicks, developers can create new software projects provisioned with best practices, with code managed in the right repositories, and with the right configurations, all from a self-service UI.

Pulumi Extends Backstage Scaffolding

The Pulumi Scaffolder Backend Module extends the scaffolding with two new actions: pulumi:new and pulumi:up

With the new Pulumi Scaffolder Backend Module, organizations can standardize infrastructure creation through their own custom templates. This enables a consistent approach for every new project while minimizing manual setup and configuration by developers and platform teams.

The next component of the Pulumi Backstage integration is the Pulumi Plugin. Backstage Plugins are extensions or add-ons to the core Backstage application that allow you to integrate additional tools, services, and functionalities into the platform, enhancing its capabilities and customizing it to meet the specific needs of your development environment.

The Pulumi plugin extends Backstage by displaying essential details like the Pulumi stack, organization, project name, and description, as well as providing an activity view with aggregated information from the Pulumi Cloud.

Connecting Pulumi Cloud with Backstage with the Pulumi Backstage plugin

The Pulumi plugin extends Backstage by displaying essential details like the Pulumi stack, organization, project name, and description, as well as providing an activity view for the entity with aggregating information from the Pulumi Cloud.

Utilizing the Pulumi Backstage integration, including its plugin and scaffolder, brings forth many benefits that streamline infrastructure management, enhance developer experience, and foster a well-orchestrated DevOps culture. See the key benefits below.

Key Benefits of the Pulumi Backstage Plugin

By integrating with Backstage, Pulumi is simplifying a common scenario that many of our users encounter: empowering developers to self-service new infrastructure while enforcing common standards and best prctices. A few benefits of this integration are:

Consolidated View: Obtain a singular view of your infrastructure and application status, merging Pulumi’s infrastructure-as-code (IaC) capabilities with the Backstage developer portal.

Simplified Navigation: Manage both applications and infrastructure from a single pane of glass - even for complex multi-cloud deployments.

Transparent Communication: Facilitate seamless communication across development, operations, and infrastructure teams by providing a shared view and management platform.

Knowledge Sharing: Enable teams to share infrastructure code, configurations, and status transparently, fostering collective knowledge and expertise.

IaC Adoption: Leverage Pulumi’s infrastructure-as-code capabilities to define, deploy, and manage infrastructure efficiently.

Automated Scaffolding: Utilize the Backstage scaffolder to automate the creation and management of Pulumi projects and stacks.

Self-Service Operations: Empower developers with the ability to provision, configure, and manage infrastructure independently, reducing dependencies on operations teams.

Accelerated Development: Enhance the speed and agility of development cycles by providing developers with the tools to manage their infrastructure needs autonomously.

Standardized Workflows: Employ consistent and standardized workflows for infrastructure provisioning and management across all development teams.

Best Practices: Promote the usage of best practices in infrastructure management and application deployment through pre-defined templates and configurations.

How to Get the Pulumi Backstage Plugin

There are two ways to get the Pulumi Backstage Plugin: First, if you are hosting Backstage yourself, you can find the plugin in our GitHub repo.

Secondly, if you are using the Roadie managed Backstage service, you can follow these instructions to use the Pulumi plugin with Roadie.

You might also like:

Conclusion

The Pulumi Backstage Plugin brings simplicity and power to developers by combining Pulumi’s infrastructure management with the user-friendly Backstage developer portal. It makes managing apps and infrastructure easy, all from one place, boosting teamwork and making developers’ lives simpler. With straightforward tools and automated setups, developers can work faster and smarter, managing their projects with fewer hurdles and more clarity.

In short, it’s not just a plugin; it’s a step towards smoother, more integrated development workflows for everyone involved.