Pulumi Blog: Aaron Kao

Why Choose Pulumi Cloud Over DIY Backends?

Aaron Kao — Wed, 19 Mar 2025 09:00:00 -0700

Note: This post discusses Pulumi Copilot, which Pulumi Neo has replaced. Learn about Neo →

Pulumi Cloud empowers engineers to automate, secure, and manage modern infrastructure platforms.

Many companies are building internal developer platforms or modern infrastructure platforms to provide developer self-service while maintaining security and compliance. Companies adopt Pulumi IaC so they can apply software engineering practices to their infrastructure scaling problems and because it is fully open source with a strong community and public roadmap.

At Pulumi, we’re committed to open source—always have been; always will be. Pulumi IaC is entirely open source (Apache 2.0 license), meaning you can adopt and extend it however you like. If you’re new to Pulumi, the open source edition is an excellent way to start modernizing your infrastructure. But as your organization grows and the complexity of your environment increases, you may find yourself devoting significant time to rolling your own enterprise IaC backend features.

That’s why we built Pulumi Cloud—to help you avoid building and maintaining these capabilities from scratch while ensuring you can automate, secure, and manage your infrastructure at scale. Pulumi Cloud provides enterprise capabilities that make it easier to build modern infrastructure platforms. Companies receive increasing value from Pulumi Cloud as their organization and their infrastructure platforms grow in size and complexity.

As companies expand or platform engineering mandates become more expansive, challenges arise around collaboration, security, governance, and scaling. Some questions that need answering include:

How do teams collaborate?
How do you securely manage infrastructure state?
How do you enable self-service infrastructure?
How do you adhere to IT security standards?
How do you enforce governance policies?
How do you protect sensitive data?
How do you enforce least privileged access across the organization?
How do you maintain uptime or get support for your architecture?
How do you manage the cost and complexity of multi-cloud environments?

If some of these questions apply to you or your team, learn more about Pulumi Cloud below to understand when it makes sense to transition to a paid Pulumi offering.

What drives the most value in Pulumi Cloud?

The most frequent business values that our existing customers experience with Pulumi Cloud relate to speed, scale, security, and savings.

Speed

There are speed benefits that customers typically experience across three areas: the infrastructure platform, developer productivity, and operations. Pulumi Cloud simplifies building and running complex infrastructure automation workflows through the Automation API. It also makes it easy to componentize best practices that can be easily shared and distributed through a centralized repository.

Pulumi Cloud streamlines the software delivery pipeline with Pulumi Deployments and a wide range of 3rd party CI/CD integrations.

Pulumi Cloud provides Pulumi Insights, offering search, analytics, and AI-driven insights over your infrastructure. With Insights, you can instantly search for critical information - such as finding MySQL databases on end-of-life versions across all your cloud assets, reducing the operational time it takes to find needles in haystacks.

Scale

Part of building an infrastructure platform is so your organization can scale and make it easy to onboard new developers. Pulumi Cloud integrates seamlessly with various identity providers like Azure ActiveDirectory, Okta, G Suite, or any SAML/SSO provider, offering deep support for role-based access control (RBAC) and SCIM for automatic synchronization and revocation of access based on identity provider groups. Audit logs keep track of developer activity within an organization, recording what actions were taken, when, and by whom.

Pulumi Cloud also makes it easy to build developer portals for developer self-service. Organization Templates can provide a centralized repository for cloud components, best practices and configurations. The New Project Wizard also provides a gallery interface to pick a template and easily walk through configuration and deployment of the infrastructure.

Security

Another reason your organization is investing in an infrastructure platform is so that security and compliance can be automated as infrastructure is self-serviced. Your platform must prevent infrastructure deployments that violate security, reliability, cost, or compliance policies. Pulumi Cloud, through Pulumi CrossGuard, provides out-of-the-box support for common security and compliance policies, ensuring PCIDSS, ISO27001, SOC2, and CIS compliance for cloud applications and infrastructure.

For applications being built across the organization, securing sensitive information like database passwords, cloud credentials, and API keys is paramount. Secrets can inadvertently end up in insecurely stored state files, leading to potential breaches. Pulumi Cloud, however, encrypts all data in transit and at rest, utilizing hardware security module (HSM) based encryption. Read the Pulumi Cloud Security Whitepaper for more information.

Pulumi Cloud further enhances security with Pulumi ESC, a centralized secrets management and orchestration service. This service pulls and syncs secrets from various stores like HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, GCP Secret Manager, and 1Password, making it easy to adopt dynamic, short-lived secrets on demand. Secrets can be accessed via CLI, API, Kubernetes operator, the Pulumi Cloud UI, and in-code with TypeScript/JavaScript, Python, and Go SDKs. Pulumi ESC leverages the same identity, RBAC, Teams, SAML/SCIM, OIDC, and scoped access tokens used by Pulumi IaC, ensuring that secrets management complies with enterprise security policies.

Savings

Building a platform and managing cloud infrastructure demands considerable time and energy. As discussed above, Pulumi Insights X-Rays your entire cloud footprint, so you can observe, plan, and drive changes across your infrastructure. With this total visibility, Pulumi Insights helps you reduce waste by showing you usage information and identifying potentially stale infrastructure, which can lead to substantial cost savings.

You can also have a conversation with Pulumi Copilot about your infrastructure in natural language and get answers to operational questions that would take hours to piece together from other sources.

Why not build your own enterprise features on top of Pulumi IaC open source?

Build vs buy is a question of opportunity cost. Should you and your team build and operate your own enterprise backend service that engineers at Pulumi have already built to handle the needs and scale of the world’s biggest companies? Or is their time better spent focused on building a modern infrastructure platform that will increase development velocity across the company and drive business growth?

Maintenance costs

When you build and operate your own enterprise backend, you’ll need to dedicate one full-time engineer for every team of ten - that’s nearly 10% of your engineering expense. Running a backend requires maintenance, updates, and troubleshooting. Outages and incidents can lead to significant financial losses. Pulumi Cloud, as a fully managed service, frees up your team’s time from the operational and maintenance burdens of managing an IaC backend.

Operational reliability and availability

One of the primary functions of the enterprise backend is managing state. State management involves managing concurrency controls, preventing state corruption, ensuring backup and recovery, maintaining high availability of the state service, and providing consistent visibility across all managed resources; this is easy to get wrong. Manual human intervention is frequently required to recover and repair state files, a process that is time-consuming, can create or prolong outages, and carries significant risk.

Security and compliance

Building your own enterprise backend also requires you to design and manage the service to adhere to numerous IT security standards, including SOC 1/SSAE 16/ISAE 3402, SOC 2, SOC 3, FISMA, FedRAMP, DOD SRG Levels 2 and 4, PCI DSS Level 1, EU Model Clauses, ISO 9001 / ISO 27001 / ISO 27017 / ISO 27018, ITAR, IRAP, FIPS 140-2, MLPS Level 3, and MTCS. This involves a manual assessment and validation process for each standard, which can be time-consuming and complex.

Customer support

You will also be giving up 12x5 or 24x7 customer support, which helps ensure your organization receives immediate assistance with any architectural, cloud-related, or Pulumi best-practices issues. Pulumi also has an experienced group of Solutions Architects and Customer Success Architects that have experience scaling infrastructure as code and building modern infrastructure platforms.

Conclusion

Pulumi Cloud is an enterprise-ready solution that comes with less risk and less maintenance costs than the DIY approach. With Pulumi Cloud, you get a fully managed service that automates deployments, ensures compliance, and offers comprehensive security features out-of-the-box. You’ll get a clear understanding of the product roadmap as well as talented engineers supporting your architecture and use cases.

Pulumi Cloud gives you the platform to automate, secure, and manage your modern infrastructure platform so your company can choose to focus on innovation and growth.

Try Pulumi Cloud for free and see how it can accelerate your infrastructure automation. Sign up today and start building with Pulumi Cloud.

Infrastructure as Code: The Hidden Cost of Doing It Yourself

Aaron Kao — Tue, 10 Dec 2024 09:00:00 -0800

Note: This post discusses Pulumi Copilot, which Pulumi Neo has replaced. Learn about Neo →

Infrastructure as Code (IaC) has revolutionized how cloud resources are managed, allowing for more efficient, scalable, and repeatable deployments. We designed Pulumi IaC to let you program cloud infrastructure using familiar programming languages like TypeScript, JavaScript, Python, Go, .NET, Java, and YAML. This approach not only simplifies the process but also integrates seamlessly with existing development tools and ecosystems (e.g., IDEs, standard unit test frameworks, integration test). You can define infrastructure with code, often in just one line, for serverless, Kubernetes, AI/ML, databases, and more. You can also preview changes before deploying unlike many other IaC solutions. Pulumi IaC is fully open source with a public roadmap. We value working with the community to shape the product through feedback and contributions.

Since this post was published, Pulumi has added first-class support for HashiCorp Configuration Language (HCL). You can now write Pulumi programs in HCL directly, alongside general-purpose languages like TypeScript, Python, Go, and C#. To see how it works, see Pulumi HCL.

What is Pulumi Cloud?

Infrastructure as Code tools like Pulumi require systems for coordinating deployments, which include concurrency control, state management, and security. For the purposes of this post, let’s refer to these systems as IaC backends. There are several options for managing these backends: you can either handle them yourself, which we’ll call DIY, or you can leverage Pulumi Cloud, available as a SaaS or self-hosted solution. With Pulumi Cloud, you gain access to a comprehensive infrastructure management platform designed to handle everything running in the cloud. This platform automates your IaC deployments, centralizes secrets management and orchestration to manage secrets sprawl effectively, and employs AI to oversee infrastructure assets and ensure compliance. On the other hand, with a DIY approach, you have the flexibility to build everything Pulumi Cloud offers or opt for a more minimalistic setup tailored to your organization’s specific needs. Both options are well-supported by Pulumi, but there are distinct advantages to choosing Pulumi Cloud over the DIY method, which we’ll explore further.

Costs and Complexities of DIY Backends

Building and maintaining your own Infrastructure as Code backend involves a significant engineering and operational commitment. On the engineering front, you face security and compliance challenges, as well as the need to develop features that ensure enterprise readiness. Operationally, the backend must be highly available and reliable, and it should facilitate easy onboarding for the rest of the organization when the time comes. Beyond these immediate concerns, there are also hidden personnel costs associated with running your own backend. Let’s break down some of those hidden costs.

Materialize reduced costs and accelerated time to market by making developer onboarding 75% faster.

“Pulumi lets us manage multi-cluster Kubernetes infrastructure efficiently, all in one stack. It has hugely benefited productivity and our service’s reliability. Without Pulumi, we know that scaling and maintaining Materialize would be much harder for the team.”

–— Paul Hemberger, Engineering Manager for Materialize’s cloud team

Learn more

Operational Excellence

One of the primary functions of IaC backends is managing state. State management involves managing concurrency controls, preventing state corruption, ensuring backup and recovery, maintaining high availability of the state service, and providing consistent visibility across all managed resources; this is easy to get wrong. For instance, an incorrect Git merge of your state can result in outages, as evidenced by Spotify speaking publicly about accidentally deleting two-thirds of their production Kubernetes clusters due to a faulty state merge.

Pulumi Cloud offers total visibility into changes within your organization, allowing you to track who is deploying what, when, and where, down to the individual cloud infrastructure properties being changed and the source commits triggering them. If issues arise and you need to revert to a prior known state, Pulumi Cloud’s transactional state checkpointing protocol ensures automatic recovery from failures, and failures are common with transient cloud and networking issues. In contrast, DIY backends require manual human intervention to recover and repair state files, a process that is time-consuming, extends outages, and carries significant risk. Pulumi Cloud also automatically backs up and replicates your state checkpoints, facilitating easy recovery for audits or rollbacks, potentially preventing outages or meeting audit standards.

Managing your infrastructure with IaC demands considerable time and energy. Pulumi Cloud provides Pulumi Insights, offering search, analytics, and AI-driven insights over your infrastructure. If you’re using a DIY backend, you’ll need to build your own search clusters and grep systems. With Insights, you can instantly search for critical information - such as finding MySQL databases on end-of-life versions across all your cloud assets. You can also have a conversation with Pulumi Copilot about your infrastructure in plain English and get answers to operational questions that would take hours to piece together from other sources.

With this total visibility, Pulumi Insights helps you reduce waste by showing you usage information and identifying potentially stale infrastructure, which can lead to substantial cost savings.

When you build and operate your own IaC backend, you’ll need to dedicate one full-time engineer for every team of ten - that’s nearly 10% of your engineering expense. Pulumi Cloud, as a fully managed service, frees up your team’s time from the operational and maintenance burdens of managing an IaC backend.

Unity reduced deployment times by 80%, from weeks to hours, significantly improving their time to market.

“Terraform relies on HCL and lacks support for concepts like classes, objects and inheritance. An equivalent deployment would take more lines of code while yielding IaC that is less reusable.”

Learn more

Security and Compliance

IaC backends are mission-critical components that require robust security measures to protect the backend service, cloud assets, and applications running on the infrastructure.

At the backend service layer, you must design and manage the service to adhere to numerous IT security standards, including SOC 1/SSAE 16/ISAE 3402, SOC 2, SOC 3, FISMA, FedRAMP, DOD SRG Levels 2 and 4, PCI DSS Level 1, EU Model Clauses, ISO 9001 / ISO 27001 / ISO 27017 / ISO 27018, ITAR, IRAP, FIPS 140-2, MLPS Level 3, and MTCS. This involves a manual assessment and validation process for each standard, which can be time-consuming and complex.

Pulumi, as an Amazon Web Services Advanced Partner, undergoes a thorough review and certification of its security and compliance practices by the AWS team. Pulumi Cloud operates on AWS infrastructure, which is designed and managed in alignment with security best practices and all the aforementioned IT security standards. This infrastructure does not require or have access to personally identifiable information (PII), making it suitable for managing applications that require HIPAA and ISO 27018 compliance. Pulumi also conducts annual penetration testing of Pulumi Cloud and maintains active SOC 2 Type II compliance.

Access control at the backend service layer is crucial for enforcing least privilege access. Pulumi Cloud integrates seamlessly with various identity providers like Azure ActiveDirectory, Okta, G Suite, or any SAML/SSO provider, offering deep support for role-based access control (RBAC) and SCIM for automatic synchronization and revocation of access based on identity provider groups. This fine-grained control over state access is a significant advantage over DIY backends, where all access control mechanisms would need to be built from scratch. Audit logs play a vital role in tracking user activity within an organization, recording what actions were taken, when, and by whom. Pulumi’s audit logs are immutable, capturing the UNIX timestamp of events, the user who invoked the action, the event itself, and the source IP of the call. This level of auditing ensures that the activities of organization members are attributable, enhancing security and compliance.

At the cloud asset layer, you and your team need to prevent infrastructure deployments that violate policies related to security, reliability, cost, or compliance. Pulumi Cloud, through Pulumi CrossGuard, provides out-of-the-box support for common security and compliance policies, ensuring PCIDSS, ISO27001, SOC2, and CIS compliance for cloud applications and infrastructure. Such organization-wide enforcement and specific policy support are not readily available with DIY backends.

At the application layer, securing sensitive information like database passwords, cloud credentials, and API keys is paramount. Secrets can inadvertently end up in insecurely stored state files, leading to potential breaches, as highlighted by a Sysdig article where an attacker exploited a manually managed state file to gain control over an entire AWS account. Pulumi Cloud, however, encrypts all data in transit and at rest, utilizing hardware security module (HSM) based encryption. It also avoids capturing cloud credentials, instead relying on client-side authentication, as detailed in the Pulumi Cloud Security Whitepaper.

Pulumi Cloud further enhances security with Pulumi ESC, a centralized secrets management and orchestration service. This service allows for pulling and syncing secrets from various stores like HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, GCP Secret Manager, and 1Password, making it easy to adopt dynamic, short-lived secrets on demand. Pulumi ESC leverages the same identity, RBAC, Teams, SAML/SCIM, OIDC, and scoped access tokens used for Pulumi IaC, ensuring that secrets management complies with enterprise security policies. Every access or change to secrets or configuration values is logged for auditing, providing a secure and auditable environment for you and your team to access secrets via CLI, API, Kubernetes operator, the Pulumi Cloud UI, and in-code with TypeScript/JavaScript, Python, and Go SDKs. With a DIY backend, you have to manage integration and access controls with individual secrets stores, with potential secrets sprawl and operational overhead.

Tetrate uses Pulumi ESC to stop secrets sprawl and attain compliance more efficiently.

“With Pulumi ESC, our developers get dynamic AWS and Azure credentials on-demand. Onboarding new developers is quick and secure, with no more manually filling in .env templates.”

–— Liam White, Platform Lead

Learn more

Organizational Scaling

As your organization grows, you need an IaC backend that scales with the organization and makes it easy to onboard new teams. Pulumi IaC brings your development, infrastructure and security teams together. Your infrastructure teams can define and manage common infrastructure across the organization, collaborate with security teams to establish security and compliance guardrails, and enable easy self-service of infrastructure through custom developer platforms or shared infrastructure libraries. Teams that prefer not to program can use YAML, while those who do can leverage Python, TypeScript, Golang, or C# to define organization-wide components.

You might also like:

With Pulumi IaC, your teams can tackle the growing complexity of modern architectures using familiar software engineering principles. Programming languages offer loops, conditional logic, class inheritance, and object-oriented design, allowing your organization to craft more sophisticated infrastructure compared to legacy IaC tools.

When you’re scaling your organization, you’ll need an IaC backend that makes team onboarding efficient. DIY backends often involve ad-hoc onboarding with bespoke identity solutions, requiring custom documentation and training to troubleshoot unique problems. This can slow down onboarding and reduce end-user productivity. Additionally, DIY backends place the internal support burden on your team, requiring them to assist users in navigating the system.

In contrast, Pulumi Cloud offers standardized, well-documented capabilities and integrates seamlessly with identity systems, leading to efficient onboarding and increased productivity for new employees. Pulumi Cloud provides 12x5 or 24x7 support, ensuring your organization receives immediate assistance with any architectural, cloud-related, or Pulumi best-practices issues. Furthermore, Pulumi Cloud includes built-in RBAC, simplifying the onboarding and scaling of new users.

BMW Group used Pulumi to build a scalable and resilient hybrid cloud implementation that could handle more than eleven thousand developers.

“When we started to use Terraform to deploy our first cloud services with AWS, it didn’t scale quickly enough for our needs. With Pulumi we’re really speeding up development. We can use any language instead of some special language like HCL and by automating policies, we have compliant and secure systems from the beginning.”

–— Jan-Peter Alten, DevOps Engineer

Learn more

Conclusion

When you choose between managing your own backend and using Pulumi Cloud, you need to carefully consider all the costs. You’ll face substantial operational costs with DIY backends, requiring dedicated engineering resources for maintenance, updates, and troubleshooting. Outages and incidents can lead to significant financial losses, as demonstrated by real-world examples where improper state management resulted in major service disruptions. The cost of setup and maintenance for a DIY backend is not just financial but also involves the time and effort of your team, which could otherwise be focused on core business activities.

Moreover, the cost of an engineer dedicated to managing the backend can be nearly 10% of your engineering budget, a significant investment for any organization. Figuring it out on your own adds to the complexity and potential for errors, increasing the risk of costly mistakes. Invisible excess waste in terms of unused or underutilized infrastructure can also accumulate, leading to unnecessary expenses.

When it comes to scaling, the cost of onboarding new teams or users onto a DIY system can be high due to the need for custom documentation, training, and support. Security is another critical area where the costs can escalate, with the need to comply with numerous IT security standards, manage access control, and ensure robust secrets management.

We built Pulumi Cloud as a fully managed service to address these concerns by providing a platform that automates deployments, ensures compliance, and offers comprehensive security features out-of-the-box. You’ll reduce your operational burden, minimize your risk of outages, and make scaling and onboarding easier, ultimately leading to cost savings and increased productivity. By choosing Pulumi Cloud, your organization can focus on innovation and growth rather than the intricacies of IaC backend management. You get to optimize your infrastructure without the overhead of running your own DIY solution.

Next Steps

Each unique variable in your environment –— cloud providers, languages, team structure, business objective, etc. –— makes realizing the full benefits of Pulumi Cloud different for each organization.

Our newly launched program called Immersion Days, is a customizable, half-day event designed to expand Pulumi’s impact across your organization.

The program has 3 key components:

Strategic Leadership Engagement: Address key business challenges like security posture, cost optimization, and developer productivity.
Architectural Review: Evaluate and refine platform engineering projects to enhance efficiency and scalability.
Hands-On Workshops: Equip your engineering teams with practical Pulumi skills through interactive technical sessions.

This program will help strengthen executive buy-in for your infrastructure initiatives while showcasing Pulumi’s value to your peer engineering teams.

If this sounds valuable, let’s discuss tailoring the agenda to fit your organization’s priorities.

Meta image credit: ESO/L. Calçada

Pulumi for AWS: Automate, Secure, and Manage Your Cloud

Aaron Kao — Mon, 02 Dec 2024 09:00:00 +0000

Note: This post discusses Pulumi Copilot, which Pulumi Neo has replaced. Learn about Neo →

Pulumi is excited to be at AWS re:Invent this week, where we’re showcasing our broad and deep support for AWS across all our products. From automating infrastructure with Pulumi IaC to securing secrets with Pulumi ESC to managing cloud assets with Pulumi Insights, Pulumi makes AWS a competitive advantage. Whether you’re a developer, DevOps pro, or platform engineer, Pulumi delivers the tools you need to build and manage modern cloud applications with ease.

Stop by the Pulumi re:Invent booth #370 this week to chat with experts on the Pulumi team. If you can’t make it to re:Invent, join our workshop, Accelerating Platform Engineering with Pulumi on AWS, on December 11, 2024, to see how Pulumi can enhance your cloud operations on AWS.

Why Pulumi for AWS?

Pulumi empowers your organization to automate AWS cloud infrastructure through code, tame secrets sprawl through centralized secrets management, and manage cloud assets and compliance with the help of AI. Pulumi encourages infrastructure, platform, development, DevOps, and security teams to collaborate and accelerates time to market with greater control and minimized risk.

Pulumi will speed up your deployments and time to value, providing you with an approach to open source and community that gives stability and choice. It will also increase your productivity and velocity through better tooling. Lastly, Pulumi uses software engineering to tackle the scale of an infinite combination of cloud architectures. Read more about why engineers and developers switch to Pulumi.

Pulumi for AWS Over the Years

Pulumi was founded back in 2017 to tackle the mounting challenges of modern cloud applications, which have grown increasingly intricate, spanning multiple environments and relying on numerous cloud resources and SaaS services. This complexity creates significant challenges for managing infrastructure. Legacy tools are designed for “two virtual machines with a database” and not thousands or millions of resources across many clouds and accounts. Teams waste time wrestling with excessive YAML configurations, resorting to copy-paste solutions, or using limited domain-specific languages (DSLs) that fail to scale effectively.

The fragmentation between development, infrastructure, and security teams only compounds these challenges. Pulumi’s founding vision addresses these pain points by enabling teams to build and manage cloud infrastructure through infrastructure as code using familiar programming languages and tools. Pulumi Infrastructure as Code (IaC) was launched with strong support of the AWS cloud. We wanted to be the best way for you to manage and scale your AWS infrastructure through code. Over the years, we launched many AWS-specific features to simplify how to manage AWS infrastructure at scale.

Well-Architected AWS Best Practices as a Component

One of the first innovations made for AWS was Pulumi Crosswalk for AWS, a collection of libraries that automatically use well-architected best practices to make common infrastructure-as-code tasks in AWS easier and more secure. Released in 2019, Crosswalk for AWS supports “day one” tasks, such as creating your initial container-based workloads, using Amazon Elastic Container Service (ECS) – including Fargate or Kubernetes (EKS) – and creating serverless workloads using Amazon API Gateway and AWS Lambda. Secure and cost-conscious defaults are chosen so that simple programs automatically use best practices for the underlying infrastructure, enabling better productivity with confidence.

Crosswalk for AWS also supports “day two and beyond” tasks, such as scaling your workload, securing and integrating it with your existing infrastructure, and going to production in multiple complex environments. This includes Amazon Virtual Private Cloud (VPC) for network isolation, AWS Auto Scaling for dynamic scaling, and AWS Identity and Access Management (IAM) for securing your infrastructure.

Earlier this month at KubeCon 2024, we launched the Pulumi Elastic Kubernetes Service (EKS) Provider v3.0.0, which is one of the Crosswalk libraries that makes Kubernetes management on AWS even easier. This release simplifies multi-cluster deployments, improves resource handling, and offers enhanced compatibility with the latest AWS EKS features.

AWS Policies as Code

We released Pulumi CrossGuard to enforce your organization’s cloud governance — security, compliance, cost controls, and more. This enables Policy as Code within your organization so that you can define guardrails for your infrastructure, ensuring engineers are following best practices and putting security first. CrossGuard helps your organization prevent mistakes before they occur and respond rapidly to any incidents.

Building Controls Planes

Many AWS customers build internal developer platforms (IDP) for developers to self-service infrastructure. In 2020, Pulumi Automation API was created to help automate infrastructure for these IDP control planes. Automation API exposes the full power of infrastructure as code through a programmatic interface, instead of through CLI commands, and it lets you use the Pulumi engine as an SDK, enabling you to create software that can create, update, configure, and destroy infrastructure dynamically. This enables you to use Pulumi to build IDPs that are custom-tailored to your team, organization, or customers.

AWS CDK for Pulumi

In 2022, Pulumi brought the ability to use AWS Cloud Development Kit (CDK) constructs from within a Pulumi deployment. For users already using AWS CDK, this provides Pulumi as a new option for orchestrating deployments in place of CloudFormation, offering improved deployment speed and integration with the full set of features of the Pulumi Platform. Even better, you can also now combine AWS CDK and Pulumi resources in a single Pulumi infrastructure as code project - passing outputs from Pulumi resources into AWS CDK constructs, and outputs from AWS CDK constructs into Pulumi resources.

AWS Cloud Development Kit (CDK) on Pulumi is now generally available. This powerful integration includes full support for CDK features like assets, custom resources, aspects, and context values/methods. It provides access to the full ecosystem of over 1,200 CDK Construct libraries, and it also supports all 180+ Pulumi providers, allowing for CDK applications to include resources outside of AWS.

Cloud Control API

In 2021, AWS launched Cloud Control API as a standard set of APIs to Create, Read, Update, Delete, and List (CRUDL) resources across hundreds of AWS Services. Pulumi supported that launch with a native provider called the Pulumi Cloud Control Provider, formerly known as AWS Native. The provider is now generally available, enabling day 1 support for new AWS resources, seamless compatibility with Pulumi IaC projects that currently use the Pulumi AWS provider, and extended support for third-party resources.

Beyond Pulumi IaC

Over the last few years, our customers have shared a clear message: IaC alone is insufficient. The complexities of modern cloud environments demand more than just IaC to manage cloud resources, improve security, and ensure compliance. In response, Pulumi’s offerings have expanded into a comprehensive suite of tools that go beyond IaC.

Pulumi Insights: Intelligent Cloud Management

The complexities of modern cloud environments create significant challenges in making sense of the millions of resources and resource updates per month, across hundreds of clouds, regions, and accounts. Identifying cost saving opportunities is difficult amidst soaring cloud expenditures, and maintaining compliance and security is mission critical because errors can cause catastrophic damage.

Pulumi Insights empowers organizations to tackle cloud complexity by delivering visibility, actionable insights, and AI-assisted optimization for their infrastructure. With Pulumi Insights, companies can scan and sync their entire AWS resource inventory – including resources not managed with Pulumi IaC – ensuring a complete picture of their cloud assets. This Account Discovery capability is now in public preview.

Compliance checks can also be run against standards such as PCI DSS, ISO 27001, and CIS benchmarks, helping maintain regulatory compliance and mitigate risks. The AI-powered Pulumi Copilot provides an interactive assistant to gain visibility into a team’s activity, discover cost saving opportunities, get compliant, and debug cloud failures. Pulumi Insights enables companies to discover, understand, manage, and improve their cloud infrastructure on their journey towards intelligent infrastructure management.

Pulumi ESC: Centralized Secrets Management

Cloud complexity creates significant challenges in managing sensitive credentials, API keys, and configurations across development, testing, and production environments. As organizations scale, they often face issues of secrets and configuration sprawl, duplication of secrets, and too many long-lived static secrets. None of the existing secrets management solutions adequately address these challenges, exposing enterprises to errors that can lead to security breaches, unintended exposure of sensitive data, and unauthorized access to critical resources.

Pulumi ESC addresses these challenges by offering seamless two-way integration – both pull and sync – with AWS Secrets Manager and other popular secrets stores, as well as the capability to consume secrets securely across any application, tool, and CI/CD platform. Developers gain easy, secure access to secrets via CLI, API, Kubernetes operators, and SDKs – eliminating the need for .env files and reducing the security risks associated with them. Additionally, Pulumi ESC can issue dynamic, short-lived AWS authentication tokens on demand via OpenID Connect (OIDC), reducing the risks that come with long-lived access tokens, while RBAC, versioning, and a detailed audit log let you trust (and prove) your secrets are secure.

Try Pulumi Cloud Today

Check out the Pulumi roadmap for upcoming features, let us know your feature requests by creating an issue on GitHub, connect with the Pulumi community on Slack, and if you haven’t yet, sign-up for Pulumi Cloud to get access to our full suite of products – Pulumi IaC, Pulumi ESC, and Pulumi Insights – so you can automate, secure, and manage everything you run in the cloud.

Meta image credit: ESO/M. Zamani

Why Switch to Pulumi for Infrastructure as Code?

Aaron Kao — Tue, 23 Jul 2024 19:47:50 -0700

The cloud promised to revolutionize your business.

Faster innovation. Lower costs. Unlimited scalability.

But for many companies, that promise remains frustratingly out of reach. Instead of accelerating product development, infrastructure has become a bottleneck. You and your team (DevOps, platform, or infrastructure engineering teams) are bogged down by:

Clunky tools and manual processes
Provisioning a simple test environment takes days
Rolling out updates across regions takes weeks
The combinations of modern cloud architectures seems infinite

You know there has to be a better way. A way to truly harness the power of the cloud and turn it into your competitive advantage.

But how?

Enter Pulumi – the open source infrastructure as code (IaC) platform that gives 10x better scale, productivity, and time to value for thousands of companies worldwide.

So why should you switch to Pulumi? How is it different from other infrastructure as code tools? This post should answer all those questions.

What is Pulumi?

Pulumi is an infrastructure as code platform that allows you to manage and scale infrastructure, configurations, policies, and secrets with programming languages. Pulumi facilitates clear collaboration across your infrastructure, development, and security teams.

Pulumi’s approach is better.

10x Better. Pulumi takes a unique and 10x better approach to infrastructure as code by empowering you with the familiar languages and tools you love for application development. While modern programming languages have evolved to provide powerful features like AI coding agents, Intellisense, linting tools, testing frameworks, and CICD pipelines, infrastructure management has lagged behind, relying on rigid scripting languages and error-prone manual processes. Pulumi bridges this gap by allowing you to use industry-standard programming languages to manage infrastructure with the same level of sophistication and tooling you enjoy for application development. Pulumi embraces the change and direction the industry is going, so you never fall behind with your IaC tooling.

Powering the Next Wave. Pulumi is at the forefront of the industry. It helps you embrace the latest practices (e.g., Platform Engineering, GitOps) and builds the latest technologies (e.g., AI agents like Pulumi Neo) into the core user experience. It also helps you build and manage new technology paradigms (e.g., AI workloads, LLMs, internal developer platforms).

Powerful Automations. Another unique approach is that Pulumi makes it easy to automate and scale modern cloud architectures. In the last decade, the industry has moved from monolithic to microservices architectures, which operate as distributed systems over shared infrastructure platforms, to achieve greater resilience, team agility, flexible scaling, and modular codebases. The cloud's programmability, infinite elasticity, and on-demand nature made it easy to spin up micro-sized services tailored to business demands, facilitated by technologies like containers, Kubernetes, and serverless. Pulumi empowers you to apply software engineering to manage infrastructure of this modern and immense scale.

AI-Powered. Pulumi builds AI into the core infrastructure management experience through Pulumi Neo, an agentic infrastructure engineer that understands the real state of your cloud environments and works inside your existing workflows. Neo runs multi-step tasks, generates previews for approval, and opens pull requests for the changes it proposes, so you stay in the loop while it does the legwork of finding and acting on any resource in your infrastructure.

Ok, but why does this matter enough for me to switch?

Only you can answer that question, but there are some compelling reasons to adopt the platform trusted by hundreds of thousands of developers.

Faster Time to Value. Your company moved to the cloud to increase innovation and reduce costs. However, getting infrastructure to developers is a frequent bottleneck for you which can slow down prototyping new products or shipping new features. Spinning up new development or testing environments takes days and rolling out production updates across many regions can take weeks. The existing tools don’t allow you to set security and compliance guardrails and enable easy self-service of infrastructure.

Pulumi will speed up your deployments and time to value.

Open Platform Commitment. Some platforms, such as Terraform, have altered their licensing and introduced uncertainty. They are no longer true open source, and they tie their previously open source software to their commercial services. The lack of an open source approach fragments the community and introduces proprietary constraints to cloud infrastructure.

Pulumi provides you with an approach to open source and community that provides stability and choice.

Increased Productivity. You expect your tools and workflows to keep up with the industry: AI coding agents, Intellisense, linting tools, testing frameworks, and CICD pipelines. But most of these innovations are just for building applications and services and not for infrastructure, configurations, policies, and secrets. Existing infrastructure tools are fraught with bad UX, rigid scripting languages, and error prone manual processes.

Pulumi will increase your productivity and velocity through better tooling.

Infinite Scale. Modern cloud architectures are distributed systems that are microservices that are dynamic and ephemeral in nature. The number of infrastructure resource types and the configurable input properties is staggering. You face the daunting task of how to combine these resources to solve their unique problems. Without software engineering, managing these modern distributed systems is fruitlessly manual.

Pulumi uses software engineering to tackle the scale of infinite combinations.

Customers of Pulumi frequently experience 100% productivity increases, 99% time saved, and 10x acceleration when using Pulumi compared to what they were using before.

Frequently Asked Questions - Why Pulumi?

Here are some frequently asked questions about why you should choose Pulumi based on the following use cases and alternatives:

Why Pulumi, by use case?
- Why Pulumi for Internal Developer Platforms?
- Why Pulumi for AI Workloads?
Why is Pulumi better than the alternatives?
How can I switch to Pulumi?

Why Pulumi, by use case?

There are many different use cases for why you might use Pulumi: infrastructure CICD, modern applications, internal developer platforms, AI/ML workloads, and infrastructure modernization. Below are more details on two popular use cases.

Why Pulumi for Internal Developer Platforms?

You and your team may build internal developer platforms (IDPs) to maximize the use of the cloud at scale across the company while being secure and compliant, so your development teams can ship faster. There are a number of components/layers that are considered basic requirements when building an internal developer platform. The layers are as follows:

Developer Control Plane. Curated experiences that empower developers by meeting them at their level of expertise, whether it's an abstracted developer portal, custom CLI, or shared IaC templates.
Integration & Delivery. Automations to version control, test, trace, and deploy all infrastructure from resources, configurations, environments, and secrets as well as orchestration automations to manage provisioning workflows.
Monitoring & Logging. Components to log, monitor, and observe all infrastructure for greater operational control as well as optimize against unnecessary costs.
Security & Identity. Security and compliance guardrails that regulate every piece of infrastructure from policies to fine-grained access controls to secrets.
Resources. Providers that support modern cloud architectures such as Kubernetes, containers, serverless, generative AI, machine learning, data lakes, hybrid cloud/on-premises, and more.

This diagram illustrates the different layers of an internal developer platform:

Most solutions struggle to keep up with the requirements of each layer. Many of the alternative solutions can’t handle the scale of resources you need to manage, find it difficult to tie infrastructure automation directly into your core business, have leaky secrets, and enforce compliance and security incompletely.

Pulumi is a platform engineering solution that enables you to build a bridge to your developers and empower them to leverage the cloud with security, scalability, repeatability, and consistency. It provides the building blocks for each of the five layers.

Developer Control Plane. Pulumi is the simplest and most intuitive way to manage cloud infrastructure because of its ability to use standard programming languages, including YAML. This removes the friction to the basic requirements of managing cloud infrastructure well. Pulumi Automation API makes it simple to code any user interface for a developer portal / CLI. Pulumi also provides private templates that integrate with developer portals like AWS Proton and Backstage.
Integration & Delivery. Pulumi Automation API can embed IaC programs directly in applications, resulting in 10x greater management of resources per engineer. Pulumi can take advantage of all existing testing frameworks supported by the selected programming language.
Monitoring & Logging. Pulumi Insights adds advanced search, analytics, and AI to any cloud infrastructure, giving unique insights into cloud usage and cost optimizations. Pulumi partners with leading observability solutions making it easy to manage monitoring and logging resources through IaC.
Security & Identity. Pulumi CrossGuard provides policy as code with auto-remediation. Pulumi ESC makes it easy to manage secrets and configurations for every environment. Access to each cloud resource and secret can be granularly controlled and audited.
Resources. Pulumi supports modern cloud architectures such as Kubernetes, containers, serverless, generative AI, machine learning, data lakes, hybrid cloud/on-premises, and more. Pulumi makes it simple to create components that abstract away the complexity of managing thousands of resources across hundreds of distinct clouds.

Pulumi is purpose built to make all aspects of platform engineering vastly simpler.

Why Pulumi for AI Workloads?

Your company may be using AI to increase innovation and reduce costs. AI gives them the ability to design richer and more intuitive interfaces for their products and/or services to connect better with their customers. The hardest part of AI is at times not the AI pieces but the cloud infrastructure parts: how to provision and manage the infrastructure that AI workloads run on (e.g., compute and networking) and depend on (e.g., databases and storage).

There are many layers to building and managing AI applications: model training, data pipelines, backend services, frontend applications.

Pulumi provides an abstraction across all the different layers of the AI stack (web framework, LLM, containers, databases, secrets, policies, configurations, etc) as a simple Python library. Through this abstraction you can manage stacks of AI infrastructure as code.

Pulumi makes it trivial to take local AI development to production in the cloud.

Why is Pulumi better than the alternatives?

There are many different tools you can use to manage infrastructure. However, Pulumi provides you with a more productive authoring experience and more powerful automations that will allow you to ship faster and manage the infinite scale of cloud infrastructure. Read on for how Pulumi compares to specific options: clickops, Terraform, CDK, and IDP services.

Why Pulumi vs. clickops?

Manually provisioning and managing infrastructure for production via the cloud console (i.e., clickops) is a bad idea. There isn’t repeatability or consistency which leads to errors, and that can lead to downtime or worse, security breaches. That is why infrastructure as code was invented as a way to have a single source of truth for all infrastructure with changes 100% automated.

If you don’t believe this, just ask any cloud or devops subreddit, Slack group, or Discord server.

Why Pulumi vs. Terraform?

Pulumi and Terraform are both infrastructure as code (IaC) platforms, however they have fundamental differences in how they approach your needs of infrastructure management. First, here are the similarities.

Both Pulumi and Terraform include the ability to create, deploy, and manage infrastructure as code on any cloud. Both Terraform and Pulumi follow a desired state infrastructure as code model, where the IaC code represents the desired state of the infrastructure. The deployment engine compares this desired state with the current state of the stack and determines the necessary actions, such as creating, updating, or deleting resources. Both Terraform and Pulumi support many cloud providers, including AWS, Azure, and Google Cloud, plus other services like CloudFlare, Digital Ocean, and more.

However, beyond the basics of infrastructure as code there are significant differences that affect productivity, scalability, and collaboration.

Increased Productivity. As discussed at the beginning, Pulumi’s promise is to build in all the latest advancements in both the developer and operations experience. When you write Pulumi code, you now have AI coding agents that can pair program with you, IDEs that have autocompletion and Intellisense squiggles, powerful libraries of low level and abstract functions, testing frameworks, code review tools, automated release controls via CICD pipelines, and great software packaging tools. When it comes to managing infrastructure with Pulumi, you also have Pulumi Neo, an agentic infrastructure engineer that works inside your existing workflow: it proposes changes, runs previews, responds to failures, and opens pull requests in tight feedback loops, grounded in the real state of your infrastructure in Pulumi Cloud. Because that infrastructure is defined in Python, TypeScript, Go, C#, or Java rather than a bespoke configuration language, Neo can read, reason about, test, and ship it the same way an AI coding agent already handles the rest of your codebase. Terraform, by contrast, is defined in HCL, a domain-specific configuration language that AI agents can generate and reason about less reliably than a general-purpose programming language — one more reason teams switch.
Greater Scalability. Pulumi embraces software engineering as a way to solve and manage the exponentially increasing complexity of modern architectures. Programming languages with their loops, conditional logic, class inheritance, object orientedness allows engineers to design more complex and sophisticated infrastructure compared to using HCL. Pulumi also allows you to build custom workflows atop infrastructure programs, giving rise to event-driven automations or internal developer portals that provide self-service.
Better Collaboration. Pulumi makes it easy to prototype new products and quickly ship them into production because it is easier for the platform, development, and security teams to collaborate. You (platform engineering and devops teams) can define and manage common infrastructure across the company, work with the security teams to set security and compliance guardrails, and enable easy self-service of infrastructure whether through custom developer platforms or shared infrastructure libraries. To define common company-wide components, if you don’t want to program can use use YAML; if you do, you can use Python, TypeScript, Golang, C#, etc. These components can be consumed by the development team in their own IaC program in any programming language with the development tools they already know. It's easy to start with YAML and move to other languages when more power is desired. With Terraform, you must understand and use HCL. It is difficult to build self-service infrastructure platforms because it lacks the programmability and ability to apply software engineering. It's just harder to bring together platform, development, and security teams and empower them with tools that work. Whereas Pulumi unblocks infrastructure as the bottleneck for software delivery.

Why Pulumi vs. AWS Cloud Development Kit (CDK)?

Pulumi and CDK are similar in that both allow you to use programming languages to write infrastructure as code. However, there are some key differences:

No Vendor Lock-In. CDK supports only AWS, whereas Pulumi supports over 150 cloud and SaaS providers, with more being added all the time. CDK depends on CloudFormation as the deployment engine; it shares many of the same benefits and limitations as CloudFormation (see Pulumi vs. CloudFormation)
Execution > Translation. Pulumi and CDK support similar programming languages but differ fundamentally in their deployment approaches. Pulumi's engine directly understands these languages and communicates with cloud providers. In contrast, CDK transpiles code into AWS Cloud Assembly, an intermediate format consisting of CloudFormation templates and other assets, before deployment via CloudFormation. This difference impacts your development speed and correctness. CDK's approach can lead to slower deployments and delayed error discovery, potentially hours into the process since the errors aren’t caught during compile time. Additionally, you need expertise in both CloudFormation and CDK for effective debugging and successful deployments, whereas Pulumi's direct approach simplifies this process.
Versatility. CDK and Pulumi both support automated testing, but Pulumi offers more versatile options. While both allow unit testing, Pulumi's deep integration between language and runtime enables fast, offline tests (in-memory) with mocked cloud provider calls. In contrast, CDK only permits assertions against synthesized CloudFormation templates and lacks offline testing capabilities. This makes Pulumi's testing approach more comprehensive and flexible than CDK's.

Why Pulumi vs. 3rd party internal developer portal providers?

There are many third party providers of internal developer portals (think Port, Cortex, Backstage). Some are services and others are open source software. These providers are similar in that they offer simple developer portals that are a single part of a greater infrastructure management platform.

As discussed earlier, internal developer platforms have five layers: developer control plane, integration & delivery, monitoring & logging, resources management, and security & identity. Most of these providers offer easy to use GUI consoles, simple CICD integrations, comprehensive suite of monitoring & logging, and some form of role-based access controls. However, they lack the infrastructure as code fundamentals and automation capabilities that Pulumi has, which enable you to build powerful customizations through software engineering. Many of these solutions also lack strong secrets management and policy enforcement capabilities that are critical for production and enterprise deployments.

That said, if you need an off the shelf solution, then one of these services or software might be a good choice for you. Some of these solutions can integrate with Pulumi, and if your needs become more complex, you can always switch to Pulumi later.

Have a particular Pulumi vs X comparison that you need thoughts on? Feel free to join the Pulumi Community Slack if you have quick technical questions, or talk to a Solutions Architect if you need more detailed consultation about your specific architecture.

How can I switch to Pulumi?

Switching to Pulumi doesn’t have to be intimidating. We’ve done this with thousands of customers before, and we can guide you through it.

We provide self-service conversion tools that allow you to Import infrastructure no matter how it was provisioned, click-ops included. You can also use tools to convert your HashiCorp Terraform, AWS CloudFormation, Azure Resource Manager (ARM) templates, or Kubernetes YAML.

Self-service Tool Guide

If you need help, we have a team of cloud experts who can answer your questions, give you a demo, or roll up their sleeves to get your migration done.

Contact Expert Services

We have done lots of migrations from all types of infrastructure as code tools, and we are happy to help you think through how to make switching to Pulumi as easy as possible.

Switch to Pulumi

Meta image credit: ESO/M. Kornmesser

Celebrating 20,000 Stars: A Milestone for the Pulumi Community

Aaron Kao — Thu, 23 May 2024 18:48:50 +0000

Today, we’re excited to announce that the Pulumi open source project has crossed the incredible milestone of 20,000 stars on GitHub. 🎉 This is a huge achievement, and it wouldn’t have been possible without y’all - our incredible global community of developers.

Since its inception, Pulumi’s vision has been rooted in enabling teams to build and manage modern cloud infrastructure using familiar languages and tools. Over the years, your feedback, contributions, and passion have been invaluable in shaping Pulumi into what it is today. Seeing so many developers embrace our modern approach to cloud infrastructure management is really exciting. We are deeply grateful for the trust y’all have placed in us, and we will continue to push the boundaries in how teams manage cloud infrastructure.

Fueled by Community Contributions

Over the past 7 years, the Pulumi project has had 4,400+ contributors that made 75,000+ pull requests. Pulumi is depended upon by 6,600+ GitHub projects and supports over 160+ packages in the registry. It has been downloaded more than 100 million times by 170,000+ developers. In just the last month, Pulumi has been the most active Infrastructure as Code (IaC) open-source project compared to Terraform or OpenTofu. Pulumi has had 117 merged pull requests (67 TF, 83 OpenTofu), 82 closed issues (49 TF, 54 OpenTofu), and 53 new issues (27 TF, 29, OpenTofu).

Notable Features and Milestones

We also want to take this time to reflect on some of the notable features we shipped that got us here. The Pulumi roadmap is public so we can be transparent about the features being worked on and to encourage deeper collaboration with the community. Here are some of the most popular features that were added to Pulumi over the last 7 years.

Mar 2017 - Pulumi founded
Jun 2018 - Pulumi open source project launched
Aug 2018 - Kubernetes native provider
Oct 2018 - Series A fundraise
Oct 2018 - Pulumi Cloud launched
Nov 2018 - Stack Reference to allow access of outputs of one stack from another
Jun 2019 - AWSX as higher-level components for the AWS platform
Sep 2019 - Pulumi 1.0
Sep 2019 - Transformations to modify the properties and resource options for child resource of a component or stack
Nov 2019 - .NET support
Dec 2019 - CrossGuard for policy as code
Apr 2020 - Pulumi 2.0
May 2020 - Golang support
Sep 2020 - Azure Native Provider for same day access to new Azure features
Oct 2020 - Series B fundraise
Oct 2020 - Automation API exposes IaC through a programmatic interface
Oct 2020 - Import command to import resources not managed by Pulumi into a stack
Apr 2021 - First PulumiUP
Apr 2021 - Pulumi 3.0
Apr 2021 - Pulumi Packages to provide multi-language support for components and providers
Oct 2021 - Pulumi Kubernetes Operator
May 2022 - Java support
Nov 2022 - Pulumi Deployments to automate infrastructure deployments
Nov 2022 - YAML support
Dec 2022 - Hierarchical config
Apr 2023 - Pulumi Insights for intelligence, search, and analytics over any infrastructure
Apr 2023 - First Pulumi User Group (PUG)
Jul 2023 - Terraform conversion improvement as an easy way to migrate TF to Pulumi
Sep 2023 - Pulumi AI added to pulumi CLI
Sep 2023 - Go generics support added
Oct 2023 - Series C fundraise
Oct 2023 - Pulumi ESC to manage secrets and configuration complexity
Oct 2023 - Pulumi for Platform Teams through developer portals and more
Apr 2024 - Drift detection, TTL stacks, and scheduled deployments

We also shipped many other features that didn’t make this post, so please see our blog for all those details.

A Heartfelt Thank You

We are immensely grateful to our community for helping us reach this incredible milestone and making Pulumi a thriving ecosystem. Your contributions, feedback, and support have been invaluable in shaping Pulumi’s journey. We also want to thank the Puluminaries - our community leaders and advocates whose contributions have been pivotal to our success.

If you’re passionate about cloud infrastructure, consider joining the Puluminaries. As a Puluminary, you’ll gain direct access to our engineering team, preview roadmaps, and help contribute to code, content, and community support.

We look forward to continuing our innovations to democratize the cloud and empower developers worldwide. We encourage you to join our Slack community to engage with fellow developers and be part of the ongoing conversation about the future of how teams manage cloud infrastructure.

Thank y’all for being part of this journey. Here’s to the next 20,000 stars and beyond!

Local Testing With Pulumi

Aaron Kao — Fri, 28 Apr 2023 00:00:00 +0000

If you’ve been following along with our IaC Recommended Practices series, then you’re already familiar with Zephyr Archaeotech Emporium, the fictional company at the center of the series. Today, you’ll get an inside look at how Zephyr starts using Pulumi for locally testing the application code for their online store and accelerating the inner dev loop for their development team.

In the event you’re not familiar with the term “inner dev loop,” it’s used to describe the iterative code/build/test loop that every developer goes through to write code and verify that it works before they push their code to version control to be reviewed and merged. Speeding up the inner dev loop makes it possible for developers to iterate on their code more quickly, reducing the time spent waiting on tools or environments.

Structured as a conversation between two employees at Zephyr—Alice, a developer on the team for Zephyr’s online store application, and Bob, a member of Zephyr’s platform engineering team—this post shows one way to use Pulumi to help accelerate the inner dev loop. Let’s listen in…

Listening in at Zephyr HQ

In a conversation at Zephyr HQ between Bob and Alice one day, Bob commented, “I’m really enjoying Pulumi. It’s nice for the platform team to be able to use TypeScript to define all the infrastructure we need.”

“Or to deploy our application onto that infrastructure,” Alice responded. “I just wish there was a way to speed up testing changes to our code.”

“Do you mean testing the Pulumi code?”

“No, no, I was talking about our application code—the code for the online store.”

“Oh, what’s wrong with the current way it’s handled? Is there something we can do better?”

Alice paused. “Well, don’t get me wrong, the way we use per-developer stacks makes it super easy for me to deploy an entire copy of our environment. The problem is that…well, it takes a fair amount of time to create all the AWS stuff. Sometimes I just need something faster, just to test some quick changes to the code before deploying the app to my dev stack.”

Bob nodded thoughtfully. “I see. You know, Pulumi has a Docker provider; in fact, they recently released a new version of their Docker provider.”

Alice’s eyes lit up. “So, you mean I can automate my local Docker instance using Pulumi?”

“Yeah, why?”

Alice smiled. “I have an idea.”

Using Pulumi for the inner dev loop

A couple days later, Alice contacted Bob and said, “I want to show you what I’ve come up with.”

To see the code that Alice created for local testing of the Zephyr online store with Pulumi, visit the zephyr-app repository and look at the blog/inner-dev-loop branch.

When Bob met up with Alice, Alice showed him a TypeScript program she’d written with Pulumi’s Docker provider that automates building and deploying the Zephyr online store to a local Docker daemon.

“With this code,” Alice explains, “I can just run pulumi up and it will deploy the entire Zephyr online store locally, all in just a few minutes. Here, let me just run pulumi up real quick.”

“But why use this instead of Docker Compose?”

“Oh, there’s absolutely nothing wrong with Docker Compose; it’s a great tool!” Alice replied. “But this allows me to work with TypeScript, and to build logic into my Pulumi program. Like this feature here—I can specify whether I want to build a container, or use the released version of a container.”

“See, at the top I have the Pulumi program check for a configuration value that I set using pulumi config set. If the value isn’t set, it defaults to build.”

// Get configuration values
const config = new pulumi.Config();
const srcRepoPath = config.get("srcRepoPath") || "../..";
const assetsFlag = config.get("assetsFlag") || "build";
const cartsFlag = config.get("cartsFlag") || "build";
const catalogFlag = config.get("catalogFlag") || "build";
const checkoutFlag = config.get("checkoutFlag") || "build";
const ordersFlag = config.get("ordersFlag") || "build";

“There’s one for each service in the online store?” Bob asked.

“Right, except for the UI service, which is always built. This is so I can independently test changes to only a specific service. Farther down, the Pulumi code checks for this value and then either builds the container—pulling in whatever changes I’ve made—or uses an already released version of the container.”

// Build the assets image or pull remote image, depending on value of assetsFlag
var assetsImageRef: pulumi.Input<string>;
if (assetsFlag == "build") {
 const assetsImage = new docker.Image("assets-image", {
 build: {
 context: `${srcRepoPath}/src/assets`,
 dockerfile: `${srcRepoPath}/src/assets/Dockerfile`,
 platform: imagePlatform,
 },
 imageName: "zephyr-assets:latest",
 skipPush: true,
 }, { retainOnDelete: true });
 assetsImageRef = assetsImage.id;
} else {
 const assetsRegistryImage = docker.getRegistryImage({
 name: "public.ecr.aws/aws-containers/retail-store-sample-assets:0.2.0",
 });
 const assetsRemoteImage = new docker.RemoteImage("assets-image", {
 name: assetsRegistryImage.then(assetsRegistryImage => assetsRegistryImage.name),
 pullTriggers: [assetsRegistryImage.then(assetsRegistryImage => assetsRegistryImage.sha256Digest)],
 }, { retainOnDelete: true });
 assetsImageRef = assetsRemoteImage.repoDigest;
};

You can see that the released container image is hard-coded in this example. You could also pass this in as a configuration value to allow the Pulumi program to more easily account for new releases.

“By setting the flag for the assets service to build and all the other services to something else—the code only builds if the value is set to build—then I can isolate changes made to that service for testing, to be sure I haven’t broken something elsewhere.”

“That seems useful.”

“Indeed! And you can see here that we are building a Docker network for these containers; this is what enables connectivity between the various parts of the online store when it’s running locally via Docker.”

// Create a network
const network = new docker.Network("network", {
 name: "zephyr-net",
});

// Create an assets container
const assetsContainer = new docker.Container("assets-container", {
 capabilities: {
 drops: ["ALL"],
 },
 envs: [
 "PORT=8080",
 ],
 hostname: "assets",
 image: assetsImageRef,
 memory: 64,
 name: "assets",
 networksAdvanced: [
 {
 name: network.name,
 },
 ],
 restart: "always",
});

Bob nodded. “Yep, this all makes sense. Interesting, I hadn’t really considered Pulumi for this particular use case. Usually when folks mention Pulumi the first thought in my head is provisioning cloud infrastructure.”

“Oh, certainly, and it works really well for that—but it also works really well for this case, too.”

“And since we are already using Pulumi for deploying to Kubernetes in test and production, then this brings some consistency across environments. It’s just a pulumi up regardless of whether you’re running the app locally or publishing to a Kubernetes cluster in your per-developer stack, the test stack, or even the prod stack.”

“Exactly. Oh, look here. It’s done. Looks like it took just over two minutes, and now our online store app is available at this URL on localhost.”

“That’s pretty fast.”

Alice nodded. “Yeah, a lot faster than waiting on a Kubernetes cluster to provision. We still need to run this in our dev stacks to be sure that it runs in an environment that closely mirrors production, but for quick feedback on changes I make this is very useful for me as a developer.”

“So if you make a change to the code and want to see the results, what’s that look like?”

“Here, let’s do it real quick.” Alice switches to her code editor, makes a few changes, and then flips back to the terminal. “I’ll run a pulumi preview so you can see what it’s going to do.”

Alice pointed to the terminal output. “Here, Pulumi has recognized that the source code for this container has changed, and so it is replacing the container image, and restarting all the containers. Let me run pulumi up.”

Bob watches as the operation completes in about 17 seconds.

“Wow, that’s really fast!”

“Exactly, Bob. That’s what makes using Pulumi for this use case so helpful. While it doesn’t eliminate the need for our per-developer stacks, it gives developers a way to quickly see changes we’re making to the source code in very little time. When we are satisfied that the changes are working as we expect, then we can deploy to our dev stack to be sure that the changes are ready to work their way to production.”

Speed up your inner dev loop

Use Pulumi and the Docker provider to build and deploy your application locally, then promote the same program to your cloud stacks. Get started in Pulumi Cloud.

Get started

Try this out yourself

Now that you’ve seen how Alice uses Pulumi to streamline testing the Zephyr online store’s application code locally, feel free to try this out yourself! The code that you saw in this blog post is available in the zephyr-app repository on GitHub. Just select the blog/inner-dev-loop branch and go to the develop/pulumi folder.

IaC Best Practices: Understanding Code Organization & Stacks

Aaron Kao — Mon, 20 Feb 2023 00:00:00 +0000

This is the first in a series of blog posts that explores how a fictional company—Zephyr Archaeotech Emporium—uses Pulumi to manage their online retail store. This post explores a couple of common questions that users ask when working with Pulumi; specifically, where should I store my Pulumi code? And how do I support multiple environments with Pulumi? This post will provide some guidance and Infrastructure as Code best practices around these topics, using Zephyr and their online store as the use case.

The ultimate goal of this series is to discuss recommended practices for using Pulumi to manage a fairly complex containerized application. However, it’s important to note that these practices will emerge over the course of the series—not all immediately, and not all in the beginning. This is a deliberate decision to allow you to see how Zephyr’s use of Pulumi evolves as the company grows and their online retail store application changes to accommodate their growth.

Setting Up the Scenario

Zephyr is short for Zephyr Archaeotech Emporium, the fictional company in our scenario. Zephyr is an online retailer that specializes in the sale of “rare arcane artifacts and replicas.” Over the past few years, the company has experienced a significant increase in its online presence, making it easier for customers to purchase unique and mysterious items. Zephyr’s collection includes a variety of rare and unusual objects that are difficult to find elsewhere, making it a popular destination for collectors, enthusiasts, and adventurers.

Moving to a Containerized Architecture

Zephyr’s application is the second generation of their online store. Their first generation was a monolith that they manually deployed. It was sufficient in Zephyr’s early days, but as word about Zephyr spread and the volume of traffic to their online store increased, Zephyr found it increasingly difficult to deploy small, incremental changes to the store’s code. As a result, the velocity of their development efforts slowed. In preparation for the next phase of their growth—driven by robust sales of arcane artifacts—Zephyr’s team evaluated a number of different architectures that would enable them to increase development velocity and give them greater flexibility in scaling different aspects of their online store. In the end, they settled on a containerized architecture deployed to Kubernetes because some of the existing team was already familiar with these technologies.

Here’s a diagram of their second-generation application architecture:

As you can see in the diagram, the application has been broken down into microservices. The UI service communicates with other backend services via HTTP APIs; the various backend services communicate with various databases like MySQL and Redis for data persistence and/or caching. The backend services are responsible for implementing the functionality of a typical e-commerce site like orders, checking out, and managing the inventory catalog. While the choice of Kubernetes does afford Zephyr some much-needed functionality—like the ability to easily scale different services within the application—it also brings with it an added level of complexity that did not exist with their previous monolithic architecture. The Kubernetes manifest to deploy their application tops out at just over 1,000 lines of YAML!

Astute readers may note that Zephyr’s application looks very similar to this application; in truth, this series uses a fork of the AWS Containers retail store example as the application in this fictional scenario. (Many thanks to the AWS Containers DA team for their hard work here!) This is an admittedly over-engineered application, but it does afford the opportunity to address a range of customer scenarios in this blog series.

Managing Infrastructure and Application Deployments

As part of the switch to their new architecture, Zephyr decided they wanted to use Pulumi to manage both the infrastructure and the application deployment. Why Pulumi? Zephyr’s team recognized that moving to microservices on Kubernetes was going to make it more difficult to create reproducible, consistent implementations of their application. Zephyr’s team also knew that adopting infrastructure as code would help them with fast and repeatable deployments with little additional complexity. Another bonus was being able to use programming languages they already knew.

As they prepared to embark on using Pulumi, a couple of questions came up for the Zephyr team:

Where should they store their Pulumi code? The ability for Zephyr’s developers to use a full programming language to manage the infrastructure and deployment of their online store is a huge plus, but should this code go in the same repository as their application code? Or a different repository?
How do they address the need for multiple environments? It’s clear to the Zephyr development team that they’ll need multiple instances of the online store (for things like testing, QA, or development). What’s the best way to handle this when using Pulumi?

Let’s examine each of these issues in a bit more detail, and see how Zephyr chose to proceed.

Storing Pulumi Code

When it comes to answering the question of where to store Pulumi code relative to the application(s) it supports, there are two basic options:

In the same repository as your application code: This approach has the advantage of being simpler, but may not offer the necessary flexibility when dealing with multiple teams or varying levels of access control.
In a separate repository from your application code: Using a separate repository adds some complexity, but does provide greater flexibility to more easily address a variety of organizational requirements and varying levels of access control.

So which approach is best? That will depend on a number of different factors, many of which are outlined in the organizing projects & stacks documentation. Some other factors that users need to take into consideration include:

Who is responsible for maintaining the code? If the application code and the Pulumi code are managed by different teams, then using separate repositories may be the best approach.
What sort of access controls are needed? If the organization needs different access controls for the application code and the Pulumi code, then separate repositories are generally needed.
What is the relationship between the infrastructure resources managed by Pulumi and the applications? If the infrastructure is “shared,” meaning it is used by multiple applications, then keeping the Pulumi code in a separate repository may be preferred. If the infrastructure is dedicated to a specific application, then keeping the application code and the Pulumi code together may be preferred.
What about CI/CD? If an organization will use a CI/CD pipeline, that may affect how you organize your code. (This is something we’ll delve into more deeply later in this series.)

It’s worth stressing that the answers to these questions may change over time as organizations grow and their applications evolve. Addressing that change and its effects on an organization’s Pulumi projects and stacks is something we’ll be discussing later in this series.

The discussion above is working from the assumption that Pulumi users will store their Pulumi code in a version control system, like Git. Strictly speaking, using a version control system isn’t required for Pulumi to function (there is one exception we’ll touch on later in this series when we discuss Pulumi Deployments), but we very strongly recommend that all users use Pulumi in conjunction with version control.

In the case of Zephyr, their team felt like a monorepo approach (storing Pulumi code and application code together) was most appropriate for their specific requirements. Zephyr is a relatively small organization with a single team of developers who are responsible for managing both the cloud infrastructure as well as the development and deployment of the online store application. Zephyr’s team also decided to use a single Pulumi project—for now. (Be sure to follow this blog series to see how that evolves, and why!)

Addressing the Need for Multiple Environments

The second question Zephyr encountered is how to handle the need for multiple instances of their application. In addition to a production instance—which is the instance behind Zephyr’s public-facing online store—Zephyr also felt they needed an environment for the developers to use in testing changes to the online store.

This use case—needing to have multiple, separate instances of the infrastructure and applications created by a single Pulumi program—is exactly what Pulumi stacks were designed to address. Each stack is a separate instance of the resources created by a Pulumi program within a project. Further, each stack has its own independent state, and each stack has its own configuration values. Stacks can be short-lived (meaning the associated resources are also short-lived), or stacks can be long-lived. Aside from the cloud resources created by the Pulumi program, stacks are lightweight and simple to create or delete.

Zephyr decided to initially start with two stacks: a production stack (named “prod”) and a development stack (named “dev”). As you’ll observe throughout this series, this is a decision that is easily adjusted over time as your organization’s requirements change.

Organize your infrastructure with stacks

Model separate development, staging, and production environments as Pulumi stacks in a single project, each with its own state and configuration.

Get started

Viewing the First Iteration of Code

You can view the first iteration of Zephyr’s Pulumi and application code—the iteration that corresponds to the decisions described in this blog post—by navigating to the Zephyr App GitHub repository. From the branch/tag dropdown, switch from the main branch to viewing the blog/original branch.

From that GitHub repository, you can also choose to deploy the Pulumi code yourself. Full instructions for deploying the code are found in the repository.

Summarizing Best Practices: Organizing Code and Stacks

This article introduced you to Zephyr, the fictional company for this blog series, and reviewed the application architecture for Zephyr’s online store (a containerized application running on Kubernetes). Throughout the blog series, this application serves as the use case around which various recommended practices are discussed.

In this post the following guidelines were covered:

Use version control for your Pulumi code. Although it isn’t strictly required in many cases, a number of the benefits of adopting infrastructure as code are lost without the use of version control.
Place your Pulumi code into version control repositories in a way that aligns with organizational requirements. There is no “right” or “wrong” approach; choose the approach that enables you to satisfy your organization’s requirements around access control, code/repo ownership, and other organizational requirements.
Leverage Pulumi stacks to create multiple instances of cloud resources with a single Pulumi program. Pulumi stacks were created to address the need for organizations to create multiple instances of cloud resources, each with its own state and its own set of configuration data, from a single Pulumi program.

For now, Zephyr has decided to go with a monorepo approach—a single repository that contains both their application code and the Pulumi code to manage the infrastructure resources. All of the resources are defined in a single Pulumi project, with multiple stacks that correspond to development and production environments. Over the course of this series, we’ll see how Zephyr’s use of Pulumi changes as Zephyr grows and their application evolves.

The next Zephyr blog post will examine how the Zephyr team adjusts their use of stacks and Git branches to support developer productivity. Stay tuned!