Pulumi Blog: Open source

Celebrating 20,000 Stars: A Milestone for the Pulumi Community

Aaron Kao — Thu, 23 May 2024 18:48:50 +0000

Today, we’re excited to announce that the Pulumi open source project has crossed the incredible milestone of 20,000 stars on GitHub. 🎉 This is a huge achievement, and it wouldn’t have been possible without y’all - our incredible global community of developers.

Since its inception, Pulumi’s vision has been rooted in enabling teams to build and manage modern cloud infrastructure using familiar languages and tools. Over the years, your feedback, contributions, and passion have been invaluable in shaping Pulumi into what it is today. Seeing so many developers embrace our modern approach to cloud infrastructure management is really exciting. We are deeply grateful for the trust y’all have placed in us, and we will continue to push the boundaries in how teams manage cloud infrastructure.

Fueled by Community Contributions

Over the past 7 years, the Pulumi project has had 4,400+ contributors that made 75,000+ pull requests. Pulumi is depended upon by 6,600+ GitHub projects and supports over 160+ packages in the registry. It has been downloaded more than 100 million times by 170,000+ developers. In just the last month, Pulumi has been the most active Infrastructure as Code (IaC) open-source project compared to Terraform or OpenTofu. Pulumi has had 117 merged pull requests (67 TF, 83 OpenTofu), 82 closed issues (49 TF, 54 OpenTofu), and 53 new issues (27 TF, 29, OpenTofu).

Notable Features and Milestones

We also want to take this time to reflect on some of the notable features we shipped that got us here. The Pulumi roadmap is public so we can be transparent about the features being worked on and to encourage deeper collaboration with the community. Here are some of the most popular features that were added to Pulumi over the last 7 years.

Mar 2017 - Pulumi founded
Jun 2018 - Pulumi open source project launched
Aug 2018 - Kubernetes native provider
Oct 2018 - Series A fundraise
Oct 2018 - Pulumi Cloud launched
Nov 2018 - Stack Reference to allow access of outputs of one stack from another
Jun 2019 - AWSX as higher-level components for the AWS platform
Sep 2019 - Pulumi 1.0
Sep 2019 - Transformations to modify the properties and resource options for child resource of a component or stack
Nov 2019 - .NET support
Dec 2019 - CrossGuard for policy as code
Apr 2020 - Pulumi 2.0
May 2020 - Golang support
Sep 2020 - Azure Native Provider for same day access to new Azure features
Oct 2020 - Series B fundraise
Oct 2020 - Automation API exposes IaC through a programmatic interface
Oct 2020 - Import command to import resources not managed by Pulumi into a stack
Apr 2021 - First PulumiUP
Apr 2021 - Pulumi 3.0
Apr 2021 - Pulumi Packages to provide multi-language support for components and providers
Oct 2021 - Pulumi Kubernetes Operator
May 2022 - Java support
Nov 2022 - Pulumi Deployments to automate infrastructure deployments
Nov 2022 - YAML support
Dec 2022 - Hierarchical config
Apr 2023 - Pulumi Insights for intelligence, search, and analytics over any infrastructure
Apr 2023 - First Pulumi User Group (PUG)
Jul 2023 - Terraform conversion improvement as an easy way to migrate TF to Pulumi
Sep 2023 - Pulumi AI added to pulumi CLI
Sep 2023 - Go generics support added
Oct 2023 - Series C fundraise
Oct 2023 - Pulumi ESC to manage secrets and configuration complexity
Oct 2023 - Pulumi for Platform Teams through developer portals and more
Apr 2024 - Drift detection, TTL stacks, and scheduled deployments

We also shipped many other features that didn’t make this post, so please see our blog for all those details.

A Heartfelt Thank You

We are immensely grateful to our community for helping us reach this incredible milestone and making Pulumi a thriving ecosystem. Your contributions, feedback, and support have been invaluable in shaping Pulumi’s journey. We also want to thank the Puluminaries - our community leaders and advocates whose contributions have been pivotal to our success.

If you’re passionate about cloud infrastructure, consider joining the Puluminaries. As a Puluminary, you’ll gain direct access to our engineering team, preview roadmaps, and help contribute to code, content, and community support.

We look forward to continuing our innovations to democratize the cloud and empower developers worldwide. We encourage you to join our Slack community to engage with fellow developers and be part of the ongoing conversation about the future of how teams manage cloud infrastructure.

Thank y’all for being part of this journey. Here’s to the next 20,000 stars and beyond!

Pulumi adoption made easy with the new Migration Hub

Joe Duffy — Tue, 22 Aug 2023 00:00:00 +0000

Today we are launching Pulumi’s new Migration Hub, a comprehensive guide to help you seamlessly adopt Pulumi no matter where you are coming from, whether that’s Terraform, CloudFormation, … or even manually provisioned resources not yet governed by an infrastructure as code solution. Our new Expert Services group is ready to roll up their sleeves to help you adopt Pulumi faster. The Migration Hub also features many commercial offers for open source foundations, startups, and complementary migration, to minimize switching costs and risks. It’s never been easier to adopt Pulumi.

Why Pulumi?

Nearly 2,000 customers and over 130,000 practitioners have chosen Pulumi for their infrastructure as code needs. About half the time, these end users have an existing infrastructure as code solution that has stopped meeting their needs. The other half of the time, infrastructure may have been created in the cloud console (“click-ops”), using a cloud CLI, or some other manual provisioning tool, but now needs to become repeatable and scalable using infrastructure as code.

The most common reasons teams prefer Pulumi include:

Best-in-class productivity
Accessible to the entire organization, thanks to industry standard languages
Ability to deal with complexity at scale, thanks to stacks, config, sharing and reuse, and more
Perfect confidence thanks to policies and testing
Secure by-default, with built-in support for secrets
Highly programmable engine can be embedded anywhere, to build custom platforms and tools
Instant collaborative bridge between developers and infrastructure experts

Thanks to Pulumi, teams are able to do more with less, ship faster with confidence, tackle very challenging cloud infrastructure projects and scale with them, empower more of the team to be self-serve and collaborate more closely, and ultimately turn the cloud into a competitive advantage.

Pulumi is also true Apache 2.0 open source and we welcome anyone who wishes to build atop it.

Migration

Of course, it’s one thing to choose Pulumi, but another thing entirely to adopt it and get all of your infrastructure under its management, especially if you have existing infrastructure. That might include converting existing code, or even just importing existing manually provisioned cloud resources.

There are two major approaches to this challenge.

Self-Serve Migration

Pulumi offers many tools and techniques should you want to perform a migration on your own.

In the most general form, Pulumi provides an import command which can import any existing infrastructure no matter how it was originally created, click-ops included.

Pulumi also offers conversion tools for other popular infrastructure as code solutions, including HashiCorp Terraform, AWS CloudFormation, Azure Resource Manager (ARM), or Kubernetes YAML.

Although the time to migrate existing infrastructure varies wildly based on the size and complexity of your scenario, migrating a single cloud environment can often take as little as a few hours. And thanks to Pulumi being multi-language at its core, these tools will support you the same no matter which language you choose, JavaScript, TypeScript, Python, Go, C#, Java, and YAML alike.

To learn more about performing your own migration, please read this user guide.

Let Us Help You Out with Expert Services!

Although self-serve is an option, there can often still be uncertainty around this approach.

It may not be clear how much time a full migration will take, leaving the possibility that you’ll be left with multiple infrastructure as code solutions – either temporarily or even permanently. Although the tools are a huge help, they don’t always get you 100% of the way there. We also often hear that skilling yourself up on a one-time migration activity takes away from critical time spent on business initiatives.

All of this adds friction, risk, and uncertainty to the Pulumi adoption decision. The last thing we want is for the one-time migration to prevent you from adopting Pulumi. Thankfully, we are here to help!

Today we are announcing the formation of Pulumi’s Expert Services group. This team of industry and infrastructure as code experts is on standby and ready to roll up their sleeves to help you with migration, training, and other services efforts. No project is too small: we are currently working with customers with 100KLOCs of legacy infrastructure as code that needs to be migrated, for example.

Today we are also announcing an extension to our Terraform Migration Offer, in which we will bundle migration of your Terraform projects with your Pulumi Enterprise or Business Critical subscription. Today we are excited to announce that this same package is now available for all migration efforts, including AWS CloudFormation, Azure Resource Manager (ARM), …, or even infrastructure that was manually provisioned. We are also prepared to scale up from there depending on your needs.

Success Stories

Over time, Pulumi is becoming a more comprehensive platform, with new services like Pulumi Deployments for multi-cloud deployment workflows and Pulumi Insights for search, analytics, and AI for your cloud infrastructure. Pulumi also offers policies and secrets as core features of the platform. Enterprises are increasingly betting on Pulumi for their entire cloud management and platform needs.

By moving to Pulumi’s platform, many customers have seen considerable benefits:

Atlassian significantly increased developer velocity, increasing productivity by more than 2x and substantially decreasing the amount of time spent on maintenance tasks.

Mercedes-Benz got the whole team shipping faster and with more confidence by building a platform that helped developers be self-serve with infrastructure while scaling up to 100s of Kubernetes clusters worldwide.

Snowflake rapidly improved time to market by betting on Pulumi, empowering their engineers, and delivering the Data Cloud. Snowflake ultimately beat their IPO deadlines and knew legacy infrastructure as code wouldn’t do it.

Washington Trust Bank modernized their cloud infrastructure and practices, speeding up internal delivery, while also increasing their confidence thanks to policy as code guardrails.

Fauna adopted a born in the cloud mindset and was able to build a multi-cloud, highly scalable, modern SaaS application, actually increasing reliability even in the face of growing complexity.

In each of these cases, the customer evaluated Pulumi against alternative infrastructure as code solutions, but ultimately chose Pulumi due to the aforementioned benefits.

Special Offers

In addition to migration services, Pulumi offers two programs to help get up and running quickly:

Open Source: Any open source foundation or project can use Pulumi Team Edition free of charge.
Startup Discount: Any early stage startup gets $10,000 free credits and access to Pulumi Enterprise.

Calling All Partners!

As Pulumi’s market adoption has skyrocketed, we’ve begun to build out our official partner program. Although we don’t have anything major to announce (yet), we’d love to talk to you if infrastructure as code is an important part of your own customers’ needs. That includes ISV product integrations, global and regional systems integrators and consultants, and more. Note that Pulumi is already available in the AWS and Azure marketplaces. If you are interested in discussing a partnership, please contact us.

Get Started Today

Migrating from an existing solution can seem like a daunting task, but with Pulumi’s new Migration Hub and Expert Services, it has never been easier to adopt Pulumi than it is today. To learn more, visit the Migration Hub or simply contact us – we would love to hear from you.

Pulumi 💜's Open Source

Joe Duffy — Fri, 11 Aug 2023 00:00:00 +0000

Pulumi stands with the open source community.

We are unaffected by HashiCorp relicensing their software yesterday, and express profound support for many of our cloud friends who have been affected.

Pulumi is true open source, uses the Apache 2.0 license, and does not and never will depend on BSL-licensed software in any way, HashiCorp owned or otherwise.

We look forward to continuing to serve our new and loyal customers, always with open source and our amazing, fast-growing community at our core.

How Webiny Built a Serverless Application Framework

Adrian Smijulj — Thu, 03 Jun 2021 00:00:00 +0000

Building an open-source framework for building serverless applications has many challenges, one of which is deploying cloud infrastructure resources. In this article, learn how Webiny uses Pulumi to enable its users to easily deploy and develop applications built on top of serverless cloud technologies.

What is Webiny?

Webiny is an open-source framework for building serverless applications, completely written in TypeScript.

In other words, it is a framework that enables developers to architect, build, and deploy solutions on top of the serverless cloud infrastructure, like AWS Lambda, Amazon DynamoDB, Amazon S3, and others. Some examples of what you can build with Webiny today are:

full-stack applications
websites
REST and GraphQL APIs
microservices

With the framework, Webiny also built a content management system called the Webiny Serverless CMS. It is free, open-source, and by default, included with every new Webiny project.

Finally, it’s worth mentioning that, at the moment, Webiny exclusively works with AWS. But, bringing Webiny to other cloud providers like Microsoft Azure and Google Cloud Platform (GCP) is definitely something we’ll be looking at in the future.

Building the Framework - The Challenge

Building a serverless framework certainly has its challenges. One of which is the deployment of cloud infrastructure, which, in the world of serverless, is one of the fundamental operations developers need to do, even while the application is still in development.

Before the version 5 release, Webiny relied on an infrastructure provisioning technology called Serverless Components (not to be confused with Serverless Framework). Using a concept of components, it is a system that enables developers to deploy ready-made blocks of cloud infrastructure resources configured for a specific use-case. Some examples of components are Website, REST API, a Full Stack Application, and more.

At the time, it seemed that the overall developer experience and the simplicity of the Serverless Components product was the right choice for our users. So, we decided to go with it.

But, over time, we noticed several things that, unfortunately, made us question our decision. And if I were to choose the top four, I’d mention the following.

1. Hard to Customize

While the idea around components for different use-cases certainly sounded interesting, ultimately, it was not ideal for Webiny. We frequently received questions regarding further component configuration and customization, which was not easy to perform.

“How do I configure a different database?”, “How do I set a VPC?”, “How do I set up a specific configuration parameter for my S3 bucket?” were just some of the questions we received. The answer almost always included us having to implement support for a particular feature. To some degree, this worked, but it was really us patching the holes, which we knew would only get us so far.

2. Using YAML Instead of Code

At Webiny, we believe cloud engineering and infrastructure-as-code is the future, so, naturally, we wanted our users to use familiar code (ideally TypeScript) and development tools to define their cloud infrastructure. Within Serverless Components, the components are configured via YAML files, of which we were never really big fans, simply because of the fact that writing code instead of configurations gives more flexibility to developers.

3. Vendor Lock-In

Vendor lock-in was introduced at a later stage of the Serverless Components product development. Essentially, to deploy a component, the user is now forced to use a proprietary service that comes with it. And while, among other things, the service enables much faster deployments, from our perspective, we saw this as an additional point of friction for our users. Ideally, users should be able to set up Webiny with only an AWS account.

4. Lack of Support for Other Cloud Providers

At the time, Serverless Components’ support for different cloud providers just wasn’t there. This would be a major roadblock for us as we get to multi-cloud integration.

Discovering Pulumi

Once we realized Serverless Components wasn’t an ideal solution for Webiny, we started looking for an alternative.

The key features that we were looking for were the following:

open-source
use code (preferably TypeScript) to create cloud infrastructure resources
cloud infrastructure code should be flexible, meaning, users should be able to make adjustments to it
support for multiple cloud providers is a must
no vendor lock-in

So, the search began, and soon enough, we discovered Pulumi. And as it turned out, it not only checked all the boxes we had initially listed but much more:

different options when it comes to storing cloud infrastructure state files: a managed SaaS (pulumi.com) with a console and self-hosted, (for example Amazon S3)
ability to deploy cloud infrastructure into multiple environments using its concept of stacks
advanced features like Policy as Code and watch mode
great documentation
a vibrant community of developers and a responsive team behind the product

I probably forgot to mention a thing or two here, but most certainly, at this point, we were ready to make a move and start experimenting with Pulumi.

Integrating Pulumi with Webiny

For us, the integration of Pulumi with Webiny consisted of three steps:

figure out how to integrate Pulumi’s programming model with Webiny
integrate Pulumi CLI into Webiny CLI
figure out the optimal way of handling cloud infrastructure state files

Let’s see how we tackled each of these three steps.

Integrating Pulumi’s Programming Model With Webiny

In terms of project organization, every Webiny project consists of two key concepts: packages and project applications (or just applications).

Packages are just regular NPM packages, or in other words, folders with a package.json manifest file and some code. On the other hand, project applications are higher-level organizational units formed from one or more packages that form applications, as the name itself suggests. Applications consist of both application code and cloud infrastructure needed to run them.

A single Webiny project can contain multiple project applications, where each one has its own independent set of necessary cloud infrastructure resources that need to be deployed.

As an example, a default Webiny project includes three project applications:

API - essentially, represents your GraphQL HTTP API
Admin Area - the Admin Area (React) application
Website - the public website, a (React) application with static site generation (SSG) in the cloud

If we were to compare these by the complexity of the necessary cloud infrastructure to deploy, the Admin Area is the simplest. It only relies on a single Amazon S3 bucket and an Amazon Cloudfront distribution. On the other hand, the API project application is the most complex one as it needs to deploy multiple AWS Lambda functions, Amazon DynamoDB tables, Amazon S3 buckets, and more.

Ultimately, we decided that each project application should be a Pulumi project. With this approach, we gave developers the ability to both define and deploy respective cloud infrastructures independently. And, with the concept of stacks, they are also able to deploy them into multiple environments, which we’ll show in a moment.

Pulumi CLI and Webiny CLI

Once we understood how to use Pulumi concepts with Webiny’s project organization, the next step was integrating the Pulumi CLI with the Webiny CLI.

And although the Pulumi CLI is great, we still wanted to keep it super simple for the user and make the overall developer experience as straightforward and unified as possible. For starters, we didn’t want our users to install the Pulumi CLI manually. We wanted it to happen automatically.

We’ve created our version of the Pulumi SDK, which lets us use the Pulumi CLI more programmatically. It also enables us to make the Pulumi CLI download experience as smooth as possible. Essentially, whenever a user runs a deployment-related command, all of the necessary Pulumi CLI binaries and plugins are downloaded and stored inside the project’s node_modules folder.

Although we could’ve saved us some time by using Pulumi’s Automation API (instead of creating the mentioned Pulumi SDK), at the time, the Automation API was still in preview and not generally available. And since the setup we already had was working well, we decided to keep it and hopefully revisit the Automation API integration in the future.

Once we had that in place, we were ready for the next step, exposing a couple of fundamental deployment-related commands via the Webiny CLI. The following examples show us some of the commands users can use:

# Deploys a project application (internally runs "pulumi up")
webiny deploy {PROJECT_APPLICATION_FOLDER} --env {ENVIRONMENT}

# Previews a project application deployment (internally runs "pulumi preview")
webiny deploy {PROJECT_APPLICATION_FOLDER} --env dev --{ENVIRONMENT}

# Destroys cloud infrastructure resources previously deployed within
# the specified project application (internally runs "pulumi destroy")
webiny destroy {PROJECT_APPLICATION_FOLDER} --env {ENVIRONMENT}

# Starts watching specified project application and continuously deploys
# changes to the cloud (internally runs "pulumi watch")
webiny watch {PROJECT_APPLICATION_FOLDER} --env {ENVIRONMENT}

Note the --env argument appended to each command. With it and Pulumi’s concept of stacks, users can easily deploy their project applications into multiple environments.

Cloud Infrastructure State Files

The last piece of the puzzle was storing cloud infrastructure state files. Here we went with the following approach.

For local development, users’ cloud infrastructure state files are stored locally within their Webiny project using the Local Filesystem Backend, which we’ve seen worked great for developers.

On the other hand, for ephemeral environments spawned in CI/CD or long-lived environments like staging or production, through our documentation, we advise our users to use centralized and remote storage by using backends like Amazon S3 and even Pulumi Cloud (pulumi.com). Both backends have their pros and cons, and we let the users choose the one they want to use.

Show Me the Code

Let’s take a look at how it all works in the actual code.

As mentioned, by default, every Webiny project comes with three project applications: API, Admin Area, and Website, which are located in the api, apps/admin, and apps/website folders, respectively:

.
├── api # API
│ ├── code
│ ├── pulumi
│ ├── Pulumi.yaml
│ ├── tsconfig.json
│ └── webiny.application.ts
├── apps
│ ├── admin # Admin Area
│ │ ├── code
│ │ ├── pulumi
│ │ ├── Pulumi.yaml
│ │ ├── tsconfig.json
│ │ └── webiny.application.ts
│ └── website # Website
│ ├── code
│ ├── pulumi
│ ├── Pulumi.yaml
│ ├── tsconfig.json
│ └── webiny.application.ts
└── (...)

As we can see, every project application follows the same general organization. The two folders in each project are:

code - contains application code (one or more packages)
pulumi - contains cloud infrastructure (Pulumi) code

Furthermore, if we opened each of these pulumi folders, we’d see different cloud infrastructure resources clearly defined via multiple TypeScript classes. As a simple example, if we were to open the apps/admin/pulumi (Admin Area) folder, we’d find the following three files:

app.ts - deploys an Amazon S3 bucket that hosts the Admin Area (React) application
cloudfront.ts - deploys an Amazon CloudFront distribution for improved availability
index.ts - the Pulumi entrypoint file, imports classes defined within separate files

This code organization makes it much easier for developers to grasp the overall cloud infrastructure. It also makes it easier for them to adjust the code to their needs eventually.

A more complex example is the cloud infrastructure code in the API project application (api/pulumi), which deploys many different cloud infrastructure resources, like AWS Lambda functions, DynamoDB tables, ElasticSearch clusters, VPCs, and more.

But I will leave this up to you to check out, as pasting multiple chunks of code here might not be that productive.

Instead, I wanted to focus on a couple of other interesting and useful features that Pulumi and the infrastructure-as-code approach enabled us to create.

API Project Application - Different `dev` and `prod` Stacks

As mentioned, the API project application contains many different cloud infrastructure resources. But, as it turns out, when it comes to active development, we don’t have to unleash the full potential of the cloud, like we’re doing in staging and production environments.

For example, it probably makes no sense to deploy an Amazon ElasticSearch cluster into multiple availability zones (AZs) in most cases. A single AZ is enough for development purposes.

Another good example are VPCs and potentially NAT Gateways. Again, for development purposes, in most cases, we could probably get away without these too.

These extra resources make it slower to perform deployments from developers’ machines and incur extra costs for an organization because developers are deploying more resources, some of which are charged hourly.

So, with that in mind, we split the API project application’s cloud infrastructure into two stacks - the dev and prod. And, as you might’ve already guessed, only the prod variant will deploy absolutely all necessary cloud infrastructure resources.

What is even more interesting is the fact that this can be achieved with a simple if statement, which we placed in the index.ts entrypoint file:

// (...)

export = async () => {
 // (...)

 // WEBINY_ENV represents the environment which was passed upon running
 // the `webiny deploy` command. This is inserted automatically by the
 // Webiny CLI.
 if (process.env.WEBINY_ENV === "prod" || process.env.WEBINY_ENV === "staging") {
 // Import "prod" resources config and initialize resources.
 return await import("./prod").then(module => module.default());
 }

 // Import "dev" resources config and initialize resources.
 return await import("./dev").then(module => module.default());
};

Automatic Tagging of Cloud Infrastructure Resources

Another useful feature is the automatic tagging of the deployed cloud infrastructure resources. In other words, every taggable cloud infrastructure resource will be tagged with WbyProjectName and WbyEnvironment tags. For developers, this makes it much easier to see all of the deployed resources within their Webiny project.

We created a tagResources function, which essentially registers a global stack transformation via pulumi.runtime.registerStackTransformation function to achieve this.

It is also applied in the same index.ts entrypoint file we saw in the previous section:

import { tagResources } from "@webiny/cli-plugin-deploy-pulumi/utils";

export = async () => {
 // Add tags to all resources that support tagging.
 tagResources({
 WbyProjectName: process.env.WEBINY_PROJECT_NAME as string,
 WbyEnvironment: process.env.WEBINY_ENV as string
 });

 // (...)
};

Protect Feature

Finally, to protect our users from accidental deletions of mission-critical cloud infrastructure resources, we’ve used Pulumi’s protect feature:

The protect option marks a resource as protected. A protected resource cannot be deleted directly. Instead, you must first set protect: false and run pulumi up. Then you can delete the resource by removing the line of code or by running pulumi destroy. The default is to inherit this value from the parent resource and false for resources without a parent.

So, for Webiny users, we’ve made sure that this feature is automatically enabled for resources like DynamoDB tables, Cognito User Pools, Elasticsearch Cluster, and similar.

Future Plans

With the Webiny v5, we managed to introduce many significant improvements to both the Webiny Serverless Application Framework and Webiny Serverless CMS. However, we still have a lot of new cool things to unravel.

Webiny aims to be a multi-cloud compatible solution, and this is something Pulumi will significantly assist us with because deploying to multiple cloud providers is supported.

With that, we would also like to create various project templates, which would, for example, enable users to quickly start working on a new GraphQL or REST HTTP API, a simple React application, a microservice, and more. All of these require specific cloud infrastructure resources to be deployed, and again, this is where Pulumi will make our users’ lives much easier.

Finally, I already mentioned the Automation API as something that we might look into in the future, as it seems it’s something that can be seamlessly integrated with Webiny.

Conclusion

For Webiny, choosing Pulumi as the default infrastructure as code framework was certainly a big leap in the right direction.

It enabled us to leave the complexities of cloud infrastructure deployments to a product and company whose primary focus is exactly that. And this is what ultimately helped us completely focus on what we care about the most - creating an awesome serverless application framework.