Pulumi Blog: Luke Hoban

Announcing the General Availability of Pulumi ESC - Welcome to a New Era of Secrets Management

Arun Loganathan — Wed, 18 Sep 2024 08:59:00 +0000

At Pulumi, we’ve been at the forefront of revolutionizing infrastructure management and DevOps processes. We’ve built the industry’s most comprehensive secrets management solution for IaC, securing production cloud credentials and secrets for countless organizations. Now, we’re taking that expertise and applying it to the broader challenges of managing secrets at scale with an open ecosystem approach.

Pulumi Environments, Secrets, and Configuration (Pulumi ESC), launched in preview in October 2023, is now generally available.

This next-generation secrets management and orchestration service is built with a robust software engineering approach, designed to bring security and simplicity to even the most complex secrets management needs.

Challenges in Secrets Management

In today’s cloud-native world, secrets are the keys to the cloud estate, but managing them effectively is a growing challenge. Modern applications rely on a complex web of secrets, but traditional approaches to secrets management are struggling to keep pace. This complexity creates major challenges for security and engineering teams:

Secrets Sprawl: Secrets scattered across numerous systems and platforms make it nearly impossible to track their usage, enforce consistent security policies, and prevent unauthorized access, significantly increasing the likelihood of a breach.
Long-Lived Credentials: Storing long-lived credentials, especially locally, creates a significant security liability. If these credentials are compromised, they can grant attackers prolonged access to sensitive systems.
The Trade-off Between Speed and Security: Many security measures designed to safeguard secrets can inadvertently impede development velocity. Complex storage and retrieval processes create unnecessary friction, leading to frustration and delays.
Insecure .env File Sharing: Developers often copy/paste secrets across many different files, including formats like .env files stored locally on their machine or shared through insecure channels (like Slack), creating blind spots for security teams and increasing the risk of data leaks.

Introducing Pulumi ESC: A Modern Approach to Environments, Secrets, and Configuration

Pulumi ESC is the modern secret management platform designed to streamline and secure how you manage environments, secrets, and configuration across your entire organization. Built with flexibility and security at its core, Pulumi ESC empowers you to:

Centralize and Simplify Secrets Management:

Unify Your Secrets: Say goodbye to secrets scattered across systems. Pulumi ESC lets you pull and synchronize secrets from any source—HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, 1Password, and more—and consume them securely from anywhere.
Streamline Access and Control: Access secrets seamlessly via CLI, API, Kubernetes operator, the intuitive Pulumi Cloud UI, or directly within your code using our TypeScript, Python, and Go SDKs. Robust RBAC ensures the right people have the right access at all times.
Eliminate .env File Risks: Stop sharing secrets through insecure channels. Pulumi ESC provides a secure and centralized way to manage and access secrets, eliminating the need for risky .env files.

Boost Developer Velocity and Security:

Empower Developers, Not Slow Them Down: Pulumi ESC is designed for developer convenience without compromising security. Access secrets programmatically, integrate with your existing workflows, and leverage flexible authoring options: Pulumi Cloud Editor, VS Code Plugin, SDKs, the Pulumi Service Provider, and our REST API - the choice is yours!
Gain Complete Visibility and Control: Track all secret access and changes with detailed audit logs. Understand the downstream impact of changes and ensure your secrets are used securely and consistently across your organization.
Secrets as Code: Pulumi ESC supports flexible composition of environments by allowing imports between one another. Construct hierarchical environments that avoid the need to copy/paste secrets and configurations, ensuring that changes are consistently propagated across all related environments.

Built for the Modern Cloud, Open by Design:

Use It Your Way: Pulumi ESC integrates seamlessly with other Pulumi products and features such as Pulumi IaC and Pulumi Deployments for a unified infrastructure management experience. But its flexibility extends to any platform or toolset, including HashiCorp Terraform.
Embrace the Open Ecosystem: We believe in freedom of choice. That’s why Pulumi ESC is built on an open-source foundation (Apache 2.0 license) and designed to retrieve and synchronize secrets with various platforms seamlessly. We empower our customers to choose the best tools for their needs and supercharge their workflows with Pulumi ESC.

Customer Stories

Since launch, we’ve seen an exponential increase in our customers using Pulumi ESC. Our users have reduced the duplication of their configuration and secrets by over 90% and made Pulumi ESC a core part of running and managing their mission-critical infrastructure and applications.

Hear from our customers directly what they had to say about Pulumi ESC:

“With Pulumi ESC, our developers get dynamic AWS and Azure credentials on-demand, that removes the need for long-lived tokens and enhances security. Onboarding new developers is quick and secure, with no more manually filling in .env templates, since a common ESC dev environment with RBAC is all we need. The integration with the Pulumi SDKs is a huge productivity boost, and allows us to integrate secrets seamlessly across all of our development workflows." states Liam White, Platform Lead at Tetrate.”
“Pulumi ESC has been a lifesaver for us”; said JK Jensen, Team Lead at Mysten Labs, “it’s nice to throw everything behind an ESC environment and eliminate one-off granting IAM permissions and other issues related to static credentials. It gives us peace of mind knowing that we can grant permissions quickly and revoke easily, limiting the blast radius for any access.”

“With Pulumi ESC, our developers get dynamic AWS/GCP credentials on-demand; that removes the need for long-lived tokens and enhances security. ESC allows for Pulumi programs to share secure credentials and access secrets in their given cloud environments. ESC is a great configuration ‘/and secrets management sharing tool to allow for a DRY (Don’t Repeat Yourself) approach to development,” says Richard Genthner, Lead DevOps Engineer at Boost Insurance.

Watch the snippet from our PulumiUP Keynote that walks you through how to achieve all these benefits our customers have experienced. Sign-up here to watch the full keynote.

Beyond Preview: The Evolution of Pulumi ESC

Since the launch of Pulumi ESC in preview, over the last 11 months, we have launched numerous features and enhancements driven by our customers. Some of the major ones include:

SDKs for TypeScript, Python, and Go to leverage Pulumi ESC within all your applications
Versioning that maintains full history of the changes you have made to your secrets and configuration and rollback to specific versions when needed
Version Tags allow you to assign tags and import specific versions by tags within your applications such that moving tags across versions changes the configuration but requires zero changes in your application code
Syncing your secrets to several external sources including GitHub and AWS SecretsManager
Kubernetes secrets operator to use Pulumi ESC within your Kubernetes applications to retrieve secrets and configuration using external secrets operator during runtime
VS Code Extension to author and view your environments directly from where you code
Webhooks to trigger your automated CI/CD workflows, restart applications with the latest secrets and configuration, notify your teams
Pulumi OIDC Trust relationship to get Pulumi short-term credentials using your existing Identity provider
Projects and Environment Tags to help you organize your Pulumi ESC environments and allow them to scale as your organization evolves
1Password integration to pull secrets stored in 1Password during runtime
In-built Document Editor Enhancements to ease authoring with Pulumi Cloud
Table Editor to support users who prefer point-and-click UI-driven method to secrets management
Pulumi Stack reference provider offering tighter integration with Pulumi IaC
Files support that allows getting Kubernetes credentials dynamically
Audit logs to monitor your environments

Pricing

Starting October 1st, 2024, Pulumi ESC will be billed per secret. We have heard from customers that this is a familiar pricing dimension that aligns to the value customers get out of ESC. To help you get started, we have included a generous free tier of secrets and API calls.

	Free	Team	Enterprise	Business Critical
Price per secret/month	25 free	$0.50	$0.75 / Custom	Custom
Price per plaintext config	Free	Free	Free	Free
Price per 10K API calls	10K free	$0.10	$0.10	$0.10
Commitment discounts			✔	✔

Check out our pricing page for more information about the various packages available. Contact us if you are interested in a demo, have any questions, or would like to discuss volume discounts.

Conclusion

Pulumi ESC represents a significant advancement in secrets and config management. By addressing the challenges faced by organizations today, such as secrets sprawl and the trade-off between speed and security, Pulumi ESC offers a modern, flexible solution. Its unique features, including support for dynamic, short-lived credentials and seamless integration with major cloud providers, GitHub, etc. make it an invaluable asset for teams looking to maintain agility without compromising on security. Pulumi ESC is set to redefine the standards of secrets management, enabling organizations to achieve their security and operational goals more effectively.

We can’t wait to see how you leverage Pulumi ESC. Sign-up for Pulumi Cloud to start using Pulumi ESC and check out our getting started guide to learn more about Pulumi ESC.

Introducing: Support For Using Any Terraform Provider with Pulumi

Luke Hoban — Thu, 29 Aug 2024 10:00:00 -0600

For the most up-to-date information about using any Terraform provider with Pulumi, see the Any Terraform Provider documentation.

One of our core goals at Pulumi is to provide access to manage any cloud infrastructure with a single unified programming model. Whether it’s multi-cloud (AWS+Azure+Kubernetes), hybrid cloud (GCP+VMWare+Cisco), or managed services (Databricks+GitHub+Cloudflare), Pulumi makes it easy to deploy and manage infrastructure across all of your cloud environments using any of the 150+ cloud providers in the Pulumi Registry.

We’re excited to take this even further by introducing support for using any Terraform or OpenTofu provider from within your Pulumi programs. If there is a long tail Cloud or SaaS platform that has a provider for those ecosystems, it now works with Pulumi as well. And if your organization has built your own custom Terraform or OpenTofu provider to support an internal cloud platform, you can use it from Pulumi as well, without having to publish it to any registry.

Terraform providers are the closest thing the industry has today to a “defacto standard” for describing the resource model of a cloud platform. The vast majority of these providers are implemented by cloud providers, partners, or cloud ecosystem contributors - as a way to describe what resources are available in their platform that can be managed via various kinds of infrastructure tooling. These investments become even more impactful when the work done to define the resource model can be widely leveraged by other tools for a variety of purposes - from IaC to Policy to Asset Inventory and more. There is nothing innately Terraform-specific about these providers - they are simply a standard resource model and the implementation of CRUD (Create-Read-Update-Delete) operations against that model.

At Pulumi, we’ve supported bridging Terraform providers for use in Pulumi from the beginning. Pulumi’s provider model is even more expressive than Terraform’s and we have a defined way to map Terraform providers (as well as other provider ecosystems) into the Pulumi provider model to be used by Pulumi’s CLI and deployment orchestration engine. Many of the most used providers in the Pulumi ecosystem are bridged Terraform providers. However, until now, bridging a new provider required a decent amount of up front and ongoing work, to maintain a separate repository, build and push separate releases, and publish resulting SDKs to every language’s package manager. This meant that while the largest and most used Terraform providers were available in Pulumi, the long tail of providers were not as easily accessible. With this launch, we are now opening up easy access to all of these providers.

Using any Terraform or OpenTofu provider takes just a few seconds using a single command:

$ pulumi package add terraform-provider <your-terraform-provider>

This is enabled by our new Local Packages feature, which lets you generate Pulumi Packages locally into your Pulumi project, instead of relying only on packages published to the Pulumi Registry. Support for generating local packages is available for all Pulumi languages.

Local packages are available as a convenient alternative to the existing support for authoring Native or Bridged Pulumi Packages. You can read more about all of the options available to authoring and using Pulumi Packages in the Pulumi Packages documentation, or about how to generate a local package for a Terraform or OpenTofu provider in the Any Terraform Provider docs.

Walkthrough

Imagine you are working on a new cloud infrastructure project, and come across a technology you want to adopt, but don’t yet see a Pulumi provider in the Pulumi Registry. What do you do?

Let’s walk through a concrete example. We’ll work with a Python project:

$ pulumi new python

Planetscale is an exciting new fully-managed MySQL database platform. Although there isn’t yet an official Pulumi provider for Planetscale in the Pulumi Registry, we can easily use Planetscale from Pulumi if we want to. First, we generate a local package using the pulumi package add command.

$ pulumi package add terraform-provider planetscale/planetscale
Successfully generated a Python SDK for the planetscale package at /Users/lukehoban/github/lukehoban/anytftest/sdks/planetscale

To use this SDK in your Python project, run the following command:

 echo sdks/planetscale >> requirements.txt

 pulumi install

You can then import the SDK in your Python code with:

 import pulumi_planetscale as planetscale

We ask to use the terraform-provider package, which implements support for any Terraform or OpenTofu provider, and then ask specifically for support for the planetscale/planetscale provider. By default, this is pulled from the OpenTofu registry, but you can also specify a local path to the provider, or a fully-qualified reference to any Terraform Registry API-compatible server.

As we see from the output, this generates a local ./sdks/planetscale folder with a Python SDK for using the Planetscale provider.

After running the two commands provided in the output, we now have the local SDK incorporated into our project:

$ echo sdks/planetscale >> requirements.txt
$ pulumi install

We can add some code that uses this new SDK:

import pulumi
import pulumi_planetscale as planetscale

db = planetscale.Database(
 resource_name="mydb",
 organization="lukehoban",
 cluster_size="PS-10"
)

pulumi.export("db_url", db.url)

We get strong typing, completion lists, and error reporting over this SDK from within our IDE.

Finally, we can deploy our code, deploying a new Planetscale database directly from Pulumi:

$ pulumi up
Updating (luke-pulumi-corp/dev2)

View in Browser (Ctrl+O): https://app.pulumi.com/luke-pulumi-corp/anytftest/dev2/updates/2

 Type Name Status
 pulumi:pulumi:Stack anytftest-dev2
 + └─ planetscale:index:Database mydb created (1s)

Outputs:
 + db_url: "https://api.planetscale.com/v1/organizations/lukehoban/databases/mydb-77af236"

Resources:
 + 1 created
 1 unchanged

Duration: 3s

Clicking on the link, we see our new managed database ready for connections! Within just a few seconds, we’ve incorporated support for this new service into our Pulumi infrastructure!

Planetscale is an exciting example, but it is just one of thousands of 3rd party (published in a public registry) and 2nd party (published internally within an organization) providers that are now available for use within Pulumi.

Why Support All Terraform Providers?

Pulumi has a rich and expressive Native Provider model for building providers directly against the full capabilities of the Pulumi Infrastructure as Code resource model. Many of the most important Pulumi providers are built directly on this foundation, including our Azure Native and Kubernetes providers. So why are we excited to support using Terraform and OpenTofu providers from Pulumi?

In short, we believe in giving Pulumi users access to every ecosystem of great providers and components available across the industry and in meeting partners where they are. If a partner or user has built something exciting for managing infrastructure in the cloud, we want Pulumi users to benefit from it. Similarly, we want the partner or user to easily extend the value of their provider to also be able to reach the Pulumi user base, and all of the benefits that Pulumi can offer for Infrastructure as Code management, with no extra effort on their part.

We’ve embraced this approach in many other areas as well:

Pulumi enables deploying and managing the lifecycle of any Kubernetes YAML file or Helm Chart. There’s a great ecosystem and library of Helm charts, and we make those all available to Pulumi users without any wrappers or indirections.
We introduced the Pulumi AWS CDK Adapter to enable Pulumi users to incorporate AWS CDK constructs directly into their Pulumi programs without rewriting in Pulumi first.
And, of course, Pulumi offers features for deploying and managing Azure Resource Manager templates and CloudFormation templates, enabling easy deployment of existing cloud provider IaC artifacts.

With the new support for Any Terraform Provider, we’re bringing this same experience to the Terraform and OpenTofu provider ecosystems.

Use any Terraform provider

Add any Terraform or OpenTofu provider to your Pulumi program with a single command, and manage it alongside the rest of your cloud infrastructure in one workflow.

Learn more

How Does This Work?

Two new technologies in the Pulumi open source project underpin support for using any Terraform or OpenTofu provider.

Parameterized Providers offer support for building a single provider that can implement many different “flavors.” For example, the terraform-provider provider can be parameterized to provide support for any Terraform or OpenTofu provider.

Local Packages offer the ability to generate a Pulumi SDK for the language of your Pulumi project into a local folder, which can be directly incorporated into your project via your existing package manager along with pulumi install.

Local packages are versioned when you include a version in the pulumi package add command, and parameterized providers support many different formats for their parameters. For example, the most general form of the command to use an existing Terraform provider is:

$ pulumi package add terraform-provider [<registry>/]<author>/<name> [version]

Or:

$ pulumi package add terraform-provider /path/to/my/terraform-provider-binary

What’s Next?

Today’s release is a big step in opening up access to even more providers for Pulumi users. But there is still a lot more we have planned to build on top of these new foundations.

First, with locally generated SDKs, the documentation is available within the language SDK you generate (and so lights up in your IDE or development environment!). Still, there is no static docs page available in the Pulumi Registry to find and link to on the web. In the future, we plan to offer the ability to publish documentation for locally bridged providers in the Registry. (tracked in registry#5302) In that case, the registry entry will be only the docs - the SDKs and provider binary will not need to be published/hosted in the Registry itself.

Second, the underlying Local Packages and Parameterized Providers technologies for building different flavors of a single provider are applicable to many other use cases. In addition to having a single provider that can be “parameterized” to act as any Terraform provider, we have plans to apply this technology to additional use cases, such as generating strongly typed providers over any Kubernetes CRD, any Helm Chart, any other Pulumi program, and many more ecosystems of exciting infrastructure components which can be deployed via Pulumi.

Take the new support for using any Terraform or OpenTofu provider from Pulumi for a spin today:

Introducing Pulumi Copilot: Intelligent Cloud Management

Luke Hoban — Wed, 12 Jun 2024 04:00:00 -0700

Note: This post discusses Pulumi Copilot, which Pulumi Neo has replaced. Learn about Neo →

AI is transforming how users interact with every category of technology today, and cloud infrastructure is no exception. Last year we launched Pulumi AI to combine generative AI with Pulumi’s knowledge of cloud infrastructure, helping users solve complex cloud development problems using Infrastructure-as-Code. Pulumi AI has seen rapid adoption and engagement, with tens of thousands of users leveraging Pulumi AI over the last year.

Today, we’re excited to introduce Pulumi Copilot, a huge leap forward in applying AI to cloud infrastructure management. Pulumi Copilot is a new conversational chat interface integrated throughout Pulumi Cloud, enabling Pulumi Cloud users to quickly accomplish a variety of cloud infrastructure management tasks by leveraging the power of large language models plus the rich capabilities of Pulumi Cloud.

Through Pulumi Copilot, you can explore your cloud infrastructure and gain insights across an incredible breadth of use cases, including:

Pulumi Copilot Key Features

Access any data in Pulumi Cloud

The state of every resource you are managing with Pulumi across any Cloud, any account, and any region. With Pulumi Insights’ Cloud Supergraph support for 160+ cloud providers, this offers an unprecedented breadth of cloud infrastructure data to explore and interrogate with Pulumi Copilot.
Pulumi stacks, projects, updates, deployments, environments, policies, audit logs and more - enabling historical understanding of what happened when, by who, and why across all of your cloud engineering systems managed by Pulumi.

Pulumi IaC Authoring and Deployment

The same great Pulumi AI features for authoring IaC are now available inside Pulumi Copilot as well, enabling you to quickly solve new IaC problems within Pulumi Cloud, and even deploy code directly from Pulumi Copilot.

Access cloud metadata from the clouds themselves

Through the use of new skills, Pulumi Copilot can access cloud metadata in real time in AWS, Azure, Kubernetes, and more, allowing it to join Pulumi’s IaC world view with information about usage, costs, and more – as well as infrastructure not yet under the management of Pulumi.

Pulumi Copilot combines the best generative AI models available in the industry today with Pulumi Copilot “skills” which enable Pulumi Copilot to access the data and actions from within Pulumi Cloud needed to help you with your questions and explorations. Pulumi Copilot also incorporates the context of where the user is in the Pulumi Cloud console to easily answer questions about “this stack”, or “the latest update”, offering an even more natural, conversational and persistent experience across Pulumi Cloud.

Pulumi Copilot is Enterprise-Ready on day one - it adheres to all the same identity and role based access control (RBAC) rules that exist across Pulumi Cloud, ensuring that users only have access to stacks, environments and resources that they have permissions for within Pulumi Cloud. This means that each organization gets AI generated responses based solely on their own organizational data.

Pulumi Copilot is available now in public beta, and is free while in beta for all organizations in Pulumi Cloud. Organization administrators can turn on Pulumi Copilot for their organization by going to Settings > Access Management > Pulumi Copilot within the Pulumi Cloud console.

Try Pulumi Copilot now at https://app.pulumi.com/.

Walkthrough of Pulumi Copilot

Let’s jump in and take Pulumi Copilot for a spin!

We can start off simple - it’s a conversational chat, so we can introduce ourselves. Pulumi Copilot knows who we are from context and we get a light hearted programming related joke.

Perhaps we need to gain access to additional infrastructure within Pulumi Cloud and need to contact an admin, we can ask who the admins are.

We get a hyperlink at the end, enabling us to jump directly to the members page, where we can see all of the details about the members and admins of the organization. Pulumi Copilot helps to not just get information, but then navigate within the Pulumi Cloud console to continue to explore.

Pulumi Copilot has access to everything that Pulumi Cloud manages, including Pulumi ESC environments. We can find all of the environments we have that manage access to GitHub through a natural language query, and then ask a follow up question to get a direct link to where each environment is defined.

If we do a Pulumi update, and it fails, we often get a complex error message, caused by some combination of a coding problem or a cloud provider configuration problem or a transient failure in cloud providers backing service. We can use Pulumi Copilot to ask why an update failed, and get a plain language explanation, leveraging the AI’s existing knowledge of programming languages and cloud providers, combined with access to all of the update logs from within Pulumi Cloud. Notably, we can ask why “this” update failed, taking advantage of the context of the page we are on within Pulumi Cloud to understand which update we are referring to.

Perhaps we received an email from our compliance team telling us there was an issue with the VPC with id vpc-04a116f7075fb7ca5, but we have no idea where that is. Given we have hundreds of AWS accounts and dozens of regions in each of these accounts, finding this resource isn’t simple. We can just ask Pulumi Copilot, and it immediately tells us what AWS account it is in. We can follow this up with a question about which Pulumi project and stack is managing it, and get a link to go directly to the stack to learn more about how it is configured.

All of the capabilities of Pulumi AI for writing Pulumi IaC code are also available in Pulumi Copilot, so we can ask how to write a program for a new use case. But we can also have Pulumi Copilot incorporate information from an existing stack, leveraging multiple Pulumi Copilot skills to help with this task.

If we are thinking about removing some infrastructure we may want to know what other infrastructure potentially depends on it. We can get a list of stacks which depend on our stack, with hyperlinks directly to those stacks so we can explore their dependencies and take action to update them.

If we’re getting started on compliance with a new compliance framework, we can get guidance on areas to focus on for compliance review, ahead of applying more formal compliance tools like Pulumi Crossguard Compliance Ready Policies.

And we can get insights into what infrastructure is exposed to the internet which we may need to look deeper into with a quick question:

And our questions don’t have to be limited to English. Early Spanish-speaking users have found themselves working with Pulumi Copilot in their native language, getting responses automatically in the same language of their question.

All of these examples are just scratching the surface of what Pulumi Copilot can already do today. One of the most exciting things about conversational chat interfaces is their wonderfully wide surface area of potential applications - almost any question you might have related to Cloud Infrastructure managed within Pulumi Cloud is something Pulumi Copilot can help with!

Future of Pulumi Copilot

All of the examples above are things you can do today, but we’re not stopping there. There’s a lot more coming up, and we wanted to give a sneak peek into some of the directions we’re taking Pulumi Copilot based on early customer feedback.

Taking Action

In the initial Beta, Pulumi Copilot is restricted to get data and answer questions, but cannot take action directly, beyond guiding users to a place within the Pulumi Cloud console to take the action themselves. In the near future, we will be extending Pulumi Copilot with the ability to propose actions and then act on them with user approval.

Check out pulumi/cloud-requests#414 for more details.

Pulumi CLI Integration

One of the most common requests we have gotten from early Pulumi Copilot users has been for integration into the Pulumi CLI where many users spend much of their time day-to-day. We have started experimenting with CLI integration to help with diagnosing Pulumi update failures, and will be adding this to the CLI in the near future.

Check out pulumi/cloud-requests#415 for more details.

Pulumi Docs Website Integration

Pulumi Copilot is initially offered as part of Pulumi Cloud. But we are working to extend Pulumi Copilot across the rest of pulumi.com, including integration with the Pulumi Docs website. A new Pulumi Docs skill will allow Pulumi Copilot to help users learn Pulumi concepts and chat with Copilot about these to go deeper on concepts than what is already provided directly within the Pulumi Docs website.

Check out pulumi/cloud-requests#417 for more details.

Cloud CLI Skills

While Pulumi Copilot already has access to all the Cloud Infrastructure you are managing with Pulumi Cloud, there are many cases where it is useful to reach out to AWS directly to ask questions, get operational metrics, and take action. We are already working on extending Pulumi Copilot with the ability to run aws and kubectl CLI commands, using short-lived credentials from one of your Pulumi ESC environments. A sneak peek of that in action - showing how the AWS skill, combined with the ability to get details about resources and ESC environments from within Pulumi Cloud can enable collecting operational metrics from AWS and presenting them for analysis.

Check out pulumi/cloud-requests#416 for more details.

Conclusion

We’re excited to introduce Pulumi Copilot, the first conversational AI for general multi-cloud infrastructure management. Pulumi Copilot is the next big step toward reimagining cloud infrastructure management via AI-based user experiences. We’re just getting started, with many improvements and new capabilities coming soon on top of the core Pulumi Copilot foundation. Pulumi Copilot is brand new and improving quickly in response to user feedback. If you have questions or suggestions:

Get started with Pulumi Copilot in Pulumi Cloud.

Secrets and Configuration for Software Engineers: Pulumi ESC Versioning, SDKs and More

Luke Hoban — Wed, 05 Jun 2024 04:00:00 -0700

Last October we launched Pulumi ESC, a new secrets and configuration management offering from Pulumi that manages and tames secrets and configuration complexity across all of your cloud applications and infrastructure. We’ve seen incredible adoption of ESC over the last 6 months, with thousands of developers now managing their secrets and configuration via ESC.

Today, we are excited to announce the most significant collection of Pulumi ESC enhancements since the initial launch, with three major new features that bring best-in-class software engineering capabilities to bear on your Secrets and Configuration management needs.

Versioning: Pulumi ESC now supports versioning of environments, allowing you to see and audit every change to the secrets and configuration for an environment, pin references to an environment to a specific version or version tag, and safely roll back an environment to a previous version.
SDKs: Pulumi ESC now has SDKs available for Python, TypeScript/JavaScript and Go, enabling ESC to be used directly within applications, tools and services to retrieve and manage secrets and configuration values at runtime.
Environments as Code with IaC: New support for defining and managing Pulumi ESC environments, secrets, and configuration from within Pulumi IaC programs allows source-controlled environment specification and managing secrets and configuration lifecycles via code.

All three of these new features double down on Pulumi ESC’s unique approach to secrets and configuration management, which combines a hierarchical and composable approach to defining environments, integration with a wide variety of existing secrets stores and a wide variety of secrets consumers, and a rich configuration-as-code approach.

Together, these provide the richest experience in the market today for developers to apply software engineering best practices to their secrets and configuration management, enabling the management of secrets and configuration complexity at scale across complex collections of environments and applications.

What is Pulumi ESC?

Pulumi ESC is a new category of configuration-as-code product, motivated by our experience working with hundreds of Pulumi IaC customers to address their needs in managing secrets and configuration at scale within their Pulumi infrastructure and across other cloud applications and infrastructure projects.

Pulumi ESC enables teams to aggregate secrets and configuration from many sources, manage hierarchical collections of configuration and secrets (“environments”), and consume those configuration and secrets from a variety of different application and infrastructure services. Pulumi ESC works hand-in-hand with Pulumi IaC to simplify configuration management, but also works independently from Pulumi IaC, as a solution for managing environments, secrets and configuration for any application or infrastructure project.

ESC Versioning

Safely managing the rollout of secrets and configuration changes to applications and infrastructure is one of the fundamental and critically important jobs that a software engineering team does. And yet, state-of-the-art secrets and configuration versioning still typically involve inflexible and bespoke systems for modifying configuration and/or copy/pasting across different environments. Pulumi ESC offers the ability to compose environments to avoid copy/paste, and also now supports rich versioning features to enable full auditability and safe rollout of configuration changes across environments.

There are two core features to Pulumi ESC versioning:

Visibility into the history of every change made to an environment.
The ability to pin any environment reference to a specific revision, or specific tagged version.

These offer the same sort of rich versioning typically seen in other software engineering ecosystems, like Git, Docker, or NPM - but now applied to your secrets and configuration management as well.

You can see the history of changes to an environment:

And then see the diff between any two versions of an environment:

Revisions can also be tagged to provide semantically meaningful names to specific revisions. These tags can be moved later on to point to new versions when you are ready to roll them out to consumers. The latest tag automatically always points to the latest revision.

Most importantly though, you can also then refer to a specific version of an environment anywhere you reference one.

When you run a command with pulumi env run you can specify providers@v2 to pick the v2 tag instead of the default latest:

$ pulumi env run acmecorp/providers@v2 -- aws sts get-caller-identity
{
 "UserId": "AROAY65FYVYP33NYY5FOU:environments-session",
 "Account": "358361613858",
 "Arn": "arn:aws:sts::358361613858:assumed-role/environments-oidc/environments-session"
}

When you import another environment you can import a pinned version. This enables changes to be made to the target environment without automatically being picked up by the consumer. This allows phased rollout, with careful testing of new major versions before they are pushed out to production environments.

imports:
 - ai-staging-aws@v3
values:
 pulumiConfig:
 clusterPetName: gold-kiwi
 clusterTags:
 cluster: ${pulumiConfig.clusterPetName}
 aws:defaultTags:
 tags: ${pulumiConfig.clusterTags}

Read more about Pulumi ESC versioning in the launch blog post.

ESC SDKs for Python, TypeScript/JavaScript and Go

Pulumi ESC was designed from the beginning to support consuming secrets and configuration from any application or infrastructure project. While we have seen lots of early usage combining ESC with Pulumi IaC, the two are not fundamentally tied together at all, and ESC is designed to serve the needs of any complex configuration and secrets management workload.

One of the most important places where secrets and configuration are consumed is in application and service code at runtime. This can take many forms - a few illustrative examples include:

An application that pulls configuration data at startup time
An operational tool that pulls in temporary credentials to perform operations within a cloud account
A control plane service that pushes configuration and secrets changes into other systems to keep configuration up to date and secrets actively rotated

In all cases, it’s important to be able to embed ESC within your application code. At launch, ESC supported a public REST API, but now this has been extended with convenient and easy to use SDKs in the most popular languages - Python, TypeScript/JavaScript and Go.

To use ESC from within your applications, just import the SDK and add a few lines of code:

import pulumi_esc_sdk as esc
import os

configuration = esc.Configuration(access_token=os.environ['PULUMI_ACCESS_TOKEN'])
client = esc.EscClient(configuration)

env, vals, txt = client.open_and_read_environment("acmecorp", "dev-stacks")

print(vals['aws']['secrets']['stripeApiKey'])

Running this gets the Stripe API Key for this environment:

$ python3 __main__.py
rk_test_3c88Pc8ZfdBredactedrEhDU11b42wI5KZ

Lots more details on the features and capabilities of the new Pulumi ESC SDKs in their launch blog post.

Environments as Code with IaC

Pulumi ESC helps to manage configuration and secrets complexity at scale via composition and reuse. But many users still want to go further and version control the definitions of their environments, secrets and configuration.

To enable this, along with many other powerful code-based environment management scenarios, we’ve added support for defining and managing Pulumi ESC Environments via Pulumi IaC itself. New Environment and EnvironmentVersionTag resources are available now in the Pulumi Cloud Provider in the Pulumi registry.

Managing the desired state of environments via infrastructure-as-code programs brings the most robust tools for managing version-controlled desired state definition possible to the realm of secrets and configuration management. And leveraging Pulumi’s best-in-class IaC capabilities and programming model offers the deepest possible software engineering capabilities for defining and managing configuration and secrets as code across environments.

Environment contents can be specified either inline, or in external files, and changes deployed into your Pulumi Cloud account via a simple pulumi up.

import * as service from "@pulumi/pulumiservice";
import * as pulumi from "@pulumi/pulumi";

var environment = new service.Environment("testing-environment", {
 organization: "service-provider-test-org",
 name: "testing-environment-ts",
 yaml: new pulumi.asset.StringAsset(`\
values:
 myKey1: "myValue1"
 myNestedKey:
 myKey2: "myValue2"
 myNumber: 1`,
 ),
});

For more on managing Pulumi ESC Environments using Pulumi IaC, check out the launch blog post.

Conclusion

Pulumi ESC is a secrets and configuration management offering that provides best-in-class software engineering features for managing secrets and configuration complexity at scale. Today, we are extending this even further with:

Rich versioning support for auditability and controlled configuration change rollout.
SDKs to easily embed ESC at runtime within applications, tools and services.
Support for source controlling environment specifications using Environment resources in Pulumi IaC programs.

Pulumi ESC has already been adopted by thousands of engineers to manage secrets and configuration complexity. Check it out and get started with Pulumi ESC today!

You can learn more about Pulumi ESC at:

The Present and (Near) Future of AI and Infrastructure as Code

Luke Hoban — Wed, 14 Feb 2024 07:00:00 -0800

Note: This post discusses Pulumi Copilot, which Pulumi Neo has replaced. Learn about Neo →

AI is impacting almost every industry today, and for good reason - we are seeing fundamentally new experiences being made possible across a wide variety of products, and a set of new AI capabilities that promise even more incredible change in the near future.

Software development is among the earliest and most prominent fields to realize the benefits of AI, evidenced by the rapid adoption of tools like Github Copilot which is now one of the most heavily adopted developer tools of all time. Developers are benefiting from an incredible increase in their productivity with better scale and faster time to market.

We’re seeing the impacts of AI in the cloud Infrastructure development space in two impactful and complimentary directions:

🤖➜☁️: AI is transforming how we author, build and manage cloud infrastructure
☁️➜🤖: Cloud infrastructure tooling is changing how we build and deliver AI-based applications

At Pulumi, we’ve already seen profound impacts from AI in both of these directions.

Infrastructure engineers are using Pulumi AI to discover solutions for cloud infrastructure needs, leading to a dynamic experience where you can generate solutions in seconds with a few simple prompts, and using tools like GitHub Copilot to quickly iterate on and test their infrastructure.

At the same time, thousands of organizations are using Pulumi and Pulumi Cloud to manage the fundamentally cloud-native infrastructure and services available to build and scale AI-based applications - managed vector databases, elastic compute services, model serving, streaming API management services, and workflow management - all evolving at the rapid pace of the AI space, and scaling with the demand that organizations are seeing for AI.

In this article, we’ll dig into both of these trends and the impacts that they are already having today, as well as the trends we are expecting to see (and build) for the near future.

AI transforming Infrastructure as Code

Today

The scale of the cloud is truly incredible, and the sheer size of the AWS API is huge relative to almost any other computing platform, while AWS is just one of over 135 cloud providers for which Pulumi supports building and managing infrastructure. The result is more than 50,000 infrastructure resource types that can be managed, and over 1,000,000 configurable input properties on those resources. In practice, cloud engineers face a daunting task as they figure out how to combine these resources and properties to solve their unique problems. Finding an example or guidance for how to combine these together, given the enormous variety of possible ways, can be hard or impossible.

As a result, one of the most fundamental challenges in enabling developers to use the cloud is helping them discover how to solve their problems using the enormous collection of available cloud infrastructure building blocks. For example, two years ago, I tried looking up how to “deploy Metabase on Azure using managed PostreSQL and containers” to solve a problem I was working on. I couldn’t find any content on the internet that directly answered this. Not just for Pulumi, there was no answer using any other IaC or deployment technology.

AI is perfectly suited for this challenge. With Pulumi AI, I can ask a question and get back an answer to the exact things I’m looking for - even if that specific use case has never been answered before. Pulumi AI doesn’t always get it completely right on the first try, but it typically gets very close, and points me in the direction I’m looking for. When I try its suggestion and get errors, I can ask it to fix them to iterate to a working solution.

The adoption of Pulumi AI since its release less than a year ago has been incredible, with tens of thousands of users submitting hundreds of thousands of prompts. And, perhaps surprisingly, usage hasn’t been driven only by or primarily by new Pulumi users. We’ve seen many of the most advanced Pulumi users with the deepest cloud knowledge leverage Pulumi AI the most heavily, helping them accelerate the development of new infrastructure solutions and discover more effective approaches to solving their infrastructure needs using the best of their cloud platforms.

On top of providing individual users a way to answer their unique questions, it is also helpful to get answers for the most common questions available on the internet, avoiding the need to generate new answers when solutions are needed. With Pulumi AI Answers, we’ve made answers to many of the most asked questions about cloud infrastructure available directly on the internet, making it even easier to find solutions to cloud infrastructure problems and IaC code to solve for them.

As Infrastructure as Code tooling and cloud development generally embraces traditional programming languages like in Pulumi or CDK, we see the breadth of amazing tools and tooling ecosystems available in the Node.js, Python, Go, .NET and Java ecosystems all come to bear to support scalable cloud infrastructure management. AI-based tooling is no exception, with tools like GitHub Copilot, Amazon CodeWhisperer and SourceGraph Cody changing how developers write, iterate on and test their code, including their infrastructure code in popular languages that have huge training data sets. These tools are simply incredible at understanding and working with these popular languages, and modern IaC users are benefiting from these tools for productivity and discoverability.

Of course, it’s not enough just to have the code you need; you actually have to deploy that code to create and manage your cloud infrastructure. This means getting the code into source control, setting up continuous delivery, and connecting to the configuration and secrets needed for the environment you are deploying to. Recently, we added “Deploy with Pulumi” support to each Pulumi AI answer, along with pulumi new support in the Pulumi CLI, enabling you can go from a few simple text prompts to a working solution fully integrated with source control, CI/CD and Pulumi ESC in less than 30 seconds.

Near Future

Even with the rapid progress over the last year, we’re clearly still just getting started in realizing the impact of AI on cloud infrastructure management.

Tightening the feedback loop for tools like Pulumi AI will continue to enable AI to iterate and solve more complex cloud infrastructure challenges, and arrive at correct working solutions. And instead of asking questions and getting back code, these tools will soon be able to actually do the deployments and verify that they work. The result is a natural language interface, which can build and evolve cloud infrastructure. Even as the interface becomes natural language, Infrastructure as Code is a uniquely good “code-behind” for this because it maintains a desired state configuration for the infrastructure, which AI tools can easily iterate on and manage. The incredible pace of improvement in AI model skill and performance for writing code in particular will accrue directly into improvements here.


An early prototype of interactive AI infrastructure development

AI-based tooling will also extend well beyond just creating and managing infrastructure, most importantly to the operational tooling used to understand, support, and diagnose that infrastructure, including policy enforcement across cost, compliance, and security. Giving AI-based tools and agents the ability to access things like cloud resource search, the ability to run operational CLI commands inside target environments, policy definition, and the opportunity to trigger deployment workflows will empower these tools to support and ultimately automated broad classes of cloud infrastructure management tasks.

Infrastructure as Code transforming AI

Today

One of the biggest imperatives in almost every development organization today is to identify and invest in ways to leverage AI to improve software-based products, tools and processes. This has led, in turn, to massive growth in the need for cloud infrastructure to support and enable building and delivering these applications.

One of the most striking things about modern AI-based applications is that they are significantly more cloud-native than previous generations of application architectures. This is true for several reasons:

Modern AI technology and services have been built around cloud-native assumptions
AI-based applications frequently have especially bursty compute needs
AI applications and the space overall is scaling rapidly, requiring elasticity and low operational overhead
New managed services, like vector datastores, managed model serving APIs, managed data and training pipelines

This cloud-native quality of AI-based applications makes Infrastructure as Code even more important and valuable for AI applications than in other, more traditional applications.

The services to support building AI applications have all been built over the last two years around the assumption of the cloud. GPUs are dominantly accessed through cloud services. LLMs are invoked via managed services from OpenAI or via managed hosting of open-source models via SageMaker, Huggingface, or others. Retrieval Augmented Generation (RAG) is powered by dozens of managed database offerings, from dedicated vector databases like Pinecone, to vector extensions in Amazon Aurora PostgreSQL and more. Developers coming to AI assume they will stitch these services into rich application offerings.

At Pulumi, we’ve seen dozens of AI-focused companies adopting and using Pulumi to manage their infrastructure, motivated by a combination of these AI-oriented infrastructure needs, along with the ease of using Python (already heavily used within the AI space), and TypeScript (popular due to a wide variety of AI-oriented front end frameworks). Pulumi also exposes the ability to manage many of these services directly, provisioning and managing the lifecycle of AI services across your infrastructure through support for things like Azure OpenAI Service, Pinecone Indexes, Amazon SageMaker Endpoints, and much more.

As these applications move from prototypes to production, the need for robust and scalable infrastructure to deploy these applications in the major cloud platforms is essential, and IaC offers the best way to manage this at scale. For example, deploying a LangChain application in production, or deploying the production-grade infrastructure for deploying AI applications leveraging Pinecone’s vector database at scale. Both of these highlight ready-to-deploy cloud architectures for running an AI application in a production environment.

The cloud-native foundations of the modern AI application wave also lead to a lot of adoption and usage of Kubernetes as an orchestration platform. Infrastructure as code with Pulumi helps teams manage the Kubernetes cluster, the workloads, and the external AI service dependencies all in a consistent way, using a unified programming model and tools, and ensuring consistent delivery of changes, the ability to easily spin up test and development environments, and consistent application of policy and insights.

One particular trend in the rapid growth of IaC is in fast-growing companies building the new managed AI infrastructure services themselves - from data services (e.g. Pinecone) to compute (e.g. Modal). These products face the need to manage complex cloud infrastructure as part of both multi-tenant and single-tenant solutions in order to offer simple abstracted data and compute services for their AI-application developer customers. Automation of infrastructure provisioning and changes via tools like Pulumi Automation API and Pulumi Deployments has seen rapid adoption and scaling among these customers. The ability to deploy across many cloud providers to offer low latency data and compute services to customers, data sovereignty, and access scarce GPU and accelerated computing resources across all cloud providers with a single consistent programming model and deployment strategy proves more important than ever.

Near Future

The growth in the development of AI-based applications is still in its early days, and all the trends we’re seeing will only accelerate over the coming year.

As these AI applications transition from prototypes to production deployment, their infrastructure needs are growing and maturing. Simple prototyping environments like Jupyter notebooks or Vercel deployments will transition into AWS/Azure/Google Cloud for the security, cost, compliance and management benefits. Larger development teams and faster customer-facing iteration will require standing up testing and ephemeral development environments, and some of the managed service dependencies will be brought in-house into self-managed open source model-service, training infrastructure and data pipelines.

As patterns continue to solidify around common architectures, higher-level infrastructure components will abstract away some of this production infrastructure complexity, initially within organizations, and then in public components shared in the Pulumi Registry and elsewhere.

And of course, the set of managed cloud services to support the AI ecosystem will continue to grow, expand and mature. Whether delivered as part of a major cloud platform or via a fast-growing startup, it will all be available to use productively at scale via infrastructure as code in Pulumi.

Conclusion

At Pulumi we often say that All Software is Cloud Software, and that All Developers are Cloud Developers. This has been truer than ever as AI has risen to the forefront of every organization’s software development priorities over the last 15 months. AI is transforming every aspect of software development, including cloud infrastructure development. Cloud infrastructure supports, enables, and accelerates how AI applications are built and scaled in production. We’ve already seen incredible impacts of both of these trends, yet there is so much more ahead on these arcs.

We’re hard at work on all of these at Pulumi, and excited to see all of the innovation across the industry to build the future of AI-enabled cloud infrastructure management. Check out Pulumi AI and the AI with Pulumi page to get started experiencing the future of AI for Infrastructure as Code today!

Deploy Cloud Infrastructure in 30 Seconds with Pulumi AI

Luke Hoban — Wed, 20 Dec 2023 15:00:00 -0800

Note: This post discusses Pulumi Copilot, which Pulumi Neo has replaced. Learn about Neo →

There are new intelligent cloud management capabilities available in Pulumi Copilot. Learn More

Earlier this year we launched Pulumi Neo, a purpose-built AI assistant that can create Infrastructure as Code (IaC) from natural language prompts using Pulumi. Since launch, we’ve seen incredible adoption of Pulumi AI, with over 200,000 questions asked so far and growing fast. Pulumi AI is popular with users new to Pulumi and/or new to the Cloud, but also heavily used by many of the most advanced IaC users and organizations to quickly discover solutions to new problems they need to solve. Over the last few months, we’ve driven major improvements to Pulumi AI through the recently launched Pulumi AI Answers pages with thousands of AI generated answers to common questions, improvements to code generation correctness and performance, and expansion of the languages supported by Pulumi AI.

Today, we are taking the next big step, introducing support for deploying cloud infrastructure directly from Pulumi AI. Going from idea to running cloud infrastructure is just a natural language prompt away!

This is accomplished through two new features:

The pulumi new command now offers the option to provide a Pulumi AI prompt instead of selecting a template. It then generates a template from your prompt, and lets you iterate on this before choosing to create a project and deploy it to the cloud.
Every Pulumi AI and Pulumi AI Answers page now has a Deploy with Pulumi button on it, which lets you click through a few prompts to deploy the given infrastructure into your cloud account using Pulumi Deployments and Pulumi ESC in Pulumi Cloud or the pulumi new command.

Check out Pulumi Neo or pulumi new in the CLI today.

Hope Pulumi AI Works with `pulumi new`

The pulumi new command is the starting point for almost every new Pulumi project today, offering users a choice from hundreds of templates across various cloud providers, languages and architectures.

Now, pulumi new users working with the Pulumi Cloud backend have the additional option to use Pulumi AI to generate a template on demand instead of selecting from one of the pre-defined templates.

The Pulumi AI response is streamed back directly into the CLI, and if Pulumi AI didn’t get it quite right, you can request to “revise” the question to ask Pulumi AI to adjust it’s answer with a follow-up prompt. When the results look good, you can hit “yes” to deploy the template, generating the source code in your chosen language and installing dependencies for the target cloud providers. With a simple pulumi up your new cloud infrastructure is deployed and running in the cloud.

There’s also a new –ai flag to pulumi new, so you can create a new project from an AI prompt with a command like one of the following:

pulumi new --ai "run NGINX in EC2 with a public port"
pulumi new --ai "a static website in Azure with a CDN in front"
pulumi new --ai "managed postgres database and Fargate compute for a containerized app"

Going from natural language idea to running cloud infrastructure in just seconds!

Deploying Cloud Infrastructure with Pulumi AI

Thousands of developers are already discovering answers to their questions via Pulumi AI, and we’ve been expanding on this with links to Pulumi AI from the Pulumi Registry API docs and with the archive of high quality questions and answers as part of Pulumi AI Answers. To date, users have used these answers to then copy/paste code snippets into their own Pulumi programs, presenting something of a small but meaningful barrier to entry to quickly experimenting with the resulting infrastructure.

Now, every Pulumi AI and Pulumi AI Answers page includes its own Deploy with Pulumi button. The Deploy with Pulumi button can be used along with any template or Pulumi code example to deploy that code using Pulumi Cloud via the New Project Wizard. We’ve extended that to support deploying projects defined by a Pulumi AI answer. This includes the code snippet, but also the project dependencies, a useful description, and language-specific project files.

Recently, we introduced many improvements to the New Project Wizard in Pulumi Cloud, including the ability to deploy a template with just a few clicks using Pulumi Deployments to do the deployments, the Pulumi GitHub app to template the code into a repo for you, and Pulumi ESC to provide access to short-lived credentials for your cloud environment. Once these are configured for users in an organization, any template can be deployed via the New Project Wizard and the Deploy with Pulumi button in just a few clicks.

These same features and New Project experience are now available from every Pulumi AI page. Let’s see what that looks like.

Step-by-Step Guide: Deploying with Pulumi AI

After clicking on the Deploy with Pulumi button on any Pulumi AI page, you can choose the ESC Environment to use for configuration and credentials for your environment (here I use the dev-sandbox environment which provides short-lived OIDC credentials to a development sandbox AWS account), a name for your project and stack, and the repository and folder to generate the code into.

When I click Create Project, this adds the code to the repository and kicks off a Pulumi Deployments run to deploy this new infrastructure to the cloud.

A few moments later, once that deployment completes, I see I have a Pulumi stack with the expected outputs defined by the Pulumi AI answer - the IP address of the running instance in the cloud.

And navigating there shows me my new EC2 instance serving NGINX on the Internet.

We can now continue to iterate on our infrastructure by making changes in the new Git repo folder.

With just a few clicks, we went from a natural language idea to running cloud infrastructure, checked into a Git repo, with Pulumi Deployments configured!

FAQs: Pulumi AI and Cloud Deployment

What is Pulumi AI?

Pulumi AI is an AI-powered assistant that generates Infrastructure as Code (IaC) from natural language prompts.

How does pulumi new –ai work?

The pulumi new –ai command allows you to generate and deploy cloud infrastructure using a natural language description instead of selecting a pre-defined template.

Which cloud providers does Pulumi AI support?

Pulumi AI supports AWS, Azure, Google Cloud, and other major platforms.

Can I modify the AI-generated infrastructure before deployment?

Yes! You can revise the AI-generated code before deployment to match your needs.

Where can I provide feedback on Pulumi AI?

Join the Pulumi Community Slack or join the GitHub Discussions.

Looking Ahead: The Future of Pulumi AI

Pulumi AI has already transformed how cloud engineers discover and solve their cloud infrastructure development problems. And yet, we still see incredible opportunity ahead of us. We’ve made rapid strides on improving the accuracy of AI answers, and continue to invest in further improvements. We’ve made it significantly faster to go from idea to deployed infrastructure with the pulumi new and Deploy with Pulumi button integrations, but we also see the opportunity to make these experiences even more interactive going forward. And of course, we continue to integrate Pulumi AI even more broadly throughout our product experiences, lighting up AI assistance across the cloud engineering journey.

Try it out yourself today at pulumi.com/neo!

Pulumi for Platform Teams: New Features for Developer Portals, Policy and Deployments

Luke Hoban — Wed, 11 Oct 2023 04:00:00 -0700

Over the last two years, we’ve seen a huge surge in adoption of Pulumi by Platform Teams – centralized teams within a business responsible for building out core cloud infrastructure and providing tools to the rest of the organization to maximize the productivity, cost efficacy, compliance and velocity of application and service delivery throughout the organization. These teams use Pulumi to manage their own cloud infrastructure complexity, to offer best practices components to their organizations, to enforce organizational policy, and to drive infrastructure delivery automation.

Today, we’re excited to announce a broad range of new features that extend Pulumi Cloud’s support for Platform Teams, across the entire lifecycle of cloud delivery.

Developer Portals: Every platform team we work with is looking to expose reusable cloud infrastructure components and templates via a developer portal of some kind. Many are building their own custom developer portals on top of Pulumi Automation API and Pulumi Deployments, but we also wanted to offer options for teams that cannot (or do not want to) build a custom solution. Today, we are releasing three new options for Platform Teams:

🆕 Organization Templates and New Project Wizard: Pulumi Cloud now supports registering your own custom templates for your organization, and enabling developers in your organization to provision them entirely via point-and-click in Pulumi Cloud with the new New Project Wizard. This offers a great out-of-the box developer portal experience for any team using Pulumi Cloud.
🆕 Pulumi Backstage Plugin: For teams using Backstage, they can now integrate Pulumi directly within their Backstage instance, via support for a new Pulumi plugin as well as new Pulumi scaffolder actions.
🆕 pulumi new support for private templates: By default, pulumi new provides a number of templates provided by Pulumi, but it can also use your own private templates.

Policy and Compliance: Platform teams need robust and flexible solutions for establishing policy and compliance guardrails for their cloud infrastructure. Pulumi’s existing CrossGuard policy-as-code framework provides the richest option for defining and enforcing policy across all infrastructure in an organization, using the full capabilities of your favorite languages. Today, we are extending that support with two important new features for platform teams:

🆕 Compliance Ready Policies: New pre-built policy packs for for enforcing common security and compliance policies (PCI DSS, ISO 27001 and CIS) across a broad range of cloud providers (AWS, Azure, Google Cloud and Kubernetes), and dozens of services within each platform.
🆕 Remediation Policies: Pulumi Policy as Code now supports authoring policies which don’t just warn or error, but actually change the infrastructure directly to remediate compliance gaps. Remediation policies can be configured and applied across any subset of stacks within an organization.

Deployments and Delivery: Managing the delivery of cloud infrastructure changes across a wide variety of environments is an essential component of a cloud platform, and we released Pulumi Deployments in preview last year to provide a rich solution for infrastructure delivery directly within Pulumi Cloud. Today, we are making that support Generally Available (GA) along with a few key enhancements:

🆕 Deployments GA: Pulumi Deployments is now generally available, with pricing and support available across all Pulumi Cloud editions.
🆕 Deployments Support for GitHub Enterprise: Pulumi Deployments now supports integration with GitHub Enterprise as well as GitHub.com.
🆕 Org-Wide Deployments Overview Page: A new page where you can see all deployments across the organization, giving you a single pane of glass to manage company wide deployments.
🔜 Deployments Customer Managed Agents: Customers will soon be able to run deployments inside agents they run within their own infrastructure, for cases where this is required for unique compliance or security reasons.

Together with Pulumi ESC (Environments, Secrets and Configuration) which we announced yesterday, and the core Pulumi IaC infrastructure as code platform, these new features provide a rich collection of building blocks and out of the box experiences for empowering platform teams via Pulumi Cloud. At every layer of the Platform Engineering stack, Pulumi and Pulumi Cloud offer key technologies to accelerate your Platform Engineering projects.

Organization Templates and New Project Wizard

We’ve introduced a new wizard experience to create Pulumi projects into Pulumi Cloud, making it easier than ever to get started deploying new infrastructure with Pulumi.

The new project wizard allows anyone in your organization to pick a template they want to install, and walk through configuring the deployment of that template. We’ve added support for configuring Pulumi Deployments automatically, so that the template can be deployed without needing the Pulumi CLI locally, or any other CI/CD configuration. Just a few clicks and you have deployed your infrastructure.

We’ve also added integration with the newly released Pulumi ESC so that you can set up environments (and the configuration and credentials needed to deploy to them) that users within your organization can deploy into, determine who can use which environments using RBAC, and then allow those users to deploy templates directly into the environments they can access – no need to copy secrets, credentials or configuration around!

We’ve also added the ability for organizations to provide their own custom set of Organization Templates — templates specific to their organization, which can live in either public or private Git repositories. The Organization Template Source provided will populate the templates that are displayed to members in the New Project wizard experience.

Together, Organization Templates and the new New Project Wizard offer a foundational developer portal experience directly within the Pulumi Cloud console. Users can come to Pulumi Cloud, select one of the pre-defined templates set up by their Platform Team, and then in a few clicks have it deployed into a predefined cloud environment.

The New Project wizard is available now for all Pulumi users, and Organization Templates are available to Pulumi Enterprise and Business Critical users. Check out the documentation to get started.

Backstage Plugin

We’ve seen many developer portal technologies rapidly growing in popularity over the last few years. In particular, we’ve seen many Pulumi users adopting Backstage. Last week we launched a new Pulumi Backstage Plugin to address the needs of organizations using Backstage and Pulumi together.

The new Pulumi tab gives you direct access to all Pulumi stack activity associated with your backstage projects that include Pulumi stacks.

The Pulumi Backstage Plugin also supports two new scaffolding actions, pulumi:new and pulumi:up, which can be used to template out new Pulumi projects and to trigger updates to Pulumi stacks, fully integrated into the Backstage scaffolding system.

apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
 name: kubernetes-template
 title: Kubernetes Cluster
 description: |
 A template for creating a new Kubernetes Cluster.
 tags:
 - pulumi
 - kubernetes
spec:
 steps:
 - id: pulumi-new-component
 name: Cookie cut the component Pulumi project
 action: pulumi:new
 input:
 name: "${{ parameters.component_id }}-infrastructure"
 description: ${{ parameters.description | dump }}
 organization: ediri
 stack: ${{ parameters.stack }}
 template: "https://github.com/my-silly-organisation/microservice-civo/tree/main/infrastructure-${{ parameters.cloud }}-${{ parameters.language }}"
 config:
 "node:node_count": "${{ parameters.nodeCount }}"
 folder: .

Check out the Pulumi Backstage Plugin in the Backstage Plugin directory or the Roadie Pulumi Backstage Plugin guide to get started today.

Compliance Ready Policies

We first released our CrossGuard policy-as-code framework nearly 4 years ago, and have seen many large organizations leverage its rich expressiveness, flexibility and early prevention features to enforce key compliance criteria for their organization. Support in Pulumi Cloud for automatically enforcing these policies across all Pulumi updates within an organization allows platform teams to put compliance guardrails in place for users within their organization.

However, traditionally each organization has needed to build their own policies, resulting in some activation energy needed to start benefiting from CrossGuard. As we’ve worked with many customers to build out their own internal compliance policies using CrossGuard, we’ve identified some common compliance standards and policies that we can offer directly as part of CrossGuard.

Today, we are making a large collection of new policies available for CrossGuard users. These policies support many common compliance standards (PCI DSS, ISO 27001 and CIS) across a broad range of cloud providers (AWS, Azure, Google Cloud and Kubernetes), and dozens of services within each platform.

These policies are offered as a library, from which Platform Teams can enable the subset they want to apply. For example:

import { PolicyPack } from "@pulumi/policy";
import { policyManager } from "@pulumi/compliance-policy-manager";

new PolicyPack("aws-iso27001-compliance-ready-policies-typescript", {
 policies:[
 ...policyManager.selectPolicies({
 vendors: ["aws"],
 services: ["ec2", "eks"],
 severities: ["critical", "high"],
 frameworks: ["iso27001"]
 }, "advisory"),
 ],
});

The new Policy Pack wizard also makes it easier than ever to create a new policy pack from the compliance-ready policies, and enable it within your organization.

Check out the documentation and open source repository to get started with Compliance-Ready Policies today.

Remediation Policies

Existing CrossGuard policies have always had the ability to be set to Advisory to emit a warning when they fail, or to Mandatory to prevent the Pulumi deployment from completing when they fail. This enables strict prevention of non-compliant infrastructure deployments before they even get into production.

However, in many cases, it’s clear what change needs to be made to the infrastructure to make it compliant. Today, we’ve launched new support in CrossGuard for Remediation Policies, a new kind of policy which actually fixes up the infrastructure to be compliant instead of just warning or erroring!

This is a magical experience for both policy authors and infrastructure engineers.

const s3NoPublicRead: ResourceValidationPolicy = {
 name: "s3-no-public-read",
 description: "Prohibits setting the publicRead or publicReadWrite permission on AWS S3 buckets.",
 enforcementLevel: "remediate",
 remediateResource: remediateResourceOfType(aws.s3.Bucket, (bucket, args) => {
 if (bucket.acl === "public-read" || bucket.acl === "public-read-write") {
 // Modify the ACL and return the new bucket state to use instead.
 bucket.acl = "private";
 return bucket;
 }
 }),
};

def s3_no_public_read_remediator(args: ResourceValidationArgs):
 if args.resource_type == "aws:s3/bucket:Bucket" and "acl" in args.props:
 acl = args.props["acl"]
 if acl == "public-read" or acl == "public-read-write":
 # Modify the ACL and return the new bucket state to use instead.
 args.props["acl"] = "private
 return args.props

s3_no_public_read = ResourceValidationPolicy(
 name="s3-no-public-read",
 description="Prohibits setting the publicRead or publicReadWrite permission on AWS S3 buckets.",
 enforcement_level=EnforcementLevel.REMEDIATE,
 remediate=s3_no_public_read_remediator,
)

Check out the documentation to get started adding remediation policies to your policy-as-code configuration today.

Deployments GA

In the last year since we launched Pulumi Deployments in preview, we’ve seen incredible growth in usage and scale of Pulumi Deployments across the Pulumi user base. Customers use Pulumi Deployments for a single built-for-purpose CI/CD solution for their Pulumi infrastructure delivery, but also leverage rich API-driven deployments, Review Stacks, Click-to-Deploy within the Pulumi Cloud Console, and much more.

As part of General Availability, we are announcing pricing for deployments at $0.01 per deployment minute. The new pricing will be effective November 1st. Individual Edition users get 500 minutes free per month, and Team Edition users get 3,000 minutes free per month. Full details of the pricing can be found on the Pricing Page.

In addition to pricing, we’re making available several new features in the Deployments platform.

The new Deployments pane in the Pulumi Cloud console provides a unified view across all deployments in your organization. This includes in progress and queued deployments so that you can easily access any currently in flight deployments, as well as a full history of all deployments performed across all stacks you have access to within your organization.

And finally, we’re extending Pulumi Deployments to support additional key enterprise needs. We recently added support for connecting Pulumi Deployments to GitHub Enterprise, and we will be delivering support for self-hosted Pulumi Deployments agents which can be run within your own infrastructure later this month. You can fill out this form to register for the private preview of self-hosted agents today.

Conclusion

Pulumi Cloud offers a rich and broad platform to support any platform team today. Across the entire lifecycle of cloud infrastructure and application delivery, Pulumi Cloud supports platform teams in building cloud platforms for their organizations which empower developers via rich IaC components and developer portals, enable seamless delivery through Deployments and Automation API, provide robust compliance guardrails through Compliance-Ready Policies and Remediation Policies, and securely manage secrets and configuration across a variety of environments with Pulumi ESC.

It’s never been easier to build your internal developer platform with the help of Pulumi Cloud.

Introducing Pulumi ESC: Easy and Secure Environments, Secrets and Configuration

Luke Hoban — Tue, 10 Oct 2023 04:00:00 -0700

Today, we’re excited to introduce Pulumi ESC, a new product from Pulumi that manages and tames secrets and configuration complexity across all of your cloud infrastructure and application environments. Pulumi ESC introduces a new category of configuration-as-code product, motivated by our experience working with hundreds of Pulumi IaC customers to address their needs in managing secrets and configuration at scale within their Pulumi infrastructure and across other cloud applications and infrastructure projects.

Pulumi ESC enables teams to aggregate secrets and configuration from many sources, manage hierarchical collections of configuration and secrets (“environments”), and consume those configuration and secrets from a variety of different infrastructure and application services. Pulumi ESC works hand-in-hand with Pulumi IaC to simplify configuration management, but also works independently from Pulumi IaC, as a solution for managing environments, secrets and configuration for any application or infrastructure project.

For example, the Pulumi CLI makes it possible to give your developers immediate, just-in-time authenticated and short-lived access to cloud credentials across any cloud provider with just a single command: pulumi env run aws-staging -- aws s3 ls.

Here are just a few of the exciting use cases we’ve seen for Pulumi ESC:

Do you have configuration for dozens of environments that is copy/pasted around and prone to drift? Pulumi ESC lets you manage this as code using a simple set of environments that naturally and flexibly compose to derive all of your different configurations.
Is it hard for developers in your organization to get access to short-lived credentials to work in the environments they need to develop and deploy into? Pulumi ESC lets you configure OIDC to pull short-lived credentials and other secrets dynamically from all of your cloud and secrets providers and present them in a logical way to other tools with a single command like pulumi env run aws-staging -- aws s3 ls.
Have your Pulumi IaC configuration files grown to hundreds of lines of configuration spread across many different stack files? Pulumi ESC introduces a simple way to consume Pulumi IaC configuration directly from Pulumi ESC environments, enabling a much richer secrets and configuration management for Pulumi IaC.

Pulumi ESC is available today in preview via the Pulumi CLI’s pulumi env commands, as part of Pulumi Cloud, via the Pulumi Cloud REST API, and has direct integration with Pulumi IaC stack configuration. It also supports dynamically pulling secrets and configuration from other sources of truth, including AWS OIDC, AWS Secrets Manager, Azure OIDC, Azure KeyVault, Google Cloud OIDC, Google Secrets Manager, HashiCorp Vault, and Pulumi IaC Stack References - with many more sources like 1Password coming soon.

Pulumi ESC: How It Works & Key Features

Pulumi ESC allows you to define environments, each of which specifies a collection of secrets and configuration values. Environments are specified as code, and each environment can import one or more other environments (for composition and inheritance), and can specify its own configuration values directly, or be pulled from another secrets provider.

The following example shows a Pulumi ESC environment which dynamically pulls values from AWS OIDC, HashiCorp Vault and AWS Secrets Manager, as well as setting environment-specific configuration and overriding inherited configuration.

# imports allow you to compose other pre-existing environments
imports:
 # AWS creds via OIDC
 - aws-production
 # stripe API keys imported from vault
 - stripe-production
 # keys for signing docker images via AWS Secrets Manager
 - docker-signing-production
values:
 # environment-specific configuration
 desiredInstanceCount: 8
 # overriding imports
 aws:region: us-west-2

Why Use Pulumi ESC for Cloud Secrets & Configuration?

Working with Pulumi customers, we’ve seen the incredible benefits they get from managing IaC complexity in their code using Pulumi, and yet they frequently end up with a lot of complexity remaining at the boundary to their IaC code - at the configuration and secrets they need to get into Pulumi IaC. Users frequently show us configuration files that are hundreds of lines long, with tons of copy/paste both between Pulumi configuration and between other sources of truth and Pulumi configuration, and with more long lived static secrets being copied around than they would like - creating maintenance burden, security concerns, frequent drift, and operational risk.

As we looked at solving these problems for Pulumi we quickly realized that none of these problems are actually specific to Pulumi - they are all things that customers talked about seeing consistently throughout their infrastructure and application projects. With other IaC tools, with Kubernetes configuration, with 12-factor app environment configuration, and everywhere in between. Everyone we talked to had plenty of scar tissue built up from struggling with a common set of challenges in managing configuration and secrets across a complex set of interrelated environments:

Secrets and Configuration Sprawl: Data in many systems. Challenging to audit. Lots of application-specific logic to acquire and compose configuration from multiple sources. Divergent solutions for Infrastructure and Application configuration.
Duplication and Copy/Paste: Secrets are duplicated in many places. Frequently coupled to application/system-specific configuration stores.
Too Many Long-lived Static Secrets: Long lived static credentials are over-used, exposing companies to significant security risk. Rotation is operationally challenging. Not all systems support direct integration with OIDC and other dynamic secrets provisioning systems.

While customers frequently had good solutions for pure secrets management (AWS Secrets Manager, HashiCorp Vault, etc.), none of these solutions extended into managing the complexities of actually getting these secrets as well as non-secret configuration into the collection of logical environments that define how they will be consumed in infrastructure and applications. Customers struggled with too many walled gardens for secrets management and configuration, and lack of any consistent set of tools and workflows to help manage the complexity of how secrets and configuration flow between these systems.

Pulumi ESC was born to address these problems and needs head on. It does so through a few core design principles:

Hierarchical and Composable: Environments contain collections of secrets and configuration, but can also import one or more other environments. Values can be overridden, interpolated from other values, and arbitrarily nested. This allows for flexible composition and reuse, and avoids copy paste.
Any Secrets Provider: Support for dynamic configuration providers allow Pulumi ESC to integrate with secrets stored in any other provider. Organizations often use AWS Secrets Manager, Vault, Azure OIDC and/or 1Password plus many more sources of truth for their secrets and configuration. Pulumi ESC supports them all, providing a single interface to your configuration and secrets, no matter where their source of truth is. Pulumi ESC works with these tools to provide improved management of secrets and configuration.
Consume from Anywhere: The pulumi CLI and the Pulumi ESC Rest API enables environments to be accessed from any application, infrastructure provider, or automation system. At launch, first-class integrations are available with Pulumi IaC, local environment and .env files, GitHub Actions, and more.
Auditable: Environments must be “opened” to compute and see the set of value they provide, and this action is recorded in audit logs, including a full record of how each value was sourced from within the hierarchy of environments that contributed to it.
Authentication and RBAC: Pulumi ESC brokers access to secrets and configuration that live in other systems, and so authentication and granular RBAC are critical to ensure robust access controls across your organization. Pulumi ESC leverages the same Pulumi Cloud identity, RBAC, Teams, SAML/SCIM and scoped access tokens that are used for Pulumi IaC today, extending these all to managing access to Environments as well as Stacks.
Configuration as Code: Environments are defined as YAML documents which can describe how to project and compose secrets and configuration, integrate dynamic configuration providers, and compute new configuration from other values (construing a URL from a DNS name, or concatenating multiple configuration values into a derived value). The incredible flexibility of a code-based approach over traditional point-and-click interfaces allows Pulumi ESC to offer rich expressiveness for managing complex configuration.
Fully Managed: Pulumi ESC is offered both as a fully managed and a self-hosted service in Pulumi Cloud.

Pulumi ESC enables a new kind of solution for configuration management of modern cloud infrastructure.

Pulumi ESC Overview

We can start by creating a few simple environments using Pulumi ESC. This can be done in the Pulumi Cloud console or using the pulumi env sub-commands in the Pulumi CLI. Let’s take a look at these in action.

You might also like:

Creating and Managing Environments, Secrets and Configuration

First, in the Pulumi Cloud console, we can go to the Environments tab and create a new environment called aws-staging.

We can then define some configuration for our environment. For example, defining our AWS region:

# aws-staging
values:
 aws:
 region: us-west-2

This defines a configuration object with a top level aws key, and a nested region key whose value is us-west-2. When we save this, the preview on the right shows the definition of a configuration object that produces the following JSON:

{
 "aws": {
 "region": "us-west-2"
 }
}

We can make this more interesting by also integrating with AWS OIDC to define the AWS credentials we want to access. To do this, we change our environment to add a creds key to the aws object:

# aws-staging
values:
 aws:
 region: us-west-2
 creds:
 fn::open::aws-login:
 oidc:
 roleArn: arn:aws:iam::123456789012:role/pulumi-environments-oidc
 sessionName: pulumi-environments-session
 duration: 1h

This indicates that we want to pull a set of AWS credentials from the aws-login dynamic provider, which expects a few inputs - including a roleArn we would like to assume and a duration for the short lived AWS credentials that will be returned. If we have configured OIDC in AWS to allow this Pulumi Cloud organization and environment to request short-lived credentials, then this will allow our environment to dynamically pull in those credentials for an authenticated user, and present back a config object that includes them.

To see the values computed by an environment, we need to “open” it. That can be done via the CLI, the REST API or the integrations into other systems (like Pulumi IaC). Let’s open it with the pulumi CLI first. Install the Pulumi CLI, and then run the following commands:

$ pulumi env ls
demo/aws-staging

$ pulumi env open demo/aws-staging
{
 "aws": {
 "creds": {
 "accessKeyId": "ASIAVNHUKX5VGSMMR2DJ",
 "secretAccessKey": "REDACTED",
 "sessionToken": "REDACTED"
 },
 "region": "us-west-2"
 }
}

The aws-login provider returns three values, one non-secret and two secret, which provide the short-lived credentials necessary to authenticate with AWS.

We can also use this configuration as both environment variables and Pulumi IaC config. To do so, we extend our environment to the following:

# aws-staging
values:
 aws:
 region: us-west-2
 creds:
 fn::open::aws-login:
 oidc:
 roleArn: arn:aws:iam::123456789012:role/pulumi-environments-oidc
 sessionName: pulumi-environments-session
 duration: 1h
 environmentVariables:
 AWS_ACCESS_KEY_ID: ${aws.creds.accessKeyId}
 AWS_SECRET_ACCESS_KEY: ${aws.creds.secretAccessKey}
 AWS_SESSION_TOKEN: ${aws.creds.sessionToken}
 pulumiConfig:
 aws:region: ${aws.region}

The top-level environmentVariables key is used by convention to project out a set of environment variables for this environment. The top level pulumiConfig key is used by convention to project out a set of Pulumi IaC configuration for this environment. Note that they can pull their values from other configuration in this environment (or, as we’ll see soon, any environment they import). That includes using the values from dynamic providers like aws-login.

Now that we’ve got a complete environment configured for setting up access to our AWS account, we can use this with pulumi env run to run a command (aws s3 ls) using the environment variables:

$ pulumi env run demo/aws-staging -- aws s3 ls
2023-09-07 16:35:11 accesslogs-1284c27
2023-04-13 15:04:41 accesslogs-6652a3d
2023-01-20 16:52:04 accesslogs-7ea7130
2023-08-23 17:50:24 accesslogs-8ad28c2
...

Note that when we run this command, pulumi env run uses our local authentication credentials (including RBAC permissions) for Pulumi ESC to authenticate our access to this environment, and then uses the environment’s identity to get a short-lived AWS credential via AWS OIDC. It then brings back this unique credential just for the lifetime of the aws s3 ls subcommand. This provides very granular authentication and auditability of the operations run, while also not requiring the user to have a long lived AWS credential on their machine to run operational commands against the account.

Using Pulumi ESC with Pulumi IaC

Next, we can define a second environment for our shopping-service application. This will be deployed with Pulumi IaC, so we’ll provide some default configuration for this service in the pulumiConfig section.

We run pulumi env init shopping-service to create our new environment. And then pulumi env edit shopping-service to open an editor to populate our new environment. We can then add the following to our environment, where it will be encrypted and stored for us.

# shopping-service
values:
 pulumiConfig:
 numInstances: 4
 domainName: shopping.acmecorp.com
 dbPassword:
 fn::secret: hunter2

To use this from our Pulumi IaC program, we want to consume the information in both the shopping-service and aws-staging environments. We can define another environment which just imports these two! Let’s call this third environment shopping-service-staging and define it:

imports:
 - aws-staging
 - shopping-service
values:
 pulumiConfig:
 numInstances: 2

This uses the imports top level key which allows us to compose together configuration from multiple other environments. We don’t have to copy/paste configuration and secrets between stacks, or from other sources of truth – we just import them! This keeps our configuration as DRY as possible.

Note that we also specified a new value for numInstances which overrides what was imported. This allows us to customize the values if this particular environment needs to provide a different value than what it imports.

Now, we can use this directly within our Pulumi IaC project by modifying its stack file. Inside our Pulumi.staging.yaml file for the shopping-service project, we can remove the config section, and just replace it with:

# Pulumi.staging.yaml
environment:
 - shopping-service-staging

If I run pulumi preview now on my Pulumi IaC project (in this case, a project which just outputs the configuration values), we will see that configuration values for aws:region, numInstances, and dbPassword are all available. I didn’t have to copy them into this stack, they were automatically pulled out of the environment in Pulumi ESC. And note that the secretness of the dbPassword configuration value flowed through into Pulumi IaC’s config system to ensure this value is encrypted in any place it is used within our infrastructure as code.

Previewing update (pulumi/staging)
View in Browser (Ctrl+O): https://app.pulumi.com/pulumi/shopping-service/staging/previews/f0f9a124-88e1-4d4a-8025-178aac9127c3
Type Name Plan Info
+ pulumi:pulumi:Stack shopping-service-staging create 3 warnings
Outputs:
dbPassword : [secret]
numInstances: 2
region : "us-west-2"
Resources:
+ 1 to create

Finally, we can give other users in our organization access to this environment by configuring our Engineering team within Pulumi Cloud to be allowed to “Open” the environment.

Pulumi ESC enables us to define environments containing secrets and configuration using a flexible, code-based approach, leveraging composition in order to keep our configuration DRY, as well as dynamic providers to pull secrets and credentials from other providers like AWS OIDC. While this example used AWS, everything discussed works with any cloud, and with an even wider variety of secrets providers.

The Future of Pulumi ESC

We’re excited to launch the public preview of Pulumi ESC today, and see it solving many pressing problems with the feature set available today. We see even more opportunity to enhance this foundation, and have designed Pulumi ESC with a few new capabilities in mind. These features will ship in the coming months.

Application-level SDKs

Pulumi ESC can already be accessed from application code via the REST API, but we will be extending this to native SDKs for TypeScript, Python, Go and .NET. These SDKs will enable configuration from ESC to be directly pulled into applications at runtime, instead of, or as well as, provisioning via environment variables or other mechanisms at deployment time. See pulumi/esc#60.

Syncing Configuration to External Systems

While the pulumi CLI and REST APIs can be used today to pull configuration down into many existing places where environments will be consumed, there are also many systems that store their configuration in more “walled gardens” which would need to be pushed into for optimal usability. For example, pushing configuration values into a CI/CD system’s configuration system to avoid needing to copy/paste configuration and secrets manually into their UI. We are working on additional integrations for Pulumi ESC that can “push”/“sync” configuration into systems directly, so that you can still benefit from the usability of their native configuration systems, without having to take on the problems of configuration sprawl, long-lived static secrets and duplication and copy/paste of secrets. See pulumi/esc#58.

Dynamic Configuration Provider Ecosystem

We are already working with many additional partners to support pulling configuration and secrets from additional sources of truth into Pulumi ESC - including 1Password, Cloudflare and more. Our goal is to support rich composition of configuration and secrets no matter where they live, and in the most secure way possible, and working with these providers will enable ESC to integrate natively with their systems. As part of this, we will also be working to bring the provider plugin model into the open source pulumi/esc repository along with the existing dynamic configuration providers. See pulumi/esc#64 and pulumi/esc#65.

Versioning Environments

At launch, environments are managed directly in Pulumi Cloud. But we’ve already heard early preview users express a desire to also be able to manage these via source-controlled code. We’re exploring adding environments to the Pulumi Cloud Pulumi Provider to enable using desired state IaC to manage the definition of environments. We’ve also worked with customers looking to be able to pin their projects that consume environments to specific versions of the environment, to avoid risk of breaking when environments are updated, and are working on extensions to enable this as well. See pulumi/esc#59 and pulumi/esc#63.

Easy-to-Use Key/Value Interface in Pulumi Cloud

For the preview, we focused on document-based editing experience for environments, which provides direct access to the full richness of Pulumi ESC. For many common use cases, it’s helpful to have a simple key/value-based interface for adding and configuring secrets. We will add the ability to switch between both of these UI experiences in the Pulumi Cloud console soon. See pulumi/esc#62. Below is an early sketch of this additional table-based user interface.

Conclusion

Pulumi ESC is a new kind of configuration management offering that addresses pervasive issues faced by teams performing application and infrastructure development. We’re incredibly excited to launch the public preview, and to work with our community to continue our journey together. While in preview, Pulumi ESC is available at no additional cost to all Pulumi Cloud users.

You can learn more about Pulumi ESC at:

Pulumi Insights and AI in the Pulumi CLI

Luke Hoban — Wed, 13 Sep 2023 10:00:00 -0800

Earlier this year we introduced Pulumi Insights, a collection of features that bring intelligence to cloud infrastructure using Pulumi. Two key components of that launch were Pulumi AI, a generative AI assistant purpose-built to create cloud infrastructure using natural language, and Pulumi Resource Search, multi-cloud search and analytics across every cloud resource and environment in your organization.

Today, we are excited to bring Pulumi Insights into the pulumi CLI with the new pulumi org search and pulumi ai commands. These commands put AI and resource search at your finger tips right where Pulumi users spend most of their time, in the terminal iterating on their cloud infrastructure.

This release is the latest of a long list of improvements and enhancement to Pulumi Insights over the last few months, including AI Assist for resource search, fine-grained resource property search, and registry support in Pulumi AI. We’ve seen incredible adoption and usage of all of these features over the last few months, and they are now easier than ever to access via the pulumi CLI.

Pulumi Resource Search in the CLI

With the new pulumi org search command, you can search across all of the infrastructure managed by your organization (or individual account) using the same rich search syntax supported by Pulumi Resource Search in the Pulumi Console.

pulumi org search -q "type:aws:kinesis/stream:Stream modified:>=09/01/2023" --org my-org

If you don’t know the exact search syntax for what you want to ask, you can use AI Assist, which was recently opened up to all Pulumi organizations. Just use pulumi org search ai instead!

pulumi org search ai -q "all Kinesis streams modified since September 1st 2023" --org my-org

Notice that AI Assist comes up with the same query and runs it to get the same results we saw above.

These queries can also include complex logic which relies on looking at properties of resources, such as their tags. For example:

pulumi org search ai -q "All untagged lambda functions with memory > 2 GB" --org my-org

AI Assist determines it should run the query type:"aws:lambda:Function" .memorySize:>2048 -.tags: and shows the results of that query:

The combination of rich query syntax, AI Assist and property search makes it easy and exciting to interactively find the information you are looking for about resources in your stack.

And of course, you can jump straight into the console to drill down even further into your data.

Pulumi AI in the CLI

We’ve also taken our first step towards integrating Pulumi AI into the CLI. With the pulumi ai web command, you can type a query in the CLI and have it launch a question into a browser window.

pulumi ai web "deploy metabase on AWS with a managed postgres database"

This opens a browser to a guided Pulumi AI conversation that walks through all the steps required to accomplish this task.

Conclusion

With these new CLI commands, Pulumi Insights is available at your fingertips in the pulumi CLI. These are just the first steps to bring search and AI into the CLI, expect to see much more in the coming months. To learn more about the various Pulumi Insights features discussed here, check out the resources below:

Pulumi Kubernetes 4.0: Even More Kubernetes-Native

Levi Blackstone — Wed, 19 Jul 2023 16:00:00 -0400

Since the very earliest days of the Pulumi project, Kubernetes has been a core part of the Pulumi platform. The initial Pulumi Kubernetes provider supported the entire API surface area of the Kubernetes platform, derived directly and automatically from the Kubernetes OpenAPI specifications, and available to all of Pulumi’s familiar programming languages. Since then, we have offered day one support for every new Kubernetes version, added support for Helm, YAML, Kustomize and CRDs, added tools for converting to Pulumi (kube2pulumi and crd2pulumi) and delivered the Pulumi Kubernetes Operator. During that same time, Kubernetes usage has continued to expand within the ecosystem and among Pulumi users, with the Kubernetes provider growing from the fourth most used to the second most used provider on the platform.

We are excited to release the next major version of our Kubernetes provider - Pulumi Kubernetes 4.0.

Join us for a free, on-demand webinar with Pulumi and the CNCF to learn how to use Pulumi’s Kubernetes Provider and other open-source tools to build internal developer platforms. Register →

Pulumi Kubernetes 4.0 uses Kubernetes Server-Side Apply by default, supports upserting on all resources, and enables new Patch resources for every resource type in the provider. It also brings improved diffs, removes dependence on kubectl annotations, and provides simpler access to outputs in the Python, Go and Java SDKs, supporting any self-hosted or managed Kubernetes cluster with version 1.13 or newer.

Together, these additions make Pulumi’s support for Kubernetes even more Kubernetes-native, allowing the best of both worlds between Kubernetes’ flexible resource model and Pulumi’s robust desired state infrastructure as code model, all while using your favorite languages (TypeScript, Python, Go, .NET and Java) to define and manage your Kubernetes Infrastructure. Pulumi programs that need to manage Kubernetes resources that are also modified by other systems (external controllers, managed Kubernetes service control planes, kubectl or other IaC solutions) can faithfully represent the modifications they want to make to existing resources within the Pulumi resource lifecycle. Pulumi accurately detects any differences between the current state of resources in the cluster, and the desired state specified by Pulumi IaC.

Server-Side Apply, Upserts and Patch Resources

Kubernetes Server-side Apply (SSA) mode is now default in the v4 provider. SSA mode enables several useful new features: Kubernetes resources can be safely managed by more than one controller. Any conflicting fields will produce a detailed error message which avoids accidentally overriding changes made by other controllers. Resources can be “upserted”; in other words, create the resource if it does not exist, or apply the specified configuration to an existing resource. Patch resources add a new way to manage shared Kubernetes resources. Patch resources can be used to make targeted changes, and do not require the entire resource to be specified. This is a convenient way to update shared resources; a common example is resources that are provisioned by the cloud provider rather than as part of the Pulumi stack. Deleting a Patch resource undoes the patched change, but does not remove the shared Kubernetes resource. See the SSA how-to guide for more details and usage examples for SSA mode.

Note: Users may opt into the previous default Client-side Apply (CSA) mode by setting the enableServerSideApply provider configuration to “false”.

In this example, we define a ConfigMap as an upsert which will create the resource if it doesn’t exist.

import * as pulumi from "@pulumi/pulumi";
import * as kubernetes from "@pulumi/kubernetes";

const example = new kubernetes.core.v1.ConfigMap("example", {
 metadata: {
 annotations: {
 "pulumi.com/patchForce": "true",
 },
 name: "example",
 },
 data: {
 foo: "bar",
 },
});

import pulumi
import pulumi_kubernetes as kubernetes

example = kubernetes.core.v1.ConfigMap(
 "example",
 metadata=kubernetes.meta.v1.ObjectMetaArgs(
 annotations={
 "pulumi.com/patchForce": "true",
 },
 name="example",
 ),
 data={
 "foo": "bar",
 })

package main

import (
 corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/core/v1"
 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/meta/v1"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 _, err = corev1.NewConfigMap(ctx, "example", &corev1.ConfigMapArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Annotations: pulumi.StringMap{
 "pulumi.com/patchForce": pulumi.String("true"),
 },
 Name: pulumi.String("example"),
 },
 Data: pulumi.StringMap{
 "foo": pulumi.String("bar"),
 },
 })
 if err != nil {
 return err
 }
 return nil
 })
}

using System.Collections.Generic;
using Pulumi;
using Kubernetes = Pulumi.Kubernetes;

return await Deployment.RunAsync(() =>
{
 var example = new Kubernetes.Core.V1.ConfigMap("example", new()
 {
 Metadata = new Kubernetes.Types.Inputs.Meta.V1.ObjectMetaArgs
 {
 Annotations =
 {
 { "pulumi.com/patchForce", "true" },
 },
 Name = "example",
 },
 Data =
 {
 { "foo", "bar" },
 },
 });
});

package generated_program;

import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.kubernetes;
import com.pulumi.kubernetes.core_v1.ConfigMap;
import com.pulumi.kubernetes.core_v1.ConfigMapArgs;
import com.pulumi.kubernetes.meta_v1.inputs.ObjectMetaArgs;
import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;

public class App {
 public static void main(String[] args) {
 Pulumi.run(App::stack);
 }

 public static void stack(Context ctx) {
 var example = new ConfigMap("example", ConfigMapArgs.builder()
 .metadata(ObjectMetaArgs.builder()
 .annotations(Map.of("pulumi.com/patchForce", "true"))
 .name("example")
 .build())
 .data(Map.of("foo", "bar"))
 .build());

 }
}

resources:
 example:
 type: kubernetes:core/v1:ConfigMap
 properties:
 metadata:
 annotations:
 pulumi.com/patchForce: "true"
 name: example
 data:
 foo: bar

Improved diff experience

The process for calculating what changes a provider needs to make to achieve the desired state is called “diffing”. This is a complex process that accounts for the current and previous state of the Pulumi program (the “inputs”), the stack’s state/checkpoint, and the actual current state of any resources managed by the provider. A major challenge of this process is detecting “drift”, where resources have been changed by something other than the Pulumi stack so that these changes are not reflected in Pulumi’s view of the world. The Server-side Apply (SSA) support available in v3.x versions does detect resource drift, but the implementation was slow for larger stacks. The v4 provider now detects resource drift in both Client-side Apply and Server-side Apply by running a refresh operation. This brings the behavior in line with other providers, and offers consistent performance that scales well to larger stacks.

As part of this work, we removed the use of the kubectl.kubernetes.io/last-applied-configuration annotation, which was previously used by the provider to track input values for Kubernetes resources. This change addresses many longstanding issues caused by the use of this annotation, including challenges working with resources that were modified by tools other than kubectl.

Migrating to Kubernetes 4.0

You can migrate your existing Pulumi Kubernetes projects to 4.0 today, or start a new project targeting 4.0 with any of the Pulumi templates.

For users migrating to 4.0, check out the v4 Provider Migration Guide with full details of the additional changes in this release.

Getting Started

Try Infrastructure as Code for Kubernetes by following this quick getting started guide.

Join us for a free, on-demand webinar with Pulumi and the CNCF to learn how to use Pulumi’s Kubernetes Provider and other open-source tools to build internal developer platforms.Register →

Pulumi AI: The Fastest Way to Discover, Learn, and Build Infrastructure as Code

Luke Hoban — Thu, 27 Apr 2023 10:00:00 -0800

Note: This post discusses Pulumi Copilot, which Pulumi Neo has replaced. Learn about Neo →

There are new intelligent cloud management capabilities available in Pulumi Copilot.

We recently released Pulumi Neo, a purpose-built AI Assistant that can create cloud infrastructure using Pulumi. It builds on the power of Large Language Models (LLMs) and GPT to dramatically reduce the time it takes to discover, learn and use new cloud infrastructure APIs.

We’ve seen amazing engagement and stories from Pulumi users about the impact this tool has had for them over the past few weeks. In this post, we’ll dive deeper into this new technology, and share why we and so many other Pulumi users are so excited about Pulumi Neo.

We’ve got a few upcoming opportunities to learn more about Pulumi Insights and Pulumi AI. Come join us!

May 3rd: Reddit AMA: Pulumi Insights
May 11th: Workshop: Getting Started with Pulumi Insights

Why Pulumi AI?

It’s often hard to appreciate just how large and complex modern cloud platforms have become. Pulumi itself supports 130 cloud providers, with many individual providers having over 1,000 resources, and many of those resources have hundreds of properties. This adds up to millions of configurable properties across the entire surface area! Thank heavens Pulumi provides a consistent programming model for the community.

One of the constant challenges for teams working in the cloud is finding specific examples that cover the set of cloud infrastructure resources and configurations that they require. By far, the most common question that we see in our Pulumi Community Slack is from users looking for an example covering a specific use case – and while many times we can point them to one of the thousands listed in our Pulumi Examples repository or enumerated in the Pulumi Registry, there are also times that the specific combination of technologies that a user needs have not yet been captured and stored.

Pulumi AI addresses this need in a direct way. Instead of having to create every possible example by hand, users can just ask Pulumi AI to generate an example that suits their unique needs, no matter how specific and bespoke the request might be.

Discovering, Learning, and Building with Pulumi AI

For example, suppose that I want to run Metabase on Azure. There are a few guides that discuss this on the internet, however, it’s not immediately clear what the best compute and database options might be. Most guides only describe how to point-and-click in the Azure Portal, and several of the guides are many years old and out-of-date. It could take hours to synthesize all the information on this subject and turn it into a working Infrastructure as Code solution with Pulumi. Instead, I can just type “run Metabase in Azure” into Pulumi AI and get an excellent starting point in seconds:

The result from my prompt is a very good, short Pulumi program that defines all of the pieces needed to run Metabase in Azure. Already, this has saved me time reading through content online, picking out a set of building block technologies to use, and figuring out how to connect them.

But it’s not perfect! There may be a few errors when I copy this code into a Pulumi program. This is where Pulumi AI’s interactive mode with memory becomes invaluable. I can just let Pulumi AI know about the error and ask it to fix the problem.

Pulumi AI doesn’t just rename the property – it actually recognizes that the property expects an array, and adjusts to match this requirement.

Pulumi AI can update the code for each of the three errors, iterate on the program, and ultimately complete a fully working solution in less than five minutes. (Pulumi AI has literally saved me hours on completing this and other tasks just like it!)

The ability to interactively fine tune results to your specific needs is one of the most amazing benefits of Pulumi AI with static code examples. If Pulumi AI does something slightly different than you want, you can ask it to change it – and it’s still not quite right, you can change it again. Compare this approach to finding an article on Medium that is almost, but not quite what you want. The difference in productivity is magical.

Interactivity using the `pulumi-ai` CLI

The Pulumi AI website is fantastic for learning and discovery and the end result is optimized to show how Pulumi code might look. You can then take this code and deploy it using the pulumi CLI. However, we can go a step further and automate this process and build infrastructure directly using natural language, without needing to write (or copy/paste) any IaC code at all.

This is what the pulumi-ai CLI enables. By combining Pulumi AI with Pulumi Automation API, the pulumi-ai CLI takes a natural language query, creates a Pulumi program, and automatically deploys that program using the Pulumi Automation API, showing the user all the outputs (like IP addresses, resource names, etc.).

This integration delivers an incredible experience. With just a few words of natural language guidance, you can create cloud infrastructure, modify it, extend it, and ultimately, if you so choose see it run live on the internet.

Giving Pulumi AI the tools to build cloud infrastructure is an awesome power, and of course, with great power comes great responsibility. Use this in a development sandbox, with appropriately scoped permissions, and be sure to audit the infrastructure that is deployed.

Is this the right level of abstraction?

It’s reasonable to ask whether Pulumi AI offers the ideal level of abstraction for AI assistance for Infrastructure as Code. Kelsey Hightower mentioned this on the day that Pulumi AI launched:

We believe that fundamentally, both code and templates, while having different use cases, are important. Giving teams working the tools to discover, learn and use cloud infrastructure APIs via IaC is an incredible productivity boost and addresses real and important needs around examples, content, and tackling cloud complexity. At the same time, we all want to see the level of abstraction raised for cloud infrastructure. Indeed, one of our primary goals at Pulumi since our founding has been to raise the abstraction level from raw cloud primitives to higher level components that are aligned with the needs of specific usage scenarios.

This inspires two objectives we have in developing Pulumi AI.

The first is to enable Pulumi AI to use higher-level components when available, so that it is working at the right level of abstraction for the problem at hand. Pulumi AI does this today, for example, by favoring use of the higher level awsx and eks packages over the raw primitives from the aws package in its output. This ensures that Pulumi AI solves complex problems in simple and composable ways when possible, but falls down into lower level building blocks when it must. Just like an expert (human!) Pulumi developer would do when given the same set of tasks.

The second is to avoid the need to even think about IaC during the discovery and development phase. With the pulumi-ai CLI, you can iterate using only natural language, keeping your mental model abstraction out of the details, and focused only on the results - did I get the externally visible outputs I requested?

All of these are important, and each of them represents a direction of our investment in Pulumi AI.

The Future

Pulumi AI is already offering practical value for thousands of users. We have an incredible number of opportunities to extend and apply this technology to many aspects of cloud engineering. The impact is profound today and will be even more so in the future.

We’re just getting started and hope you join us on our journey! Experience Pulumi Neo today and join us for one of these upcoming events to learn more:

Pulumi Insights: Intelligence for Cloud Infrastructure

Luke Hoban — Thu, 13 Apr 2023 06:00:00 -0800

Note: This post discusses Pulumi Copilot, which Pulumi Neo has replaced. Learn about Neo →

You can now get Insights capabilities on all your cloud infrastructure, even resources not managed by Pulumi IaC. Get started with Insights 2.0

There are new intelligent cloud management capabilities available in Pulumi Copilot. Learn More

We’ve seen incredible acceleration of cloud adoption over the past 5 years. Pulumi’s flagship open source IaC solution gives engineers great tools to scale up their cloud infrastructure using the same programming languages and tools they already know and love. As a result, thousands of companies of every size and scale have adopted Pulumi as a lynchpin of their cloud infrastructure strategy.

Today we’re excited to announce Pulumi Insights, the next major productivity enhancement for infrastructure as code. Pulumi Insights provides intelligence, search, and analytics over any infrastructure, in any cloud across your organization, leveraging the latest advances in generative AI and Large Language Models (LLMs). Whether you have an AWS VPC, a Kubernetes CRD, or a DataDog alarm definition, Pulumi Insights enables you to intelligently find and interact with all of your resources from within the Pulumi Cloud.

Pulumi Insights was born from partnering with our community and observing their struggles to make sense of millions of cloud resources and resource updates per month, across hundreds of clouds, regions, and accounts. With Pulumi Insights, we now help teams manage cloud footprints of any size with the following key components:

🧠 Intelligence: New AI-powered tool for learning, discovering and building cloud infrastructure with Pulumi, using the power of generative AI and Large Language Models (LLMs).
🔎 Search: Multi-cloud resource search, powered by the Pulumi resource supergraph, which surfaces rich metadata and relationships between cloud infrastructure.
📊 Analytics: Visualize Pulumi resource data with your favorite BI tools, including Excel, Tableau and others, using your own data warehouse for reporting on cost, compliance, and other operational use cases.

All of these features work across Pulumi’s 130+ supported cloud providers, including AWS, Azure, Google Cloud, Cloudflare, and Kubernetes, and can be defined in any programming language, including TypeScript, Python, .NET, Go, Java and YAML.

Pulumi Insights is available for all Pulumi Cloud users today.

Intelligence - Productivity with the Power of AI

The scale of Cloud APIs is often intimidating for new and even seasoned cloud engineers. At Pulumi, we provide access to hundreds of clouds, tens-of-thousands of resources and hundreds-of-thousands of resource properties. Pulumi has many thousands of examples showing how to solve specific problems, and yet developers still sometimes struggle to find the perfect code snippet that addresses their exact set of requirements.

Our new Pulumi Neo is an assistant purpose-built for creating cloud infrastructure using Pulumi. It builds on the power of Large Language Models and GPT to dramatically reduce the time it takes to discover, learn and use new cloud infrastructure APIs. Because Pulumi uses popular programming languages that GPT has already been trained on, Pulumi AI understands how to solve complex Cloud Infrastructure problems.

Using natural language, you can ask to accomplish a specific Cloud Infrastructure goal, and receive a Pulumi program in your favorite language. In addition, Pulumi AI supports iterating on your cloud infrastructure allowing the addition of new features, improving security and performance, fixing correctness issues and clarifying requirements.

The open source pulumi-ai CLI takes these ideas even further, by interactively deploying any cloud infrastructure you request using natural language prompts. You can build arbitrarily complex cloud infrastructure, without requiring detailed knowledge of specific APIs! Of course, since we’re using Pulumi’s Automation API under the hood, you can go to the Pulumi Cloud console to see everything that has been deployed, and even jump straight into your Cloud Console to inspect the results.

Pulumi AI will enhance your team’s ability to go from concept to product. This demo shows Pulumi AI taking natural language input and deploying cloud infrastructure and services which are exposed live on the internet, constantly updating as you refine your request.

Check out Pulumi Neo today at pulumi.com/neo.

Search - Find Anything in Any Cloud

Teams don’t use just a single cloud or a single cloud account. Indeed, many teams manage resources across dozens of clouds and hundreds of accounts! It can be almost impossible to aggregate and search your infrastructure across these accounts, since every cloud and every account is (by design!) a walled garden.

Pulumi Resource Search offers multi-cloud search and analytics across every environment in your organization. You can issue queries that find all of your AWS VPCs, or all of your VPCs in AWS and Azure, or all resources with the “production” tag across all cloud environments. You also have access to statistics about cloud usage, including a breakdown by cloud provider, resource type, and department. Resource search enables you to find the needle within your cloud haystack and visualize cloud consumption trends.

Infrastructure can be discovered interactively with a rich, structured query language, for example:

package:snowflake
data-warehouse stack:production package:aws modified:>=2023-03-21
stack:production type:”github:index/issuelabel:IssueLabel”
package:aws project:pulumi-cloud-import-aws stack:-staging-us-west-2

Search is powered by the Pulumi resource supergraph, which exposes rich metadata around cloud infrastructure resources and models relationships between resources, such as the project and stack they are deployed from, the components they are part of, the provider used to manage them. Pulumi enables faceted search, which allows users to specify criteria by applying multiple filters based on classification of resources.

AI Assist for Search

We are also making available today a private preview of AI Assist for resource search that enables natural language queries. You can provide a plain English (or Spanish, or French, or Japanese!) request, to express queries where you might not know the exact syntax, type tokens, or package names. AI Assist makes it easier to gain insight over infrastructure. You can join the waitlist now for access to this additional functionality for search.

Cloud Import

While all resources managed by Pulumi are part of the resource supergraph, you can also bring resources defined and managed outside of Pulumi into the graph to make them available for Insights. We’re also opening up a private preview of Pulumi Cloud Import, a feature that lets you apply Search and Insights to resources created outside of Pulumi, whether via ClickOps or through other tools like Terraform, CloudFormation, ARM or Kubernetes YAML. You can join the waitlist for access to these new features today.

Analytics - Deeper Insights for Cloud Infrastructure

We are also making the data in the Pulumi supergraph available for export into other Business Intelligence tools for deeper analysis in combination with other key business data. You can visualize resource data with your favorite BI tools, including Excel, Tableau, Looker, PowerBI and others, using your own data warehouse, such as Snowflake, Redshift, Databricks and more, and create reporting on cost, compliance, and many other operational use cases.

The ability to join Pulumi resource data with other key business data in your data warehouse enables users to address a broad range of key business needs - answering questions like:

What infrastructure is stale and possibly orphaned / contributing to our cost?
Where are we spending the most, broken down by department or environment?
Are all my primary data stores marked as protected to avoid unintentional deletion?
What are my most commonly used clouds? What are the most common resource types?
Who has performed the most updates across my organization?

For example, here is a dashboard in Metabase, created by combining Pulumi resource data with cost and department data in a Snowflake data warehouse.

Pulumi resource Data Export is available today both interactively in the console and via REST API . Learn more about these features in the Pulumi Data Export documentation.

Conclusion

We’d like to express our thanks to the entire community that helped us design, develop, test and ship this incredible release. From Intelligence to Search to Analytics, the new features in Pulumi Insights will make all of us more productive. We’re incredibly excited by the use cases we’ve already seen our customers build, and we can’t wait to see what you build on top of these foundations!

To get started with Pulumi Insights today:

Sign up or log in to Pulumi Cloud
Try out Pulumi Neo
Watch the videos for Pulumi Insights, Pulumi AI and Pulumi Search
Check out the Pulumi Insights Documentation
Join the Pulumi Community Slack

Pulumi Crosswalk for AWS 1.0: AWSX, EKS, and AWS API Gateway

Luke Hoban — Mon, 28 Nov 2022 12:00:00 -0700

Portions of this blog post are out of date. See the AWS guides for an updated overview and examples.

We introduced Crosswalk for AWS three years ago as a library of components on top of the core AWS platform to make it easier to get from zero to production on AWS, easier to adopt AWS best practices by default, and easier to evolve your AWS infrastructure as your application needs mature. Since then, we’ve added many new capabilities, expanded the portfolio of libraries, and made these libraries available to all Pulumi languages. We’ve also seen thousands of Pulumi customers, including more than 25% of all Pulumi AWS users, adopting one or more of the Crosswalk for AWS components to aid in delivering their AWS-based applications and services.

Today, we’re excited to deliver the 1.0 releases of the three core libraries that make up Crosswalk for AWS: AWSX 1.0, EKS 1.0 and AWS API Gateway 1.0. These 1.0 releases offer a stable supported foundation, available in all Pulumi languages, for these rich infrastructure components.

Why Crosswalk for AWS?

Pulumi provides access to 100 providers (and counting!) for working with a wide variety of cloud and SaaS platforms. But since the very beginning of our work on Pulumi, AWS has been the most used platform, both by the number of organizations using the provider, and by the average number of cloud resources managed per organization.

One of our guiding lights for Pulumi has always been to help organizations to scale up the complexity, capability, and richness of what they can build in the cloud, and in specific on the AWS platform. AWS offers an enormous number of powerful building block services, and organizations can stitch those building blocks together in countless ways to accomplish their needs. Pulumi brings a variety of software engineering tools to bear to help with this - rich programming languages, IDE productivity, testing, policy, and fast iteration. But none is more important than Pulumi’s focus on enabling the creation of reusable infrastructure components - higher level components that provide their own API interface built from the powerful building blocks of the underlying cloud resources.

The Crosswalk for AWS libraries offer components that help address many of the most common use cases we’ve seen AWS users need to tackle - configuring the many components of their VPC network setup, running containers on Amazon ECS and Fargate, configuring a fully featured and production ready Amazon EKS Cluster, and setting up the infrastructure and functions needed as part of a serverless architecture. All with just a few lines of code that offers AWS best practices out of the box, while offering the ability to scale up into the full features of the AWS platform.

Crosswalk for AWS 1.0

The 1.0 releases of the awsx, eks and aws-apigateway packages offer a stable and supported foundation, available in all Pulumi languages, for these rich infrastructure components. They are available in the Pulumi Registry, and in all of the supported Pulumi package managers.

Note: For TypeScript users, the 1.0.0 version of the awsx package contains some breaking changes from the 0.40.* versions that were available only in TypeScript. The previously supported APIs are now available in the @pulumi/awsx/classic module inside the TypeScript awsx package. Users migrating from 0.40.* to 1.0.* can choose to either adapt their code and resources to the new APIs (which might require replacement or import of resources), or else to continue to use the classic submodule.

A few of the most significant components and libraries available in Crosswalk for AWS 1.0 are highlighted in the examples below, all available as part of Pulumi’s Architecture Templates.

Containers (ECS)

Amazon Elastic Container Service (ECS) provides foundational services for running containers directly on the AWS platform, both against self-managed EC2 clusters, as well as in AWS Fargate’s managed serverless container platform.

The AWSX 1.0 package provides components that make it incredibly easy to take a container, or even just a folder with a Dockerfile, and run it in a production-ready environment in AWS. And unlike some bespoke solutions for container deployment, with AWSX, configuring networking, load-balancing, secrets, managed databases, or anything else needed is just a few lines of code away.

Our recently released Container Service on AWS template provides a great starting point for leveraging the AWSX 1.0 support for ECS, Fargate, ECR and ALB.

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";

const config = new pulumi.Config();
const containerPort = config.getNumber("containerPort") || 80;
const cpu = config.getNumber("cpu") || 512;
const memory = config.getNumber("memory") || 128;

// An ECS cluster to deploy into
const cluster = new aws.ecs.Cluster("cluster", {});

// An ALB to serve the container endpoint to the internet
const loadbalancer = new awsx.lb.ApplicationLoadBalancer("loadbalancer", {});

// An ECR repository to store our application's container image
const repo = new awsx.ecr.Repository("repo", {});

// Build and publish our application's container image from ./app to the ECR repository
const image = new awsx.ecr.Image("image", {
 repositoryUrl: repo.url,
 path: "./app",
});

// Deploy an ECS Service on Fargate to host the application container
const service = new awsx.ecs.FargateService("service", {
 cluster: cluster.arn,
 assignPublicIp: true,
 taskDefinitionArgs: {
 container: {
 image: image.imageUri,
 cpu: cpu,
 memory: memory,
 essential: true,
 portMappings: [{
 containerPort: containerPort,
 targetGroup: loadbalancer.defaultTargetGroup,
 }],
 },
 },
});

// The URL at which the container's HTTP endpoint will be available
export const url = pulumi.interpolate`http://${loadbalancer.loadBalancer.dnsName}`;

from pulumi import Config, Output, export
import pulumi_aws as aws
import pulumi_awsx as awsx

config = Config()
container_port = config.get_int("containerPort", 80)
cpu = config.get_int("cpu", 512)
memory = config.get_int("memory", 128)

# An ECS cluster to deploy into
cluster = aws.ecs.Cluster("cluster")

# An ALB to serve the container endpoint to the internet
loadbalancer = awsx.lb.ApplicationLoadBalancer("loadbalancer")

# An ECR repository to store our application's container image
repo = awsx.ecr.Repository("repo")

# Build and publish our application's container image from ./app to the ECR repository
image = awsx.ecr.Image(
 "image",
 repository_url=repo.url,
 path="./app")

# Deploy an ECS Service on Fargate to host the application container
service = awsx.ecs.FargateService(
 "service",
 cluster=cluster.arn,
 assign_public_ip=True,
 task_definition_args=awsx.ecs.FargateServiceTaskDefinitionArgs(
 container=awsx.ecs.TaskDefinitionContainerDefinitionArgs(
 image=image.image_uri,
 cpu=cpu,
 memory=memory,
 essential=True,
 port_mappings=[awsx.ecs.TaskDefinitionPortMappingArgs(
 container_port=container_port,
 target_group=loadbalancer.default_target_group,
 )],
 ),
 ))

# The URL at which the container's HTTP endpoint will be available
export("url", Output.concat("http://", loadbalancer.load_balancer.dns_name))

package main

import (
 "github.com/pulumi/pulumi-aws/sdk/v5/go/aws/ecs"
 "github.com/pulumi/pulumi-awsx/sdk/go/awsx/ecr"
 ecrx "github.com/pulumi/pulumi-awsx/sdk/go/awsx/ecr"
 ecsx "github.com/pulumi/pulumi-awsx/sdk/go/awsx/ecs"
 lbx "github.com/pulumi/pulumi-awsx/sdk/go/awsx/lb"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi/config"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 cfg := config.New(ctx, "")
 containerPort := 80
 if param := cfg.GetInt("containerPort"); param != 0 {
 containerPort = param
 }
 cpu := 512
 if param := cfg.GetInt("cpu"); param != 0 {
 cpu = param
 }
 memory := 128
 if param := cfg.GetInt("memory"); param != 0 {
 memory = param
 }

 // An ECS cluster to deploy into
 cluster, err := ecs.NewCluster(ctx, "cluster", nil)
 if err != nil {
 return err
 }

 // An ALB to serve the container endpoint to the internet
 loadbalancer, err := lbx.NewApplicationLoadBalancer(ctx, "loadbalancer", nil)
 if err != nil {
 return err
 }

 // An ECR repository to store our application's container image
 repo, err := ecrx.NewRepository(ctx, "repo", nil)
 if err != nil {
 return err
 }

 // Build and publish our application's container image from ./app to the ECR repository
 image, err := ecrx.NewImage(ctx, "image", &ecr.ImageArgs{
 RepositoryUrl: repo.Url,
 Path: pulumi.String("./app"),
 })
 if err != nil {
 return err
 }

 // Deploy an ECS Service on Fargate to host the application container
 _, err = ecsx.NewFargateService(ctx, "service", &ecsx.FargateServiceArgs{
 Cluster: cluster.Arn,
 AssignPublicIp: pulumi.Bool(true),
 TaskDefinitionArgs: &ecsx.FargateServiceTaskDefinitionArgs{
 Container: &ecsx.TaskDefinitionContainerDefinitionArgs{
 Image: image.ImageUri,
 Cpu: pulumi.Int(cpu),
 Memory: pulumi.Int(memory),
 Essential: pulumi.Bool(true),
 PortMappings: ecsx.TaskDefinitionPortMappingArray{
 &ecsx.TaskDefinitionPortMappingArgs{
 ContainerPort: pulumi.Int(containerPort),
 TargetGroup: loadbalancer.DefaultTargetGroup,
 },
 },
 },
 },
 })
 if err != nil {
 return err
 }

 // The URL at which the container's HTTP endpoint will be available
 ctx.Export("url", pulumi.Sprintf("http://%s", loadbalancer.LoadBalancer.DnsName()))
 return nil
 })
}

using System.Collections.Generic;
using Pulumi;
using Aws = Pulumi.Aws;
using Awsx = Pulumi.Awsx;

return await Deployment.RunAsync(() =>
{
 var config = new Config();
 var containerPort = config.GetInt32("containerPort") ?? 80;
 var cpu = config.GetInt32("cpu") ?? 512;
 var memory = config.GetInt32("memory") ?? 128;
 var cluster = new Aws.Ecs.Cluster("cluster");

 var loadbalancer = new Awsx.Lb.ApplicationLoadBalancer("loadbalancer");

 var repo = new Awsx.Ecr.Repository("repo");

 var image = new Awsx.Ecr.Image("image", new()
 {
 RepositoryUrl = repo.Url,
 Path = "./app",
 });

 var service = new Awsx.Ecs.FargateService("service", new()
 {
 Cluster = cluster.Arn,
 AssignPublicIp = true,
 TaskDefinitionArgs = new Awsx.Ecs.Inputs.FargateServiceTaskDefinitionArgs
 {
 Container = new Awsx.Ecs.Inputs.TaskDefinitionContainerDefinitionArgs
 {
 Image = image.ImageUri,
 Cpu = cpu,
 Memory = memory,
 Essential = true,
 PortMappings = new[]
 {
 new Awsx.Ecs.Inputs.TaskDefinitionPortMappingArgs
 {
 ContainerPort = containerPort,
 TargetGroup = loadbalancer.DefaultTargetGroup,
 },
 },
 },
 },
 });

 return new Dictionary<string, object?>
 {
 ["url"] = loadbalancer.LoadBalancer.Apply(loadBalancer => Output.Format($"http://{loadBalancer.DnsName}")),
 };
});

name: aws-container
description: Containers on AWS
runtime: yaml

config:
 containerPort:
 default: 80
 cpu:
 default: 512
 memory:
 default: 128

resources:
 # An ECS cluster to deploy into
 cluster:
 type: aws:ecs:Cluster
 # An ALB to serve the container endpoint to the internet
 loadbalancer:
 type: awsx:lb:ApplicationLoadBalancer
 # An ECR repository to store our application's container image
 repo:
 type: awsx:ecr:Repository
 # Build and publish our application's container image from ./app to the ECR repository
 image:
 type: awsx:ecr:Image
 properties:
 repositoryUrl: ${repo.url}
 path: ./app
 # Deploy an ECS Service on Fargate to host the application container
 service:
 type: awsx:ecs:FargateService
 properties:
 cluster: ${cluster.arn}
 taskDefinitionArgs:
 container:
 image: ${image.imageUri}
 cpu: ${cpu}
 memory: ${memory}
 essential: true
 portMappings:
 - containerPort: ${containerPort}
 targetGroup: ${loadbalancer.defaultTargetGroup}

outputs:
 # The URL at which the container's HTTP endpoint will be available
 url: http://${loadbalancer.loadBalancer.dnsName}

Kubernetes (EKS)

Amazon Elastic Kubernetes Service (EKS) offers a managed Kubernetes service for standing up a best practices Kubernetes cluster within AWS.

The EKS 1.0 package offers support for configuring the many building blocks needed to configure a complete EKS cluster, across both AWS resources as well as in-cluster Kubernetes resources using Pulumi’s native kubernetes provider. With eks.Cluster, eks.NodeGroup, eks.NodeGroupV2 and eks.ManagedNodeGroup this package makes it easy to configure a cluster using any of the compute options available in AWS in just a few lines of code.

Our recently released Kubernetes Cluster on AWS template provides a great starting point for leveraging the eks 1.0 support for EKS, taking advantage of default support for private networking and EC2-based compute.

import * as pulumi from "@pulumi/pulumi";
import * as awsx from "@pulumi/awsx";
import * as eks from "@pulumi/eks";

// Grab some values from the Pulumi configuration (or use default values)
const config = new pulumi.Config();
const minClusterSize = config.getNumber("minClusterSize") || 3;
const maxClusterSize = config.getNumber("maxClusterSize") || 6;
const desiredClusterSize = config.getNumber("desiredClusterSize") || 3;
const eksNodeInstanceType = config.get("eksNodeInstanceType") || "t2.medium";
const vpcNetworkCidr = config.get("vpcNetworkCidr") || "10.0.0.0/16";

// Create a new VPC
const eksVpc = new awsx.ec2.Vpc("eks-vpc", {
 enableDnsHostnames: true,
 cidrBlock: vpcNetworkCidr,
});

// Create the EKS cluster
const eksCluster = new eks.Cluster("eks-cluster", {
 // Put the cluster in the new VPC created earlier
 vpcId: eksVpc.vpcId,
 // Public subnets will be used for load balancers
 publicSubnetIds: eksVpc.publicSubnetIds,
 // Private subnets will be used for cluster nodes
 privateSubnetIds: eksVpc.privateSubnetIds,
 // Change configuration values to change any of the following settings
 instanceType: eksNodeInstanceType,
 desiredCapacity: desiredClusterSize,
 minSize: minClusterSize,
 maxSize: maxClusterSize,
 // Do not give the worker nodes public IP addresses
 nodeAssociatePublicIpAddress: false,
 // Uncomment the next two lines for a private cluster (VPN access required)
 // endpointPrivateAccess: true,
 // endpointPublicAccess: false
});

// Export some values for use elsewhere
export const kubeconfig = eksCluster.kubeconfig;
export const vpcId = eksVpc.vpcId;

import pulumi
import pulumi_awsx as awsx
import pulumi_eks as eks

# Get some values from the Pulumi configuration (or use defaults)
config = pulumi.Config()
min_cluster_size = config.get_float("minClusterSize", 3)
max_cluster_size = config.get_float("maxClusterSize", 6)
desired_cluster_size = config.get_float("desiredClusterSize", 3)
eks_node_instance_type = config.get("eksNodeInstanceType", "t2.medium")
vpc_network_cidr = config.get("vpcNetworkCidr", "10.0.0.0/16")

# Create a VPC for the EKS cluster
eks_vpc = awsx.ec2.Vpc("eks-vpc",
 enable_dns_hostnames=True,
 cidr_block=vpc_network_cidr)

# Create the EKS cluster
eks_cluster = eks.Cluster("eks-cluster",
 # Put the cluster in the new VPC created earlier
 vpc_id=eks_vpc.vpc_id,
 # Public subnets will be used for load balancers
 public_subnet_ids=eks_vpc.public_subnet_ids,
 # Private subnets will be used for cluster nodes
 private_subnet_ids=eks_vpc.private_subnet_ids,
 # Change configuration values to change any of the following settings
 instance_type=eks_node_instance_type,
 desired_capacity=desired_cluster_size,
 min_size=min_cluster_size,
 max_size=max_cluster_size,
 # Do not give worker nodes a public IP address
 node_associate_public_ip_address=False,
 # Uncomment the next two lines for private cluster (VPN access required)
 # endpoint_private_access=true,
 # endpoint_public_access=false
 )

# Export values to use elsewhere
pulumi.export("kubeconfig", eks_cluster.kubeconfig)
pulumi.export("vpcId", eks_vpc.vpc_id)

package main

import (
 "github.com/pulumi/pulumi-awsx/sdk/go/awsx/ec2"
 "github.com/pulumi/pulumi-eks/sdk/go/eks"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi/config"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // Get some configuration values or set default values
 cfg := config.New(ctx, "")
 minClusterSize, err := cfg.TryInt("minClusterSize")
 if err != nil {
 minClusterSize = 3
 }
 maxClusterSize, err := cfg.TryInt("maxClusterSize")
 if err != nil {
 maxClusterSize = 6
 }
 desiredClusterSize, err := cfg.TryInt("desiredClusterSize")
 if err != nil {
 desiredClusterSize = 3
 }
 eksNodeInstanceType, err := cfg.Try("eksNodeInstanceType")
 if err != nil {
 eksNodeInstanceType = "t2.medium"
 }
 vpcNetworkCidr, err := cfg.Try("vpcNetworkCidr")
 if err != nil {
 vpcNetworkCidr = "10.0.0.0/16"
 }

 // Create a new VPC, subnets, and associated infrastructure
 eksVpc, err := ec2.NewVpc(ctx, "eks-vpc", &ec2.VpcArgs{
 EnableDnsHostnames: pulumi.Bool(true),
 CidrBlock: &vpcNetworkCidr,
 })
 if err != nil {
 return err
 }

 // Create a new EKS cluster
 eksCluster, err := eks.NewCluster(ctx, "eks-cluster", &eks.ClusterArgs{
 // Put the cluster in the new VPC created earlier
 VpcId: eksVpc.VpcId,
 // Public subnets will be used for load balancers
 PublicSubnetIds: eksVpc.PublicSubnetIds,
 // Private subnets will be used for cluster nodes
 PrivateSubnetIds: eksVpc.PrivateSubnetIds,
 // Change configuration values above to change any of the following settings
 InstanceType: pulumi.String(eksNodeInstanceType),
 DesiredCapacity: pulumi.Int(desiredClusterSize),
 MinSize: pulumi.Int(minClusterSize),
 MaxSize: pulumi.Int(maxClusterSize),
 // Do not give the worker nodes a public IP address
 NodeAssociatePublicIpAddress: pulumi.Bool(false),
 // Uncomment the next two lines for a private cluster (VPN access required)
 // EndpointPrivateAccess: pulumi.Bool(true),
 // EndpointPublicAccess: pulumi.Bool(false),
 })
 if err != nil {
 return err
 }

 // Export some values in case they are needed elsewhere
 ctx.Export("kubeconfig", eksCluster.Kubeconfig)
 ctx.Export("vpcId", eksVpc.VpcId)
 return nil
 })
}

using Pulumi;
using Awsx = Pulumi.Awsx;
using Eks = Pulumi.Eks;
using System.Collections.Generic;

return await Deployment.RunAsync(() =>
{
 // Grab some values from the Pulumi configuration (or use default values)
 var config = new Config();
 var minClusterSize = config.GetInt32("minClusterSize") ?? 3;
 var maxClusterSize = config.GetInt32("maxClusterSize") ?? 6;
 var desiredClusterSize = config.GetInt32("desiredClusterSize") ?? 3;
 var eksNodeInstanceType = config.Get("eksNodeInstanceType") ?? "t2.medium";
 var vpcNetworkCidr = config.Get("vpcNetworkCidr") ?? "10.0.0.0/16";

 // Create a new VPC
 var eksVpc = new Awsx.Ec2.Vpc("eks-vpc", new()
 {
 EnableDnsHostnames = true,
 CidrBlock = vpcNetworkCidr,
 });

 // Create the EKS cluster
 var eksCluster = new Eks.Cluster("eks-cluster", new()
 {
 // Put the cluster in the new VPC created earlier
 VpcId = eksVpc.VpcId,

 // Public subnets will be used for load balancers
 PublicSubnetIds = eksVpc.PublicSubnetIds,

 // Private subnets will be used for cluster nodes
 PrivateSubnetIds = eksVpc.PrivateSubnetIds,

 // Change configuration values to change any of the following settings
 InstanceType = eksNodeInstanceType,
 DesiredCapacity = desiredClusterSize,
 MinSize = minClusterSize,
 MaxSize = maxClusterSize,

 // Do not give the worker nodes public IP addresses
 NodeAssociatePublicIpAddress = false,

 // Uncomment the next two lines for a private cluster (VPN access required)
 // EndpointPrivateAccess = true,
 // EndpointPublicAccess = false,
 });

 // Export some values for use elsewhere
 return new Dictionary<string, object?>
 {
 ["kubeconfig"] = eksCluster.Kubeconfig,
 ["vpcId"] = eksVpc.VpcId,
 };
});

name: aws-kubernetes-cluster
description: Kubernetes Cluster on AWS
runtime: yaml
config:
 minClusterSize:
 default: 3
 maxClusterSize:
 default: 6
 desiredClusterSize:
 default: 3
 eksNodeInstanceType:
 default: t2.medium
 vpcNetworkCidr:
 default: 10.0.0.0/16
resources:
 # Create a VPC for the EKS cluster
 eks-vpc:
 type: awsx:ec2:Vpc
 properties:
 enableDnsHostnames: true
 cidrBlock: ${vpcNetworkCidr}
 # Create the EKS cluster
 eks-cluster:
 type: eks:Cluster
 properties:
 # Put the cluster in the new VPC created earlier
 vpcId: ${eks-vpc.vpcId}
 # Public subnets will be used for load balancers
 publicSubnetIds: ${eks-vpc.publicSubnetIds}
 # Private subnets will be used for cluster nodes
 privateSubnetIds: ${eks-vpc.privateSubnetIds}
 # Change configuration section above to change any of the following settings
 instanceType: ${eksNodeInstanceType}
 desiredCapacity: ${desiredClusterSize}
 minSize: ${minClusterSize}
 maxSize: ${maxClusterSize}
 # Do not give the worker nodes public IP addresses
 nodeAssociatePublicIpAddress: false
 # Uncomment next two lines for private cluster access (VPN access required)
 # endpointPrivateAccess: true
 # endpointPublicAccess: false
outputs:
 # Output the Kubeconfig for the cluster
 kubeconfig: ${eks-cluster.kubeconfig}
 vpcId: ${eks-vpc.vpcId}

Serverless (Lambda, API Gateway)

AWS Lambda and the serverless architectures it enables provide native support for event based compute triggered off of a wide variety of events from within the AWS platform. Coupled with services like AWS API Gateway, this offers simple ways to serve HTTP-based APIs and much, much more.

Pulumi’s built in support for AWS Lambda functions and the new aws-apigateway 1.0 package make it simple to compose these services to build serverless applications and services on AWS in just a few lines of code.

Our recently released AWS Serverless Application template provides a great starting point for leveraging the aws-apigateway 1.0 support for creating a complete API Gateway-based serverless application, serving both static content from Amazon S3 as well as dynamic backend logic from AWS Lambda.

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as apigateway from "@pulumi/aws-apigateway";

// A Lambda function to invoke
const fn = new aws.lambda.CallbackFunction("fn", {
 callback: async (ev, ctx) => {
 return {
 statusCode: 200,
 body: new Date().toISOString(),
 };
 }
})

// A REST API to route requests to HTML content and the Lambda function
const api = new apigateway.RestAPI("api", {
 routes: [
 { path: "/", localPath: "www"},
 { path: "/date", method: "GET", eventHandler: fn },
 ]
});

// The URL at which the REST API will be served.
export const url = api.url;

import json
import pulumi
import pulumi_aws as aws
import pulumi_aws_apigateway as apigateway

# An execution role to use for the Lambda function
role = aws.iam.Role("role",
 assume_role_policy=json.dumps({
 "Version": "2012-10-17",
 "Statement": [{
 "Action": "sts:AssumeRole",
 "Effect": "Allow",
 "Principal": {
 "Service": "lambda.amazonaws.com",
 },
 }],
 }),
 managed_policy_arns=[aws.iam.ManagedPolicy.AWS_LAMBDA_BASIC_EXECUTION_ROLE])

# A Lambda function to invoke
fn = aws.lambda_.Function("fn",
 runtime="python3.9",
 handler="handler.handler",
 role=role.arn,
 code=pulumi.FileArchive("./function"))

# A REST API to route requests to HTML content and the Lambda function
api = apigateway.RestAPI("api",
 routes=[
 apigateway.RouteArgs(path="/", local_path="www"),
 apigateway.RouteArgs(path="/date", method=apigateway.Method.GET, event_handler=fn)
 ])

# The URL at which the REST API will be served.
pulumi.export("url", api.url)

package main

import (
 "encoding/json"

 "github.com/pulumi/pulumi-aws-apigateway/sdk/go/apigateway"
 "github.com/pulumi/pulumi-aws/sdk/v5/go/aws/iam"
 "github.com/pulumi/pulumi-aws/sdk/v5/go/aws/lambda"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // An execution role to use for the Lambda function
 policy, err := json.Marshal(map[string]interface{}{
 "Version": "2012-10-17",
 "Statement": []map[string]interface{}{
 map[string]interface{}{
 "Action": "sts:AssumeRole",
 "Effect": "Allow",
 "Principal": map[string]interface{}{
 "Service": "lambda.amazonaws.com",
 },
 },
 },
 })
 if err != nil {
 return err
 }

 role, err := iam.NewRole(ctx, "role", &iam.RoleArgs{
 AssumeRolePolicy: pulumi.String(policy),
 ManagedPolicyArns: pulumi.StringArray{
 iam.ManagedPolicyAWSLambdaBasicExecutionRole,
 },
 })
 if err != nil {
 return err
 }

 // A Lambda function to invoke
 fn, err := lambda.NewFunction(ctx, "fn", &lambda.FunctionArgs{
 Runtime: pulumi.String("python3.9"),
 Handler: pulumi.String("handler.handler"),
 Role: role.Arn,
 Code: pulumi.NewFileArchive("./function"),
 })
 if err != nil {
 return err
 }

 // A REST API to route requests to HTML content and the Lambda function
 localPath := "www"
 method := apigateway.MethodGET
 api, err := apigateway.NewRestAPI(ctx, "api", &apigateway.RestAPIArgs{
 Routes: []apigateway.RouteArgs{
 apigateway.RouteArgs{Path: "/", LocalPath: &localPath},
 apigateway.RouteArgs{Path: "/date", Method: &method, EventHandler: fn},
 },
 })
 if err != nil {
 return err
 }

 // The URL at which the REST API will be served
 ctx.Export("url", api.Url)
 return nil
 })
}

using System.Collections.Generic;
using System.Text.Json;
using Pulumi;
using Aws = Pulumi.Aws;
using AwsApiGateway = Pulumi.AwsApiGateway;

return await Deployment.RunAsync(() =>
{
 var role = new Aws.Iam.Role("role", new()
 {
 AssumeRolePolicy = JsonSerializer.Serialize(new Dictionary<string, object?>
 {
 ["Version"] = "2012-10-17",
 ["Statement"] = new[]
 {
 new Dictionary<string, object?>
 {
 ["Action"] = "sts:AssumeRole",
 ["Effect"] = "Allow",
 ["Principal"] = new Dictionary<string, object?>
 {
 ["Service"] = "lambda.amazonaws.com",
 },
 },
 },
 }),
 ManagedPolicyArns = new[]
 {
 "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole",
 },
 });

 var fn = new Aws.Lambda.Function("fn", new()
 {
 Runtime = "python3.9",
 Handler = "handler.handler",
 Role = role.Arn,
 Code = new FileArchive("./function"),
 });

 var api = new AwsApiGateway.RestAPI("api", new()
 {
 Routes =
 {
 new AwsApiGateway.Inputs.RouteArgs
 {
 Path = "/",
 LocalPath = "www",
 },
 new AwsApiGateway.Inputs.RouteArgs
 {
 Path = "/date",
 Method = AwsApiGateway.Method.GET,
 EventHandler = fn,
 },
 },
 });

 return new Dictionary<string, object?>
 {
 ["url"] = api.Url,
 };
});

name: aws-serverless
description: Serverless on AWS
runtime: yaml

resources:
 # An execution role to use for the Lambda function
 role:
 type: aws:iam:Role
 properties:
 assumeRolePolicy:
 fn::toJSON:
 Version: 2012-10-17
 Statement:
 - Action: sts:AssumeRole
 Effect: Allow
 Principal:
 Service: lambda.amazonaws.com
 managedPolicyArns:
 - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

 # A Lambda function to invoke
 fn:
 type: aws:lambda:Function
 properties:
 runtime: python3.9
 handler: handler.handler
 role: ${role.arn}
 code:
 fn::fileArchive: ./function

 # A REST API to route requests to HTML content and the Lambda function
 api:
 type: aws-apigateway:RestAPI
 properties:
 routes:
 - path: /
 localPath: www
 - path: /date
 method: GET
 eventHandler: ${fn}

outputs:
 # The URL at which the REST API will be served.
 url: ${api.url}

Migration from AWSx 0.x to 1.0

Before version 1, AWSx only supported components in TypeScript. All the existing components from the 0.x releases are now available in the classic namespace. The classic namespace will remain until the next major version release but will only receive updates for critical security fixes.

If you’re currently using v0.x and want to upgrade to v1.x you can:

Change references from @pulumi/awsx to @pulumi/awsx/classic to maintain existing behaviour.
Refactor to replace the classic components with the new top-level components. This may require additional code changes and resource re-creation.

What’s Next?

Crosswalk for AWS is free and open source, and you can get started today.

For more on Crosswalk for AWS see:

Announcing: Pulumi Deployments, YAML GA, Arch Templates

Luke Hoban — Wed, 02 Nov 2022 06:15:00 -0700

Some of the code in this post is out of date. See the AWS guides for an updated overview and examples.

Our mission at Pulumi is to enable teams to scale up what they can build in the cloud. Scale up the sophistication and value of their cloud infrastructure investments through software engineering practices. Scale up the automation around delivering cloud infrastructure with software instead of just humans. And scale up the number of developers who can directly benefit from the rich cloud platform capabilities being built by central platform teams in every organization today.

As part of Pulumi Cloud Engineering Days 2022 today we are announcing a set of important new advancements in the Pulumi platform which are all designed to help organizations scale with their infrastructure as code needs.

Pulumi Deployments: The Pulumi Service now supports executing infrastructure deployments as a managed service, driven by click-to-deploy in the Console, Git push-to-deploy from GitHub, programmatic deployments via the Pulumi Service REST API as well as automation via the Pulumi Automation API. Pulumi Deployments enables organizations to scale up their infrastructure automation.
Pulumi YAML GA: In addition to supporting rich general purpose programming languages like TypeScript, Python, Go, .NET and Java, Pulumi added support last May for Pulumi YAML as a simple, declarative interface for accessing the full breadth of the Pulumi Universal Infrastructure as Code platform. Pulumi YAML is now Generally Available as part of its 1.0 release, enabling everyone in an organization to use Pulumi to manage cloud infrastructure.
Architecture Templates: New templates providing architecture-specific starting points for the most common infrastructure as code use cases - Containerized Services, Kubernetes Clusters, Serverless Applications, Static Websites and Virtual Machines - available across all major cloud platforms and supported languages.
Project Level Configuration: As cloud infrastructure projects grow in complexity, Pulumi provides many tools to manage the complexity of the software that defines these projects. But the configuration of that software also grows in complexity. New support for Project-Level Configuration across all Pulumi languages makes it easier to manage complex configuration as cloud projects scale up.

You can check out the Cloud Engineering Days 2022 Keynote to learn more about all of these features, or get started with Pulumi today!

Pulumi Deployments

We launched the Pulumi Automation API two years ago to enable organizations to build their own automated systems using the Pulumi Infrastructure as Code engine. Over the last two years, we’ve seen incredible adoption of Automation API, with millions of infrastructure updates driven by Automation API, and usage in everything from CI systems, to CLI tools, to self-service provisioning platforms, to complex deployment orchestration software.

To scale up how we deploy and manage cloud infrastructure and ultimately unlock the cloud’s value and agility, we believe it will be critical to build software systems around our Infrastructure as Code platforms - systems that scale with software, not just humans.

Today, we are taking the next step in this journey with the introduction of Pulumi Deployments to the Pulumi Service. With Pulumi Deployments, you can run a Pulumi infrastructure as code action (a preview, update, destroy or refresh) inside Pulumi’s managed service. Pulumi provides scalability, observability and security for deployments.

Pulumi Deployments are exposed as a new fundamental capability of the service via the Pulumi Service REST API. They are also now supported in Automation API as a new kind of Remote workspace that can be used in building your own custom infrastructure automation software.

We’ve also used Pulumi Deployments to build new features in the Pulumi Service console, including the ability to click to deploy infrastructure or to set up Git push to deploy workflows that trigger previews on GitHub PR commits and updates on merges to a tracking branch. This provides a simple point-and-click interface to ensure that any Pulumi user can benefit from continuous delivery of infrastructure changes.

Of course, because Pulumi Deployments exposes a convenient programmatic API, you can build your own automation on top of this. In our Pulumi Deploy Demos, we have examples of building Drift Detection automation, Time-to-live self-destruct stacks, per-PR environments and more.

Pulumi Deployments is available in preview now, and you can signup to get access today. Pulumi Deployments is free while in preview. We can’t wait to see what you’ll build on top of Pulumi Deployments! 🚀

👉 Read more in the Pulumi Deployments blog post.

Pulumi YAML

We introduced Pulumi YAML last May as a simple declarative interface to the full breadth of the Pulumi platform, ideal for smaller scale use cases and composition of higher level component building blocks. Developers who may not have been ready to learn or adopt a full blown language for their cloud infrastructure can use YAML to get started.

Our goal has always been to offer the broadest range of programming language options to empower every cloud builder so that they could benefit from the best of Pulumi’s Universal Infrastructure as Code platform.

Pulumi YAML makes it easy to compose any of the tens-of-thousands of resources available to the Pulumi Universal Infrastructure as Code platform, including mixtures of raw cloud resources (like an AWS Lambda Function below) and higher level components authored in other rich Pulumi languages (like the aws-apigateway packages high level RestAPI below).

name: serverlessapp
runtime: yaml
config:
 roleArn:
 type: string
resources:
 fn:
 type: aws:lambda:Function
 properties:
 runtime: python3.9
 handler: handler.handler
 role: ${roleArn}
 code:
 fn::fileArchive: ./function
 api:
 type: aws-apigateway:RestAPI
 properties:
 routes:
 - path: /
 localPath: www
 - path: /date
 method: GET
 eventHandler: ${fn}
outputs:
 url: ${api.url}

Pulumi YAML is the only declarative Infrastructure as Code language that supports convert - the ability to convert a YAML program into a program in any other Pulumi language (TypeScript, Python, Go, .NET and Java). If the complexity of a project ever grows beyond what YAML can comfortably express, you can seamlessly adopt another richer language to continue to scale.

pulumi convert --language python

With the Pulumi YAML GA, we’ve made dozens of improvements to Pulumi YAML. Richer type checking and error reporting makes Pulumi YAML easy to get started with. And IDE integration for VS Code and Emacs makes Pulumi YAML even more productive to work with day-to-day:

👉 Read more in the Pulumi YAML GA blog post.

Architecture Templates

The new Pulumi Architecture Templates offer the easiest way to get started with a complete working piece of infrastructure for one of the most common Infrastructure as Code use cases. Each architecture template is short (~100 lines of code), builds a deployable infrastructure project out of the box, is easy to customize, uses a mix of raw cloud resources and Pulumi components to operate at the right level of abstraction for the use case, and comes with useful documentation.

The initial set of architecture templates support common infrastructure as code use cases across AWS, Azure, Google Cloud and Kubernetes, and in all supported Pulumi languages.

For example, the AWS Containerized Service template provides this starting point, which builds and deploys a local container-based application into a set of managed infrastructure in AWS using ECS, ALB, ECR and IAM resources:

from pulumi import Config, Output, export
import pulumi_aws as aws
import pulumi_awsx as awsx

config = Config()
container_port = config.get_int("containerPort", 80)
cpu = config.get_int("cpu", 512)
memory = config.get_int("memory", 128)

# An ECS cluster to deploy into
cluster = aws.ecs.Cluster("cluster")

# An ALB to serve the container endpoint to the internet
loadbalancer = awsx.lb.ApplicationLoadBalancer("loadbalancer")

# An ECR repository to store our application's container image
repo = awsx.ecr.Repository("repo")

# Build and publish our application's container image from ./app to the ECR repository
image = awsx.ecr.Image(
 "image",
 repository_url=repo.url,
 path="./app")

# Deploy an ECS Service on Fargate to host the application container
service = awsx.ecs.FargateService(
 "service",
 cluster=cluster.arn,
 task_definition_args=awsx.ecs.FargateServiceTaskDefinitionArgs(
 container=awsx.ecs.TaskDefinitionContainerDefinitionArgs(
 image=image.image_uri,
 cpu=cpu,
 memory=memory,
 essential=True,
 port_mappings=[awsx.ecs.TaskDefinitionPortMappingArgs(
 container_port=container_port,
 target_group=loadbalancer.default_target_group,
 )],
 ),
 ))

# The URL at which the container's HTTP endpoint will be available
export("url", Output.concat("http://", loadbalancer.load_balancer.dns_name))

Pulumi Architecture Templates build on Pulumi’s support for creating your own templates. Pulumi programs can be templated by users to create their own starting points for cloud infrastructure projects using pulumi new. You can build your own templates and Deploy with Pulumi buttons to create similar architecture templates within your own team and organization.

👉 Read more in the Pulumi Architecture Templates blog post.

Project Level Configuration

On top of support for defining infrastructure as code using traditional programming languages, Pulumi has always offered a rich configuration system for specifying inputs to a program which might differ between instances of that program (“stacks”) - for example, across the dev, staging and prod environments, or across regional deployments, of what is logically the “same” infrastructure definition. Configuration in Pulumi has so far only been provided at the stack level - each stack provides the complete set of configuration needed to specify how that instance of the Pulumi program will be parameterized.

With the latest Pulumi release, we are excited to be adding Project Level Configuration to Pulumi as well. With project-level configuration, you can specify the configuration needed for each stack once, inside the project file. This specification can offer default values across all stacks, types to enforce validation of values used in each stack, descriptions, and enforcement that certain properties must be secret.

name: baseinfra
runtime: python
config:
 vm-count:
 type: integer
 description: the number of virtual machines to create
 default: 2
 subnets:
 type: array
 description: an array of subnets to create
 items:
 type: string
 vaultKey:
 type: string
 description: which vault key to use for this stack
 secret: true

This is the first step in our broader goals to bring Hierarchical Configuration to Pulumi, avoiding unnecessary copy/paste, simple abstraction of configuration data, and rich validation and contextual user interface support across all aspects of config in the Pulumi experience.

👉 Read more in the Project Level Configuration blog post.

Conclusion

Pulumi helps cloud engineering teams scale up what they can accomplish in the cloud. Today, we announced a collection of new features and enhancements that advance that platform and help you scale your cloud infrastructure teams. These features give platform team new powers to scale up what they can accomplish.

Check out the launch blog posts for each of the new features, sign up for access to Pulumi Deployments, or just dive right in and get started with Pulumi today!

Pulumi YAML General Availability

Luke Hoban — Wed, 02 Nov 2022 06:05:00 -0700

Earlier this year we launched support for Pulumi YAML as a new supported language for Pulumi’s Universal Infrastructure as Code platform. Pulumi YAML offers a simple declarative interface to the full breadth of the Pulumi platform, ideal for smaller scale use cases and composition of higher level component building blocks. And with support for pulumi convert, Pulumi YAML programs can be converted into a program in any other Pulumi language, ensuring you can easily scale up if and when needed.

Today, we’re excited to announce the General Availability of Pulumi YAML with the release of Pulumi YAML 1.0.

GA includes many significant updates to the Pulumi YAML support introduced since the initial preview launch:

Pulumi YAML 1.0: The Pulumi YAML document format is now stable and will be kept backward compatible.
Full support for pulumi convert: Now supporting all Pulumi YAML features and all Pulumi languages.
IDE Support: IDE support for VS Code and Emacs, built on a Pulumi YAML LSP (Language Server Protocol) that can be used to light up improved tooling experience in any IDE.
Type Checking and Diagnostics: Significant improved error messages, and early validation of type-correctness, to provide a great developer experience.
Simplified function syntax: A simpler function syntax to more easily invoke Pulumi Package Functions.
New str package: A new Pulumi Package with many string manipulation functions that can be used in Pulumi YAML and other Pulumi languages.
Pulumi Kubernetes Operator Inline Program Support: Support for managing Pulumi cloud resources from directly within the Kubernetes resource model.

You can get started with Pulumi YAML today!

Why Pulumi YAML?

Cloud teams love YAML and markup languages due to the simplicity and ease of getting started. At Pulumi we work with organizations every day that have reached the “late-stage YAML endgame” - hundreds of thousands of lines of markup that have been copy/pasted. Small config changes like upgrading cluster versions require hunting down dozens or hundreds of files. Cloud teams end up feeling hamstrung, spending all of their time in maintenance and unable to invest in infrastructure for new products.

That is why we introduced Pulumi YAML - a simple markup frontend with the power to grow up with your organization. Pulumi YAML is the only infrastructure markup language that supports convert - the ability to automatically translate into any Pulumi Programming language. When you feel that your infrastructure complexity is reaching the endgame, you can pulumi convert to translate your infrastructure in full fidelity into Java, Go, Python, Typescript, or C# in order to tame complexity, and use the power of programming languages to encapsulate, design, and componentize your infrastructure.

Adopting Pulumi up until the introduction of Pulumi YAML required operations teams to learn a programming language such as Python or Javascript. We are yet to see a team that is unsuccessful in this transition, but many have asked for a simple markup language that they could use as an on-ramp. This allows them to start with tools that they know, and incrementally adopt the Pulumi ecosystem of productivity tools including Automation API, Pulumi Deploy, Multi-Language Components, and Policy as Code.

Pulumi YAML is also a powerful interchange format for programmatic tools. Since YAML can be easily interpreted by machines, it is the perfect tool to support custom CLIs, REST APIs and embedding within custom data formats. For example, the Pulumi Kubernetes Operator now uses Pulumi YAML to embed Pulumi programs within a pulumi.com/v1/Program CRD.

Critically, Pulumi YAML is designed to work with Pulumi’s CrossCode technology. You can write a complex component in Go, TypeScript or Python, and expose a simple user interface in YAML - realizing both the power and encapsulation of programming languages with the simplicity of a familiar markup language.

Pulumi YAML 1.0

Pulumi YAML offers a simple YAML document model for defining infrastructure using any of Pulumi’s more than 100 resource provider packages.

This includes 4 top level sections:

resources: A collection of resource definitions, each specifying a name, type, properties and options for a Pulumi resource definition.
outputs: A set of computed output values that will be available as Pulumi stack outputs.
config: The config that each instance of this project can set independent values for.
variables: Give names to complex values that might be needed in the program.

Here is an example Pulumi YAML document, from the AWS Serverless Application template.

name: serverlessapp
runtime: yaml
resources:
 role:
 type: aws:iam:Role
 properties:
 assumeRolePolicy:
 fn::toJSON:
 Version: 2012-10-17
 Statement:
 - Action: sts:AssumeRole
 Effect: Allow
 Principal:
 Service: lambda.amazonaws.com
 managedPolicyArns:
 - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
 fn:
 type: aws:lambda:Function
 properties:
 runtime: python3.9
 handler: handler.handler
 role: ${role.arn}
 code:
 fn::fileArchive: ./function
 api:
 type: aws-apigateway:RestAPI
 properties:
 routes:
 - path: /
 localPath: www
 - path: /date
 method: GET
 eventHandler: ${fn}
outputs:
 url: ${api.url}

Since preview, the config section has been introduced to replace the previous configuration section. This change was made to enable a standard config section in Pulumi.yaml across all Pulumi languages, to enable the new Project-level config. This section has mostly the same set of features as the preview configuration section, but uses more consistent naming for the Pulumi programming model:

config:
 numInstances:
 type: integer
 default: 3
 availabilityZones:
 type: array
 items:
 type: string
 default: [us-east-1-atl-1a, us-east-1-chi-1a]

Full support for `pulumi convert`

An important goal for Pulumi YAML is to be a safe starting point for developing infrastructure, with the ability to smoothly scale up into any other Pulumi language if or when needed. This ensures that using Pulumi YAML won’t limit how far you can scale up your infrastructure. If you need features that Pulumi YAML doesn’t support (control flow, multiple files, component authoring, etc.), or if the complexity of your YAML grows beyond what you want to manage in YAML, just pulumi convert and keep going in another language.

Pulumi Convert now handles all Pulumi languages, and all features of Pulumi YAML. For example, the original example program above can be converted to Python with:

pulumi convert --language python

This generates a Python program defining the exact same resources, along with all required dependencies. A pulumi up on the new program will report no changes, and you can now continue evolving the cloud resources managed by the stack using the new language.

pulumi convert is also useful for creating examples in all Pulumi languages. Authors of Pulumi Component Packages can write examples once in Pulumi YAML, and convert into other languages using pulumi convert. This makes it possible to quickly support great documentation for all Pulumi languages when building and publishing a new component.

IDE Support (Preview)

A critical value of using other Pulumi languages is the tight feedback loop developers get when working inside their IDE. We wanted to bring as much of this benefit as possible to Pulumi YAML as well.

With the new Pulumi YAML LSP (Language Server Protocol), developers can install an extension in their favorite IDE which offers many important IDE features for Pulumi YAML programs. Extensions for VS Code and Emacs are available now, and others will be released in the future. These support the following features:

Error squiggles: Highlight invalid keywords and property names in Pulumi YAML documents.
Hover tips: Hover over types to see rich inline documentation.
Autocomplete: Get hints and autocomplete on property names.

You can see what error squiggles (for a simple type) and hover tips look like in VS Code in this example:

Type Checking and Diagnostics

Good error messages make it significantly easier to quickly identify and fix issues in a Pulumi program. Pulumi YAML now checks programs against the resource schema of the resources being created, and reports early errors with rich information about the location of the error and suggestions on potential fixes.

For example, the same typo from the screenshot above leads to this helpful diagnostic message:

Simplified function syntax

Pulumi offers core support for functions defined in any Pulumi package, and Pulumi YAML programs can use these to be able to express everything available in the Pulumi package ecosystem.

It is now even easier to use these Pulumi functions via the fn intrinsics. There are two classes of functions available in Pulumi YAML:

Built-in Functions: Functions like fn::toJSON and fn::fromBase64 are built-in to Pulumi YAML, and enable common computations on values that come from configuration or other resource outputs.
Package Functions: Functions like fn::azure-native:storage:listStorageAccountKeys and fn::oci:ObjectStorage:getNamespace invoke a function from a Pulumi Package based on its package name, module and function name (fn::<package-name>:<module-name>:<function-name>).

New `str` Package

The new [str] package in the Pulumi Registry offers a suite of string manipulation functions which can be used from Pulumi YAML, as well as in other Pulumi languages. Functions supported so far:

str:replace: Replace occurrences of a substring with another substring.
str:trimSuffix: Trim a suffix from a string.
str:trimPrefix: Trim a prefix from a string.
str:regexp:split: Split a string on a regex.
str:regexp:replace: A regex based replace on a string.

These functions can all be invoked using the simple Package Functions syntax - for example, the following function call replaces bar with buzz in the given resource output:

prop:
 fn::str:replace:
 string: ${res.out}
 old: bar
 new: buzz

Pulumi Kubernetes Operator Inline `Program` Support

The Pulumi Kubernetes Operator recently added support for defining Pulumi resources directly inside Kubernetes YAML using a new pulumi.com/v1/Program CRD.

This support builds uniquely on top of Pulumi YAML, to provide a natural way to describe a set of Pulumi resources inline inside Kubernetes YAML using the Pulumi YAML document format embedded within the program key of a pulumi.com/v1/Program resource.

This allows the Kubernetes resource model to be used to define and evolve any of the tens-of-thousands of cloud resources supported by the Pulumi package ecosystem, and with access to the full breadth of IaC capabilities that Pulumi offers.

apiVersion: pulumi.com/v1
kind: Program
metadata:
 name: staticwebsite
program:
 resources:
 bucket:
 type: aws:s3:Bucket
 properties:
 website:
 indexDocument: index.html
 index.html:
 type: aws:s3:BucketObject
 properties:
 bucket: ${bucket.id}
 content: <h1>Hello, world!</h1>
 contentType: text/html
 acl: public-read
 outputs:
 url: http://${bucket.websiteEndpoint}
---
apiVersion: pulumi.com/v1
kind: Stack
metadata:
 name: staticwebsite
spec:
 stack: lukehoban/staticwebsite/test
 programRef:
 name: staticwebsite
 destroyOnFinalize: true
 config:
 aws:region: us-east-1

Conclusion

The GA of Pulumi YAML marks a stable foundation for the simple, declarative YAML interface to the Pulumi ecosystem. We’re excited to see YAML support expand the audiences who can jump in and use the best of the Pulumi ecosystem, ensuring that Pulumi offers a great experience for every Infrastructure as Code use case and user.

Check out these resources to get started with Pulumi YAML today:

Pulumi+Kubernetes: New Flux Integration and Inline Programs

Luke Hoban — Sun, 23 Oct 2022 00:00:00 +0000

Pulumi’s Universal Infrastructure as Code platform works with all major clouds and over 100 cloud and SaaS providers, but among all its uses one of the most important is the ability to bring rich Infrastructure as Code tools and practices to Kubernetes projects and teams.

Kubernetes is one of the most used platforms in Pulumi, second only to AWS, with thousands of organizations using Pulumi to manage clusters at scale. Pulumi supports a wide variety of use cases around Kubernetes - from cluster creation and management, to rich and expressive workload definition, to continuous delivery and infrastructure GitOps.

Check out version 2.0 of the Pulumi Kubernetes Operator.

Today we are announcing a set of major updates which deepen and extend Pulumi’s support for Kubernetes and the Kubernetes ecosystem. In this post, we’ll highlight a few of these exciting enhancements:

Pulumi Kubernetes Operator v1.10: New integration with Flux for richer GitOps support, and ability to deploy Pulumi stacks from directly within the Kubernetes resource model
New Pulumi Provider for Flux: Manage Flux with Infrastructure as Code
Pulumi Kubernetes Provider v3.22: Server Side Apply and Resource Patch

You can learn more about Pulumi and Kubernetes or jump in and get started right away, or read on to learn more about what’s new for Pulumi and Kubernetes!

Pulumi Kubernetes Operator v1.10: Flux Integration and Inline Infrastructure support

The Pulumi Kubernetes Operator provides a cloud-native way to manage and deploy cloud infrastructure using Pulumi from within your Kubernetes environment. Infrastructure as Code defined in Pulumi can be automatically deployed using a GitOps model driven from within a Kubernetes environment - leveraging existing Kubernetes-based orchestration, RBAC, policy and resource model investments. We released Pulumi Kubernetes Operator 1.0 last year, and since then have seen users scale up usage of the Operator to drive hundreds of thousands of Pulumi infrastructure updates.

Pulumi Kubernetes Operator v1.10 expands the ways that Pulumi infrastructure can be defined. In addition to the existing Git-based source that has been supported to date, this release adds two new ways to define your infrastructure:

Integration with Flux Sources - enabling any Flux Source to provide the Pulumi program that defines your cloud infrastructure.
Support for Inline Infrastructure sources - a new pulumi.com/v1/Program Custom Resource that can be used to describe cloud infrastructure directly within the Kubernetes resource model.

Together, these open up a wide variety of new ways to apply the Kubernetes Operator as an interface for defining and delivering cloud infrastructure.

Integration with Flux Sources

We’re excited to work with Flux, a CNCF project, to bring even richer GitOps support to the Pulumi Kubernetes Operator. While the Pulumi Kubernetes Operator offers basic GitOps and branch tracking support, it supports only a fraction of the features that Flux’s SourceController provides. Instead of rebuilding all of these features, we chose to provide rich integration with Flux Sources, enabling Pulumi programs to be provided via Flux, and then handed off to drive infrastructure deployments using the Pulumi Kubernetes Operator Stack resource.

Using Flux offers a variety of important new features to Pulumi Kubernetes Operator users:

Support for Flux’s Notification Controller and webhooks to drive highly responsive updates when new commits are pushed
Access to sources other than Git repositories: S3 buckets and OCI repositories
Support for platforms like Azure DevOps that require specific Git implementations
Sources and Stacks can be defined by different teams, since they are separate resources which can have independent RBAC rules applied.
Many advanced Git features including: following semver ranges, excluding files from source, recursively cloning submodules, verifying revision signatures, and more.

To use the Pulumi Kubernetes Operator with Flux, install each of the necessary controllers for Flux and the Pulumi Kubernetes Operator, and then the necessary Role and RoleBindings to allow the Pulumi Kubernetes Operator to read from Flux sources.

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
 name: flux-source-access
rules:
- apiGroups:
 - source.toolkit.fluxcd.io
 resources:
 - "*"
 verbs:
 - get
 - list
 - watch
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
 name: pulumi-flux-source-access
subjects:
- kind: ServiceAccount
 name: pulumi-kubernetes-operator
roleRef:
 kind: Role
 name: flux-source-access
 apiGroup: rbac.authorization.k8s.io

Once those pre-requisites have been met, simply deploy the Flux Source (in this case a GitRepository) and Pulumi Stack resource that define your infrastructure delivery:

---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: GitRepository
metadata:
 name: pkodev
spec:
 interval: 30s
 ref:
 branch: main
 url: https://github.com/lukehoban/pko-dev
---
apiVersion: pulumi.com/v1
kind: Stack
metadata:
 name: pkodev
spec:
 stack: lukehoban/pkodev/test
 fluxSource:
 sourceRef:
 apiVersion: source.toolkit.fluxcd.io/v1beta2
 kind: GitRepository
 name: pkodev
 dir: basic/
 destroyOnFinalize: true

Note that by using the new fluxSource property on a pulumi.com/v1/Stack resource, you can configure it to use the corresponding source instead of an inline definition of the target Git repo and branch. The GitRepository could be replaced with any other Flux Source. For example, to pull our source code from an S3 bucket, use this instead:

---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: Bucket
metadata:
 name: pkodev
spec:
 interval: 5m0s
 provider: aws
 bucketName: podinfo
 endpoint: s3.amazonaws.com
 region: us-east-1
 timeout: 30s
---
apiVersion: pulumi.com/v1
kind: Stack
metadata:
 name: pkodev
spec:
 stack: lukehoban/pkodev/test
 fluxSource:
 sourceRef:
 apiVersion: source.toolkit.fluxcd.io/v1beta2
 kind: Bucket
 name: pkodev
 dir: basic/
 destroyOnFinalize: true

If we are using the GitRepository source along with GitHub, we can use Flux features like WebHook Receivers to expose a webhook that GitHub can trigger on pushes so that updates will be pushed into Flux and the Pulumi Kubernetes Operator directly (instead of waiting to pull them):

---
apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Receiver
metadata:
 name: webapp
 namespace: flux-system
spec:
 type: github
 events:
 - "ping"
 - "push"
 secretRef:
 name: webhook-token
 resources:
 - kind: GitRepository
 name: pkodev

For users who need any of these additional GitOps capabilities, you can now use the best of Pulumi and the best of Flux together with ease.

This integration is made possible by the excellent “open and extensible” GitOps toolkit that Flux v2 provides. We’re excited to work with the Flux community and the team at Weaveworks to help end-users accelerate their GitOps projects:

“When we donated Flux to the CNCF we wanted to empower teams to easily build GitOps automations with the best tools. We’re excited to welcome Pulumi to the Flux and Weave GitOps ecosystem.”

– Alexis Richardson, CEO – Weaveworks

Define Pulumi Cloud Infrastructure Directly in the Kubernetes Resource Model

One of the important benefits of the Pulumi Kubernetes Operator is that it provides the Kubernetes resource model as an interface for defining and managing cloud infrastructure expressed via Pulumi. However, to date, the cloud infrastructure definition itself still has to be stored in Git (or, with the Flux integration above, an S3 bucket or OCI repository). Many users have asked for a model where the Pulumi cloud infrastructure definition itself is provided directly within the Kubernetes resource model as well. With the new pulumi.com/v1/Program custom resource, this is now possible!

This support is enabled by Pulumi’s introduction of Pulumi YAML - the ability to describe cloud infrastructure with access to the entire Pulumi ecosystem of cloud providers and components via simple YAML programs. Importantly, those YAML programs can now be defined directly inside the Kubernetes cluster itself.

Because Pulumi offers access to over 100 cloud providers and components, with thousands of infrastructure resources available across them, the pulumi.com/v1/Program resource offers access to defining rich and complex cloud infrastructure directly from within Kubernetes. And because Pulumi YAML infrastructure definitions support all the same reliable dependency tracking and rich IaC features as any other Pulumi programs, users get all of these benefits from within Kubernetes as well.

To deploy cloud infrastructure directly from within Kubernetes, install the Pulumi Kubernetes Operator and then deploy the Program and Stack resources below with kubectl apply (or via a Pulumi program!):

apiVersion: pulumi.com/v1
kind: Program
metadata:
 name: staticwebsite
program:
 resources:
 bucket:
 type: aws:s3:Bucket
 properties:
 website:
 indexDocument: index.html
 index.html:
 type: aws:s3:BucketObject
 properties:
 bucket: ${bucket.id}
 content: <h1>Hello, world!</h1>
 contentType: text/html
 acl: public-read
 outputs:
 url: http://${bucket.websiteEndpoint}
---
apiVersion: pulumi.com/v1
kind: Stack
metadata:
 name: staticwebsite
spec:
 stack: lukehoban/staticwebsite/test
 programRef:
 name: staticwebsite
 destroyOnFinalize: true
 config:
 aws:region: us-east-1

The result will be to deploy an S3 Bucket and BucketObject exposing a static website into AWS. The Program resource defines a Pulumi program written in Pulumi YAML using any cloud provider resources Pulumi supports, ready to be deployed into as many Pulumi stacks as needed. The program declares an S3 Bucket with website configured, and an index.html file in a BucketObject which will be served from that website. The Stack resource deploys an instance of this program, using the configuration (us-east-1 region) specified.

You can access the URL of the deployed website defined as a program output url via:

$ kubectl get stack staticwebsite -o jsonpath={.status.outputs}

Of course, this could be used to define any cloud infrastructure - an AWS RDS Database, an Azure KeyVault, a CloudFlare DNS Record, a Google Cloud Function, or even another Kubernetes Cluster via EKS, AKS, GKE, Oracle Cloud, Civo, Digital Ocean and more.

Updating the Program resource will immediately drive an update to the associated Pulumi Stack, incrementally updating the managed infrastructure and maintaining the desired cloud infrastructure state.

New Pulumi Provider for Flux

In addition to Flux integration with the Pulumi Kubernetes Operator, we’ve also added a new Flux Provider to the Pulumi Registry. This provider, built by Pulumi community member @oun, enables managing Flux directly via Pulumi Infrastructure as Code.

The Flux provider can be used along with the Kubernetes provider and GitHub provider to stand up and manage a complete E2E GitOps solution from within a single deployment - no need for manual bash, Helm, or runbooks - just simple reliable Infrastructure as Code.

import * as kubernetes from "@pulumi/kubernetes";
import * as flux from "@worawat/flux";

// Get the manifests needed to install Flux with the given components and policies
const fluxManifests = flux.getFluxInstallOutput({
 targetPath: "local",
 components: ['source-controller'],
 networkPolicy: true,
});

// Deploy the manifests into our cluster
const fluxGroup = fluxManifests.content.apply(fluxYaml => {
 new kubernetes.yaml.ConfigGroup("flux-install", {
 yaml: fluxYaml,
 });
});

Pulumi Kubernetes Provider v3.22: Server Side Apply and Resource Patch

In July, we released preview support for Server-Side Apply in the Pulumi Kubernetes provider. We highlighted the benefits and needs that led to adding Server-Side Apply support:

We first launched the Pulumi Kubernetes provider in 2018, and it has grown to be one of the most used providers across the Pulumi ecosystem. Many of our customers rely on this provider to manage critical production workloads, and a common request was for an easier way to manage shared Kubernetes resources with Pulumi. We’ve been listening, and are grateful to everyone who took the time to share their feedback with us!

A lot has changed since 2018, and our provider continues to evolve to support new use cases and the ever-growing richness of the Kubernetes ecosystem. One such development is Server-Side Apply (SSA), which is a resource management strategy that was introduced in Kubernetes v1.18. Kubernetes clients that use SSA can safely share the management of Kubernetes resources by making the API Server responsible for computing diffs and resolving conflicts.

We have kept a close eye on the SSA feature since it was announced and have been waiting patiently for its broad availability in production clusters. The wait is over, and we are excited to bring the power of SSA to Pulumi users!

With the most recent v3.22 release of the Kubernetes provider, Server-Side Apply support is now out of preview and enabled by default for all Kubernetes resources.

This makes several important new features available across the breadth of the Kubernetes provider:

New Patch resource types corresponding to every Kubernetes resource kind. (e.g., NamespacePatch, DeploymentPatch)
“Upsert” support; create a resource if it does not exist, or update the existing resource with specified changes.
Diffs no longer depend on the last-applied-configuration annotation, which fixes a number of subtle issues that users of the provider previously could run into.

These features bring Pulumi and Kubernetes even closer together, enabling Pulumi’s rich Infrastructure as Code to fit even more naturally into the Kubernetes desired state and shared ownership model.

Check out the Managing Resources with Server Side Apply article for more details on these new features.

Conclusion

At Pulumi, we are passionate about pushing the state of the art of Infrastructure as Code forward, and applying modern Infrastructure as Code to support Kubernetes projects and teams is a critical component of that vision. All of these new features around Pulumi and Kubernetes help to further empower teams to build scalable and robust solutions around Kubernetes and Pulumi IaC.

We’re so happy to be able to work with the Flux project to bring all of the benefits of its composable GitOps toolkit to Pulumi users. We’re excited about the new use cases that Inline Infrastructure support in the Pulumi Kubernetes Operator enables. And we’re thrilled to bring Server Side Apply and Patch support to the Pulumi Kubernetes Provider.

You can learn more about Pulumi and Kubernetes or jump in right away by following the getting started guide.

Pulumi Universal IaC: New Support For Java, YAML and AWS CDK

Luke Hoban — Wed, 04 May 2022 07:00:00 -0700

Some of the code in this post is out of date. See the AWS guides for an updated overview and examples.

Over the last year since the launch of Pulumi 3.0, we’ve seen incredible growth in adoption and usage of the Pulumi open source project and Cloud Engineering platform, with more than a thousand new open source contributors, tens of thousands of new users, and millions of new cloud infrastructure resources being managed by Pulumi. Pulumi’s infrastructure as code tools are enabling teams to scale up their cloud infrastructure with robust software engineering tools and practices to get the most value out of their cloud platform investments.

Today, we’re excited to announce a wave of innovation across the Pulumi project with a collection of significant new feature launches. These new features bring together Pulumi’s Universal Infrastructure as Code offering, supporting the widest range of builders, clouds, programming languages, and cloud architectures.

Today’s launches include:

Pulumi support for Java: New support for Java across the entire Pulumi platform, providing full access to the rich software engineering ecosystem for Java and the JVM.
Pulumi YAML: A new simple YAML interface to the Pulumi platform, broadening the reach of Pulumi to use cases where a full blown programming language is not (yet!) necessary.
Crosswalk for AWS in all Pulumi Languages: Support for the popular Crosswalk for AWS packages (for ECS, VPC, API Gateway and EKS) in all Pulumi languages.
AWS CDK on Pulumi: Support for deploying AWS CDK constructs from within Pulumi infrastructure as code programs, enabling access to the broad array of higher-level components for AWS built by teams at AWS and the AWS CDK community.
New Cloud and SaaS Providers: More than a dozen new packages in the Pulumi Registry for popular cloud/SaaS platforms from Pulumi partners, including Oracle Cloud, Elastic Cloud and Databricks.
Pulumi Service Provider for Pulumi: A new provider for managing the Pulumi Service using Pulumi Infrastructure as Code.
Pulumi CrossCode: Key improvements in the core technology powering Pulumi’s universal infrastructure as code features, including pulumi import, pulumi convert and more.

These new features build on the Pulumi 3.0 foundation to offer a truly universal infrastructure as code platform for modern cloud engineers and teams.

Pulumi Support For Java

Java is one of the most used programming languages in the world, with heavy usage in enterprises and organizations building for the cloud. The Java ecosystem has played a leading role in offering an incredibly rich ecosystem of software engineering capabilities - IDE tooling, strong typing, design patterns, build and package management, unit testing, and more.

Pulumi now supports Java (in addition to existing support for TypeScript, Python, Go and .NET) and other languages targeting the JVM. All the existing Pulumi CLI workflows can now be used with Java, and all packages in the Pulumi Registry are now available as Java libraries in Maven Central.

You can get started with Pulumi using Java today in the Pulumi Getting Started guide and the Pulumi and Java language docs.

👉 Read more in the Pulumi and Java blog post.

Pulumi YAML

Pulumi has differentiated itself as an Infrastructure as Code tool that embraces popular programming languages and their rich software engineering ecosystems. However, our goal has always been to offer the broadest range of language support for the wide variety of use cases and users who need to interact with cloud infrastructure in modern cloud organizations.

Pulumi YAML offers a new, simple interface to the full richness of the Pulumi Platform. Pulumi YAML allows infrastructure as code to be authored in a single simple YAML file with just a few lines of YAML, immediately familiar to any engineer, operator or developer.

name: yamldemo
runtime: yaml
resources:
 bucket:
 type: aws:s3:Bucket
 properties:
 website:
 indexDocument: index.html
 index.html:
 type: aws:s3:BucketObject
 properties:
 bucket: ${bucket.id}
 content: <h1>Hello, world!</h1>
 contentType: text/html
 acl: public-read
outputs:
 bucketName: ${bucket.id}

Like all other Pulumi languages, Pulumi YAML programs have access to all the core features of Pulumi’s infrastructure as code tooling, including native providers, secrets management, stack references, Pulumi Packages, and all the features of the Pulumi Service. YAML programs can interoperate seamlessly with the rest of the Pulumi ecosystem, consuming the outputs of other Pulumi programs and using Pulumi components built in existing Pulumi languages. For example, a complete EKS cluster can be configured using the EKS package (implemented in TypeScript):

name: eks-cluster
runtime: yaml
resources:
 cluster:
 type: eks:Cluster
outputs:
 kubeconfig: ${cluster.kubeconfig}

Access to Pulumi packages authored in other Pulumi languages means complexity can be pushed into reusable infrastructure components instead of copy-pasted in YAML. And when complexity does increase, Pulumi YAML supports a new pulumi convert command to eject into any other Pulumi programming language to manage the increased complexity via rich software engineering features in those languages.

You can get started with Pulumi YAML today in the Pulumi Getting Started guide and the Pulumi YAML language docs.

👉 Read more in the Pulumi YAML blog post.

Crosswalk for AWS in All Pulumi Languages

Crosswalk for AWS is a collection of libraries that use automatic well-architected best practices to make common infrastructure as code tasks in AWS easier and more secure. This includes libraries for EKS and API Gateway as well as the AWSX library that provides VPC, ECS, LoadBalancing and more. All of these libraries are now available in the Pulumi Registry for all Pulumi languages. The AWSX library, which was previously only available for Pulumi TypeScript users has become one of the most used libraries in the Pulumi ecosystem. With today’s release, it is now available to the entirety of the Pulumi language ecosystem - including Java and YAML!

For example - to run a Docker container in ECS using default network and cluster settings, users can use the awsx.ecs.FargateService class along with an Application Load Balancer listening on port 80 at a stable address - with just a few lines of code:

import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";

const cluster = new aws.ecs.Cluster("default-cluster");

const lb = new awsx.lb.ApplicationLoadBalancer("nginx-lb");

const service = new awsx.ecs.FargateService("my-service", {
 cluster: cluster.arn,
 desiredCount: 2,
 taskDefinitionArgs: {
 container: {
 image: "nginx:latest",
 cpu: 512,
 memory: 128,
 essential: true,
 portMappings: [
 {
 containerPort: 80,
 targetGroup: lb.defaultTargetGroup,
 },
 ],
 },
 },
});

export const url = lb.loadBalancer.dnsName;

import pulumi
import pulumi_aws as aws
import pulumi_awsx as awsx

cluster = aws.ecs.Cluster("default-cluster")

lb = awsx.lb.ApplicationLoadBalancer("nginx-lb")

service = awsx.ecs.FargateService("my-service",
 cluster=cluster.arn,
 desired_count=2,
 task_definition_args=awsx.ecs.FargateServiceTaskDefinitionArgs(
 container=awsx.ecs.TaskDefinitionContainerDefinitionArgs(
 image="nginx:latest",
 cpu=512,
 memory=128,
 essential=True,
 port_mappings=[awsx.ecs.TaskDefinitionPortMappingArgs(
 target_group=lb.default_target_group
 )],
 )
 )
)

package main

import (
 "github.com/pulumi/pulumi-aws/sdk/v5/go/aws/ecs"
 "github.com/pulumi/pulumi-aws/sdk/v5/go/aws/lb"
 ecsx "github.com/pulumi/pulumi-awsx/sdk/go/awsx/ecs"
 lbx "github.com/pulumi/pulumi-awsx/sdk/go/awsx/lb"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {

 cluster, err := ecs.NewCluster(ctx, "default-cluster", &ecs.ClusterArgs{})
 if err != nil {
 return err
 }

 loadBalancer, err := lbx.NewApplicationLoadBalancer(ctx, "nginx-lb", &lbx.ApplicationLoadBalancerArgs{})
 if err != nil {
 return err
 }

 _, err = ecsx.NewFargateService(ctx, "my-service", &ecsx.FargateServiceArgs{
 Cluster: cluster.Arn,
 DesiredCount: pulumi.Int(2),
 TaskDefinitionArgs: &ecsx.FargateServiceTaskDefinitionArgs{
 Container: &ecsx.TaskDefinitionContainerDefinitionArgs{
 Image: pulumi.String("nginx:latest"),
 Cpu: pulumi.Int(512),
 Memory: pulumi.Int(128),
 Essential: pulumi.Bool(true),
 PortMappings: ecsx.TaskDefinitionPortMappingArray{
 ecsx.TaskDefinitionPortMappingArgs{
 ContainerPort: pulumi.Int(80),
 TargetGroup: loadBalancer.DefaultTargetGroup,
 },
 },
 },
 },
 })
 if err != nil {
 return err
 }

 ctx.Export("url", loadBalancer.LoadBalancer.ApplyT(func(alb *lb.LoadBalancer) interface{} {
 return alb.DnsName
 }))
 return nil
 })
}

var cluster = new Aws.Ecs.Cluster("default-cluster");
var lb = new Awsx.Lb.ApplicationLoadBalancer("nginx-lb");
var service = new Awsx.Ecs.FargateService("my-service", new Awsx.Ecs.FargateServiceArgs
{
 Cluster = cluster.Arn,
 DesiredCount = 2,
 TaskDefinitionArgs = new Awsx.Ecs.Inputs.FargateServiceTaskDefinitionArgs
 {
 Container = new Awsx.Ecs.Inputs.TaskDefinitionContainerDefinitionArgs
 {
 Image = "nginx:latest",
 Cpu = 512,
 Memory = 128,
 Essential = true,
 PortMappings = {new Awsx.Ecs.Inputs.TaskDefinitionPortMappingArgs
 {
 TargetGroup = lb.DefaultTargetGroup,
 }},
 }
 }
});
this.Url = lb.LoadBalancer.Apply(lb => lb.DnsName);

name: awsx
runtime: yaml
resources:
 cluster:
 type: aws:ecs:Cluster
 lb:
 type: awsx:lb:ApplicationLoadBalancer
 nginx:
 type: awsx:ecs:FargateService
 properties:
 cluster: ${cluster.arn}
 taskDefinitionArgs:
 container:
 image: nginx:latest
 cpu: 512
 memory: 128
 portMappings:
 - containerPort: 80
 targetGroup: ${lb.defaultTargetGroup}
outputs:
 url: ${lb.loadBalancer.dnsName}

You can get started with Crosswalk for AWS in the Pulumi Crosswalk for AWS Guides, now available with examples in all languages.

👉 Read more in the Pulumi Crosswalk for AWS support for all Pulumi Languages blog post.

AWS CDK on Pulumi

With the new AWS CDK on Pulumi project, available in public preview today, we are opening up the ability to use AWS CDK constructs from within a Pulumi deployment. For developers already using AWS CDK, this provides Pulumi as a new option for orchestrating deployments in place of CloudFormation, offering improved deployment speed, integration with the full set of features of the Pulumi Cloud Engineering Platform (like Policy as Code, Audit Logs, Secrets, and much more). But even better, you can also now combine AWS CDK and Pulumi resources in a single Pulumi infrastructure as code project - passing outputs from Pulumi resources into AWS CDK constructs, and outputs from AWS CDK constructs into Pulumi resources. This allows you to work across the >80 cloud and SaaS providers that Pulumi offers access to, while still benefiting from high level libraries from the AWS CDK project.

import * as fs from 'fs';
import * as aws_events from 'aws-cdk-lib/aws-events';
import * as aws_events_targets from 'aws-cdk-lib/aws-events-targets';
import * as aws_lambda from 'aws-cdk-lib/aws-lambda';
import { CfnOutput, Duration } from 'aws-cdk-lib';
import * as pulumi from '@pulumi/pulumi';
import * as pulumicdk from '@pulumi/cdk';
import { Construct } from 'constructs';
import { remapCloudControlResource } from './adapter';

// Use Pulumi to construct an inline Lambda function
const lambdaFn = new aws.lambda.CallbackFunction('lambda', {
 callback: (ev, ctx) => console.log(ev),
});

class LambdaStack extends pulumicdk.Stack {
 lambdaArn: pulumi.Output<string>;

 constructor(id: string, options?: pulumicdk.StackOptions) {
 super(id, { ...options, remapCloudControlResource });

 // Use the AWS CDK Rule API directly.
 const rule = new aws_events.Rule(this, 'rule', {
 // Run 6:00 PM UTC every Monday through Friday
 schedule: aws_events.Schedule.expression('cron(0 18 ? * MON-FRI *)'),
 });

 const fn = aws_lambda.Function.fromFunctionArn(this, 'lambdaFn', pulumicdk.asString(lambdaFn.arn))

 // Use the AWS CDK to add a Rule target to trigger the Function.
 rule.addTarget(new aws_events_targets.LambdaFunction(lambdaFn));

 // Export the Lambda function's ARN as an output.
 this.lambdaArn = this.asOutput(lambdaFn.functionArn);

 this.synth();
 }
}

// Construct the CDK stack as a Pulumi component.
const stack = new LambdaStack('teststack');

// Use an output from the CDK construct from Pulumi.
export const lambdaArn = stack.lambdaArn;

This can be deployed with the pulumi CLI just like any other Pulumi program.

> pulumi up
Updating (dev)
View Live: https://app.pulumi.com/lukehoban/cdk-cron-lambda/dev/updates/37
Type Name Status
+ pulumi:pulumi:Stack cdk-cron-lambda-dev created
+ └─ cdk:index:Stack teststack created
+ └─ cdk:construct:LambdaStack teststack/teststack created
+ ├─ cdk:construct:Function teststack/teststack/lambda created
+ │ ├─ cdk:construct:Role teststack/teststack/lambda/ServiceRole created
+ │ │ └─ aws-native:iam:Role lambdaServiceRole494E4CA6 created
+ │ └─ aws-native:lambda:Function lambda8B5974B5 created
+ └─ cdk:construct:Rule teststack/teststack/rule created
+ ├─ aws:cloudwatch:EventRule ruleF2C1DCDC created
+ ├─ aws:cloudwatch:EventTarget Target0 created
+ └─ aws:lambda:Permission ruleAllowEventRuleteststacklambda47768855D6EFF36B created
Resources:
+ 11 created
Duration: 48s

AWS CDK on Pulumi is available for TypeScript today in public preview.

👉 Read more in the AWS CDK on Pulumi blog post.

New Cloud and SaaS Providers

Since launching the Pulumi Registry last fall, we’ve added more than a dozen new packages, supporting a variety of new cloud and SaaS providers.

Oracle Cloud is the most recent major new cloud provider, offering access to the entire surface area of Oracle’s Cloud platform. In addition, new providers for Elastic Cloud, DataBricks, jFrog Artifactory, Event Store Cloud, Scaleway, Slack, TailScale, Twingate, Checkly, Buildkite and more continue to expand the long tail of cloud and SaaS platforms accessible from Pulumi.

Every package in the Pulumi Registry supports all Pulumi programming languages, including Java and YAML. Documentation and getting started guides for every package are available from the Registry. Contributions to the Registry are now simpler than ever, with enhanced guides for authoring and publishing your own package in the Pulumi Registry.

New Provider for the Pulumi Service

Pulumi’s users love managing all of their cloud infrastructure using Pulumi, and that includes managing the state of the Pulumi Service itself.

Today, we released a new Pulumi provider for the Pulumi Service, supporting configuration of Teams, Access Tokens, Stack Tags and Webhooks using infrastructure as code.

For example, you can create a webhook that is notified whenever a Pulumi Update completes, and log it using an AWS Lambda, with just a few lines of code using the new Pulumi Service provider along with the API Gateway package.

import * as awsx from "@pulumi/awsx";
import * as pulumiservice from "@pulumi/pulumi-service";

const api = new awsx.apigateway.API("api", {
 routes: [{ path: "/", method: "POST", eventHandler: async (ev) => {
 const body = Buffer.from(ev.body!, 'base64').toString("ascii")
 console.log(JSON.stringify(body));
 return { statusCode: 200, body: "" };
 }}],
});

const webhook = new pulumiservice.Webhook("webhook", {
 active: true,
 organizationName: "upstarts",
 displayName: "pulumipulumi",
 payloadUrl: api.url,
});

👉 Read more in the Pulumi Provider for the Pulumi Service blog post.

Pulumi CrossCode

Pulumi CrossCode is the underlying set of technologies that Pulumi uses to support a wide range of programming languages, to enable sharing components and packages between languages, to support converting infrastructure from existing formats into Pulumi, to import infrastructure directly from cloud providers into Pulumi code, and to support converting Pulumi YAML into other Pulumi languages.

Support for Java and YAML was built on (and into) CrossCode to enable these languages to leverage everything that exists in the Pulumi ecosystem and to easily plug into existing features like conversion tools, example generation and so much more value that exists in the Pulumi platform.

In addition, recent improvements to the pulumi import command were enabled by CrossCode - making it possible to generate Pulumi Programs from existing cloud infrastructure.

And of course, the new pulumi convert command is powered by the existing features of CrossCode, opening up new paths for adoption and scaling of Pulumi projects.

Learn more about the technology underlying CrossCode in GitHub.

Summary

Pulumi’s Universal Infrastructure as Code platform supports the widest range of builders, clouds, programming languages, and cloud architectures available today.

With support for a broad range of languages - from the enterprise software engineering features of Java, to the simple declarative interface of Pulumi YAML - Pulumi scales from the simplest cloud use cases up into the most complex cloud infrastructure being managed today, and bridges the gap between cloud platform teams, operations teams, and application development teams.

With support for more than 80 cloud and SaaS providers, Pulumi can be used across multi-cloud workloads, and to coordinate the entire range of cloud infrastructure and managed services that define your cloud application - not just a single cloud provider. And with high level components for AWS from both Pulumi Crosswalk for AWS and AWS CDK on Pulumi, developers working in AWS have the richest set of high level infrastructure building blocks available for productively composing well-architected cloud infrastructure on AWS.

To learn more about each of the new features - check out these blog posts with more details:

We’re incredibly excited about today’s launches, but also see an amazing opportunity ahead to continue to drive new innovation in Infrastructure as Code and the future of Cloud Engineering. Can’t wait to see what you build!

Pulumi YAML: A Simple Declarative Interface for Pulumi

Luke Hoban — Wed, 04 May 2022 06:58:00 -0700

Since we first launched Pulumi 4 years ago, a core point of differentiation between Pulumi and other Infrastructure as Code offerings has been the ability to use popular general purpose programming languages - and their rich software engineering ecosystems - in order to scale up the complexity and richness of cloud infrastructure workloads. This approach has enabled cloud builders to adopt and embrace modern Infrastructure as Code with Pulumi using a wide variety of languages, including TypeScript, Python, Go, C# and Java.

Our goal though has always been to offer the broadest range of programming language options to empower every cloud builder so that they could benefit from the best of Pulumi’s Infrastructure as Code platform.

Today, we are excited to launch Pulumi YAML, a simple YAML-based interface to the entirety of the Pulumi Infrastructure as Code platform.

Pulumi YAML is a great option for smaller scale cloud infrastructure use cases. This includes:

Teams that don’t (yet) need the richness of a general purpose programming language, and feel more comfortable with YAML.
Simple infrastructure definitions which only need to compose a small number of cloud building blocks together.
Providing a simple interface to a complex set of high-level cloud components built with existing general-purpose Pulumi languages.
Systems which will generate infrastructure as code definitions programmatically – such as self-service provisioning portals.

Like all other Pulumi languages, Pulumi YAML programs have access to all the core features of Pulumi’s infrastructure-as-code tooling, including native providers, secrets management, stack references, Pulumi Packages, and all the features of the Pulumi Service. Critically, Pulumi YAML programs can interoperate seamlessly with the rest of the Pulumi ecosystem, consuming the outputs of other Pulumi programs and using Pulumi components built in existing Pulumi languages. If the complexity of your Pulumi YAML projects increases, the new pulumi convert feature can be used to convert your Pulumi YAML program into your favorite general purpose Pulumi language.

Watch our recent workshop: Introduction to Infrastructure as Code with YAML. In this workshop you’ll learn how to build, configure and deploy a modern application using Docker.

Pulumi YAML

Pulumi YAML programs are defined in the same Pulumi.yaml file that all Pulumi projects contain. Instead of specifying a runtime like nodejs or python, Pulumi YAML programs use the yaml runtime. This allows them to also include the set of resources and outputs that their infrastructure requires. All that is needed is the single Pulumi.yaml file.

To get started with Pulumi YAML, run the pulumi new <cloud>-yaml command – templates are available for all of the most popular cloud providers.

The following example shows a Pulumi YAML program that deploys two AWS resources, an AWS S3 Bucket and an S3 Bucket Object to create in that bucket. The program outputs the name of the bucket for use in external tooling.

name: yamldemo
runtime: yaml
resources:
 bucket:
 type: aws:s3:Bucket
 properties:
 website:
 indexDocument: index.html
 index.html:
 type: aws:s3:BucketObject
 properties:
 bucket: ${bucket.id}
 content: <h1>Hello, world!</h1>
 contentType: text/html
 acl: public-read
outputs:
 url: http://${bucket.websiteEndpoint}

This can be deployed with the same familiar pulumi CLI experience - including previews, stacks, configuration and more:

> pulumi up
Updating (dev)
View Live: https://app.pulumi.com/lukehoban/yamldemo/dev/updates/32
Type Name Status
+ pulumi:pulumi:Stack yamldemo-dev created
+ ├─ aws:s3:Bucket bucket created
+ └─ aws:s3:BucketObject index.html created
Outputs:
url: "http://bucket-801e3f9.s3-website-ap-southeast-2.amazonaws.com"
Resources:
+ 3 created
Duration: 8s

And outputs can be used just as in any other Pulumi project:

> curl $(pulumi stack output url)
<h1>Hello, world!</h1>

Why YAML?

Pulumi already supports rich general purpose programming languages, so our goal with Pulumi YAML was to support the other extreme – to be the simplest, most broadly understandable and recognizable, and easiest to generate option for specifying cloud infrastructure. We didn’t want to invent a new proprietary DSL, instead we wanted to support a widely recognized existing markup language.

YAML is the lingua franca of so many existing cloud technologies in use today – from Kubernetes and CloudFormation, to Docker Compose and Serverless Framework. Pulumi YAML takes inspiration from all of these, with a simple document model and direct expression of the Pulumi resource model.

The one key feature of Pulumi YAML is that all property values are implicitly interpolations. This means that any property value can include ${...} to use the output of one resource as part of the input to another resource.

Using YAML does lead to some constraints. There are no loops, no conditionals, no unit testing, and (currently) no rich IDE tooling. All of these are things you can get from any other Pulumi language. But Pulumi YAML explicitly targets simple use cases where these are not needed and avoids the additional complexity of supporting these features.

Use Components Built in Other Pulumi Languages

Pulumi YAML programs can use any package and any resource defined in the Pulumi Registry. This includes 80+ cloud and SaaS providers, comprising tens of thousands of cloud infrastructure resources. Programs can combine resources from multiple providers – such as using AWS, Docker, PostgreSQL and Cloudflare resources from a single infrastructure definition – to describe the complete provisioning of a cloud application’s infrastructure.

Pulumi YAML programs can also use any Pulumi component package defined in another Pulumi language. Pulumi component packages are reusable infrastructure components built in one Pulumi language, and exposed via Pulumi Schema to all other Pulumi languages - including Pulumi YAML.

For example, a complete EKS cluster can be configured using the EKS package (implemented in TypeScript):

name: eks-cluster
runtime: yaml
resources:
 cluster:
 type: eks:Cluster
outputs:
 kubeconfig: ${cluster.kubeconfig}

Organizations can also build their own reusable components in existing Pulumi languages like TypeScript, Python and Go to capture key aspects of their internal design patterns and compliance criteria, and make those available to other developers within the organization to consume from Pulumi YAML. For example, a hypothetical ACMECorp organization could create a high-level serverless application framework built in Go or TypeScript, which would support deploying serverless apps with just a few lines of YAML (note that this results in something similar to Serverless Framework, but as just a simple library defined in Pulumi):

name: serverless-simple-yaml
resources:
 receiverQueue:
 type: aws:sqs:Queue
 svc:
 type: acmecorp:serverless:Service
 properties:
 provider:
 name: aws
 runtime: nodejs10.x
 iam:
 role:
 managedPolicies:
 - arn:aws:iam::aws:policy/AmazonSQSFullAccess
 functions:
 sender:
 handler: handler.sender
 environment:
 QUEUE_NAME: ${receiverQueue.name}
 events:
 - http:
 method: post
 path: sender
 receiver:
 handler: handler.receiver
 events:
 - sqs:
 arn: ${receiverQueue.arn}

Kubernetes Resources

Kubernetes resources are a very common place to use YAML in practice today, and because Pulumi’s Kubernetes provider is a native provider that projects the exact same API as the Kubernetes API Server, this means you can in most cases just copy/paste Kubernetes YAML into a Pulumi YAML program to deploy the Kubernetes resources using the Pulumi deployment engine. For example, everything nested inside properties below is directly copy-pasted from a Kubernetes YAML specification.

name: yamldemo
runtime: yaml
resources:
 pod:
 type: kubernetes:core/v1:Pod
 properties:
 apiVersion: v1
 kind: Pod
 metadata:
 name: debug-network-pod
 spec:
 containers:
 - command:
 - sleep
 - "3600"
 image: praqma/network-multitool
 name: debug-network-container

This allows you to then opt-in to additional features of Pulumi - like passing outputs of one resource into another, waiting for resource readiness before completing an update, passing secret configuration into a deployment, or mixing Kubernetes resources with other cloud infrastructure specifications, such as cloud-native data stores or IAM permissions.

Full Features of the Pulumi Platform

Although most use cases for Pulumi YAML are for simple programs, it’s possible to describe larger collections of infrastructure in Pulumi YAML, using all of the features of the Pulumi Platform.

The following example highlights the use of a variety of Pulumi features via Pulumi YAML to deploy a complete Azure Container Apps application with Pulumi:

Config: Specify the Pulumi configuration parameters and their defaults
Variables: Use Pulumi function invokes to get data from the cloud provider for use in constructing resources.
Providers: Create resource providers to connect to specific instances of a target cloud, like an Azure account or in this case a Docker registry.
Resource Options: Pass any Pulumi Resource Option to a resource definition, to define replacement behavior, resource protection, secret outputs and more.
Multiple Providers: Pass outputs from Azure resources into Docker resources, and outputs from Docker resources back into Azure resources to orchestrate a complete application deployment.

name: azure-container-apps
runtime: yaml
description: Azure Container Apps example
configuration:
 sqlAdmin:
 type: String
 default: pulumi
variables:
 sharedKey:
 Fn::Invoke:
 Function: azure-native:operationalinsights:getSharedKeys
 Arguments:
 resourceGroupName: ${resourceGroup.name}
 workspaceName: ${workspace.name}
 Return: primarySharedKey
 adminUsername:
 Fn::Invoke:
 Function: azure-native:containerregistry:listRegistryCredentials
 Arguments:
 resourceGroupName: ${resourceGroup.name}
 registryName: ${registry.name}
 Return: username
 adminPasswords:
 Fn::Invoke:
 Function: azure-native:containerregistry:listRegistryCredentials
 Arguments:
 resourceGroupName: ${resourceGroup.name}
 registryName: ${registry.name}
 Return: passwords
resources:
 resourceGroup:
 type: azure-native:resources:ResourceGroup
 workspace:
 type: azure-native:operationalinsights:Workspace
 properties:
 resourceGroupName: ${resourceGroup.name}
 sku:
 name: "PerGB2018"
 retentionInDays: 30
 kubeEnv:
 type: azure-native:web:KubeEnvironment
 properties:
 resourceGroupName: ${resourceGroup.name}
 environmentType: Managed
 appLogsConfiguration:
 destination: "log-analytics"
 logAnalyticsConfiguration:
 customerId: ${workspace.customerId}
 sharedKey: ${sharedKey}
 registry:
 type: azure-native:containerregistry:Registry
 properties:
 resourceGroupName: ${resourceGroup.name}
 sku:
 name: "Basic"
 adminUserEnabled: true
 provider:
 type: pulumi:providers:docker
 properties:
 registryAuth:
 - address: ${registry.loginServer}
 username: ${adminUsername}
 password: ${adminPasswords[0].value}
 myImage:
 type: docker:RegistryImage
 properties:
 name: ${registry.loginServer}/node-app:v1.0.0
 build:
 context: ${pulumi.cwd}/node-app
 options:
 provider: ${provider}
 containerapp:
 type: azure-native:web:ContainerApp
 properties:
 resourceGroupName: ${resourceGroup.name}
 kubeEnvironmentId: ${kubeEnv}
 configuration:
 ingress:
 external: true
 targetPort: 80
 registries:
 - server: ${registry.loginServer}
 username: ${adminUsername}
 passwordSecretRef: "pwd"
 secrets:
 - name: "pwd"
 value: ${adminPasswords[0].value}
 template:
 containers:
 - name: "myapp"
 image: ${myImage.name}
outputs:
 endpoint: https://${containerapp.configuration.ingress.fqdn}

Full Support for the Pulumi Service

For users of the Pulumi Service, including the Individual, Team, Enterprise and Business Critical offerings, all of the features you rely on from the Pulumi Service are available for your Pulumi YAML programs as well.

Your Pulumi YAML projects are shown along with the rest of your Pulumi projects authored in other languages, and each Pulumi YAML project can have multiple stacks.

Policy as Code written in any Pulumi language can be applied to your Pulumi YAML programs, and resources provisioned with Pulumi YAML are deep linked to the corresponding resources in your cloud provider. And most fundamentally, your infrastructure state is managed completely transparently.

YAML as a Compilation Target and Cue Support

Pulumi’s YAML runtime supports both YAML and JSON natively. Both of these formats are easy targets for machine generation of infrastructure specifications, which opens up a wide range of additional interesting use cases for Pulumi YAML. For example, early users have built web-based provisioning portals that generate Pulumi YAML programs on the fly to deploy with Pulumi Automation API.

This also means that the rich set of languages that are designed to compile down to YAML/JSON can be used directly with Pulumi YAML. Teams that prefer to use Cue, Jsonnet, HCL or any other DSL that compiles to YAML/JSON can be used with Pulumi YAML.

For example, this Cue program defines a collection of Azure cloud infrastructure resources using Pulumi YAML.

In Pulumi.yaml:

name: cuedemo
runtime:
 name: yaml
 options:
 compiler: cue export

In main.cue:

package main

resources: {
 (#StaticWebApp & {
 _name: "bob"
 }).resources

 (#StaticWebApp & {
 _name: "anna"
 _documents: {
 root: "website"
 index: "index.html"
 }
 }).resources
}

#StaticWebApp: {
 _name: string
 _documents: {
 root: string | *"website"
 index: string | *"index.html"
 error404: string | *"404.html"
 }

 resources: {
 "\(_name)-group":
 type: "azure-native:resources:ResourceGroup"

 "\(_name)-storage": {
 type: "azure-native:storage:StorageAccount"
 properties: {
 resourceGroupName: "${\(_name)-group.name}"
 kind: "StorageV2"
 sku: {
 name: "Standard_LRS"
 }
 }
 }

 "\(_name)-website": {
 type: "azure-native:storage:StorageAccountStaticWebsite"
 properties: {
 resourceGroupName: "${\(_name)-group.name}"
 accountName: "${\(_name)-storage.name}"
 indexDocument: "\(_documents.index)"
 error404Document: "\(_documents.error404)"
 }
 }

 "\(_name)-\(_documents.index)": {
 type: "azure-native:storage:Blob"
 properties: {
 resourceGroupName: "${\(_name)-group.name}"
 accountName: "${\(_name)-storage.name}"
 containerName: "${\(_name)-website.containerName}"
 contentType: "text/html"
 type: "Block"
 source: {
 "Fn::FileAsset": "./\(_documents.root)/\(_documents.index)"
 }
 }
 }
 }
}

Running pulumi up will now automatically compile the Cue into JSON and deploy it - no need to manually run a transpiler.

> pulumi up
Previewing update (dev)
View Live: https://app.pulumi.com/lukehoban/cuedemo/dev/previews/7ac07d0c-262e-41c8-a2c7-6d3f2734cf95
Type Name Plan
+ pulumi:pulumi:Stack cuedemo-dev create
+ ├─ azure-native:resources:ResourceGroup bob-group create
+ ├─ azure-native:resources:ResourceGroup anna-group create
+ ├─ azure-native:storage:StorageAccount anna-storage create
+ ├─ azure-native:storage:StorageAccount bob-storage create
+ ├─ azure-native:storage:StorageAccountStaticWebsite anna-website create
+ ├─ azure-native:storage:StorageAccountStaticWebsite bob-website create
+ ├─ azure-native:storage:Blob anna-index.html create
+ └─ azure-native:storage:Blob bob-index.html create
Resources:
+ 9 to create

Deploy infrastructure with Pulumi YAML

Define your cloud resources in a single Pulumi.yaml file, combine multiple providers, and reuse components built in other Pulumi languages.

Get started

Convert to Other Pulumi Languages

When the complexity of infrastructure managed by a Pulumi YAML program grows beyond what can be managed easily with YAML, users have a variety of options to “grow up” without disruption to their infrastructure.

First and foremost, complexity can be pushed into a reusable component defined in another Pulumi language, and then used from Pulumi YAML, to keep the interface simple and clean for Pulumi YAML consumers.

But for cases where the YAML program itself does need to get more complex, using languages like Cue or Jsonnet provides an easy path, similar to what is frequently found in the Kubernetes ecosystem as complexity increases. This often introduces expressivity limitations and slightly cumbersome templating as you continue to scale through.

So to give the best of both worlds, Pulumi YAML also supports the option to convert your Pulumi YAML programs into programs defined in any other Pulumi language. For example, the original example above can be converted to a Python program with just a single command:

> pulumi convert --language python --out ./bucket-py

This will generate a new Python project in the ./bucket-py/ folder which specifies the exact same infrastructure so that you can cd into that folder, pulumi up and see no changes. The __main__.py generated by pulumi convert looks exactly like what you would have written in Python.

import pulumi
import pulumi_aws as aws

bucket = aws.s3.Bucket("bucket", website=aws.s3.BucketWebsiteArgs(
 index_document="index.html",
))
indexhtml = aws.s3.BucketObject("index.html",
 bucket=bucket.id,
 content="<h1>Hello, world!</h1>",
 content_type="text/html",
 acl="public-read")
pulumi.export("url", bucket.website_endpoint)

From here, you can adopt more features available in Pulumi Python - loops and conditionals, testing, components, and much more!

Summary

We are really excited to be introducing YAML to the collection of languages supported by Pulumi’s Universal Infrastructure as Code platform. With the addition of Pulumi YAML, every cloud builder now has a great option for deploying cloud infrastructure using Pulumi. All of the benefits of Pulumi are available from YAML - the richest Native Providers for major cloud platforms, a wide range of higher-level components in the Pulumi Registry, deep support for secrets, infrastructure refactoring with aliases, a long tail of Cloud and SaaS providers, enforcement of Policy as Code and Audit Logs for all deployments in the Pulumi Service, and so much more. Because you can use components built with other Pulumi languages, the interface to Pulumi YAML can be kept simple, allowing platform teams to define simple interfaces to their cloud infrastructure platform. But if and when the Pulumi YAML does get more complex than is appropriate for YAML, the pulumi convert command provides an easy path to continue to get the full benefits of Pulumi’s general purpose programming language options.

The Pulumi YAML language support is open source at https://github.com/pulumi/pulumi-yaml, and YAML examples are available for all packages and cloud providers in the Pulumi Registry. You can get started with Pulumi YAML today in the Pulumi Getting Started guide and the Pulumi YAML language docs. We can’t wait to see what you build!

Introducing AWS CDK on Pulumi

Luke Hoban — Wed, 04 May 2022 06:56:00 -0700

This blog post contains outdated information. For the latest features, updates, and examples, see our latest release: AWS CDK on Pulumi 1.0.

One of our key goals with Pulumi’s Universal Infrastructure as Code platform is to offer access to the widest range of cloud infrastructure building blocks for use within your cloud engineering projects. Over the years, that has led us to support interoperating seamlessly with a variety of alternative infrastructure definition formats, like Helm, CloudFormation, Azure Resource Manager and Kubernetes YAML. Today we’re really excited to add support for AWS CDK constructs to the list!

The AWS Cloud Development Kit (CDK) offers a large collection of higher-level libraries (“constructs”) for working with the AWS platform, built by service teams at AWS and by the AWS CDK community. These libraries are available in the same set of general purpose programming languages that Pulumi supports, with the primary difference being that AWS CDK compiles infrastructure programs into CloudFormation. This dependence on CloudFormation limits AWS CDK to being deployed via the CloudFormation deployment service which can slow down deployments and introduce some developer productivity friction due to the impedance mismatch between the program you write and the YAML it gets transpiled into.

With the new AWS CDK on Pulumi project, available in public preview today, we are opening up the ability to use AWS CDK constructs from within a Pulumi deployment. For users already using AWS CDK, this provides Pulumi as a new option for orchestrating deployments in place of CloudFormation, offering improved deployment speed, integration with the full set of features of the Pulumi Cloud Engineering Platform (like Policy as Code, Audit Logs, Secrets, and much more). And for Pulumi users, they are now able to leverage and benefit from the decades of experience AWS teams and the AWS CDK community have invested in designing well-architected infrastructure patterns through these constructs.

Even better, you can also now combine AWS CDK and Pulumi resources in a single Pulumi infrastructure as code project - passing outputs from Pulumi resources into AWS CDK constructs, and outputs from AWS CDK constructs into Pulumi resources. This allows you to work across the more than 80 cloud and SaaS providers that Pulumi offers access to, while still benefiting from high level libraries from the AWS CDK project, and without the hassles of a transpiler.

Deploying AWS CDK Constructs with Pulumi

To deploy existing AWS CDK Constructs using Pulumi, simply do the following:

Create a class that derives from pulumicdk.Stack (which itself is derived from awscdk.Stack).
In the constructor, use any AWS CDK constructs from existing libraries such as aws-cdk-lib
Call this.synth() to finalize the stack and deploy its resources.

Constructing an instance of this pulumicdk.Stack from within your Pulumi program will then deploy all of the infrastructure defined by the CDK constructs in the Stack using Pulumi.

For example, the following program deploys two AWS CDK Constructs using Pulumi.

import * as fs from 'fs';
import * as events from 'aws-cdk-lib/aws-events';
import * as events_targets from 'aws-cdk-lib/aws-events-targets';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import { Duration } from 'aws-cdk-lib';
import * as pulumicdk from '@pulumi/cdk';
import { remapCloudControlResource } from './adapter';

class LambdaStack extends pulumicdk.Stack {
 constructor(id: string, options?: pulumicdk.StackOptions) {
 super(id, { ...options, remapCloudControlResource });

 // Use the AWS CDK Lambda Function API directly.
 const lambdaFn = new lambda.Function(this, 'lambda', {
 code: new lambda.InlineCode(fs.readFileSync('lambda-handler.py', { encoding: 'utf-8' })),
 handler: 'index.main',
 timeout: Duration.seconds(300),
 runtime: lambda.Runtime.PYTHON_3_6,
 });

 // Use the AWS CDK Rule API directly.
 const rule = new events.Rule(this, 'rule', {
 // Run 6:00 PM UTC every Monday through Friday
 schedule: events.Schedule.expression('cron(0 18 ? * MON-FRI *)'),
 });

 // Use the AWS CDK to add a Rule target to trigger the Function.
 rule.addTarget(new events_targets.LambdaFunction(lambdaFn));

 // Finalize the stack and deploy its resources.
 this.synth();
 }
}

const stack = new LambdaStack('teststack');

This can be deployed with the pulumi CLI just like any other Pulumi program.

> pulumi up
Updating (dev)
View Live: https://app.pulumi.com/lukehoban/cdk-cron-lambda/dev/updates/37
Type Name Status
+ pulumi:pulumi:Stack cdk-cron-lambda-dev created
+ └─ cdk:index:Stack teststack created
+ └─ cdk:construct:LambdaStack teststack/teststack created
+ ├─ cdk:construct:Function teststack/teststack/lambda created
+ │ ├─ cdk:construct:Role teststack/teststack/lambda/ServiceRole created
+ │ │ └─ aws-native:iam:Role lambdaServiceRole494E4CA6 created
+ │ └─ aws-native:lambda:Function lambda8B5974B5 created
+ └─ cdk:construct:Rule teststack/teststack/rule created
+ ├─ aws:cloudwatch:EventRule ruleF2C1DCDC created
+ ├─ aws:cloudwatch:EventTarget Target0 created
+ └─ aws:lambda:Permission ruleAllowEventRuleteststacklambda47768855D6EFF36B created
Resources:
+ 11 created
Duration: 48s

We can see that each CDK Construct is shown in the Pulumi resource tree, and the leaf nodes of the deployment are the underlying AWS resources needed for this cloud infrastructure. Note that they are deployed via Pulumi, not via CloudFormation. This enables faster deployments, because Pulumi coordinates directly with AWS instead of going through CloudFormation, as well as the ability to use Pulumi features like pulumi logs, Policy as Code, and the features of the Pulumi Service.

Using AWS CDK Constructs Within Pulumi Programs

We can also combine the best of AWS CDK with the best of Pulumi, by using both Pulumi resources and AWS CDK Constructs in the same deployment. This enables combining rich AWS CDK Constructs with the breadth of the Pulumi ecosystem, including support for multiple cloud and SaaS providers.

The following example shows creating an AWS Lambda Function with the Pulumi AWS SDK, and then using an output of that Function (the arn) as an input to an AWS Events Rule CDK Construct. An output from the AWS CDK Construct is then converted to a Pulumi Output value which can be used in the rest of the Pulumi program - and in particular as a Pulumi Stack Output.

import * as events from 'aws-cdk-lib/aws-events';
import * as events_targets from 'aws-cdk-lib/aws-events-targets';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import * as pulumi from '@pulumi/pulumi';
import * as pulumicdk from '@pulumi/cdk';
import * as aws from '@pulumi/aws';
import { remapCloudControlResource } from './adapter';

// Create a resource in Pulumi
const fn = new aws.lambda.CallbackFunction('lambda', {
 callback: (ev, ctx) => console.log(JSON.stringify(ev)),
});

class LambdaStack extends pulumicdk.Stack {
 ruleArn: pulumi.Output<string>;

 constructor(id: string, options?: pulumicdk.StackOptions) {
 super(id, { ...options, remapCloudControlResource });

 // Use the AWS CDK Rule API directly.
 const rule = new events.Rule(this, 'rule', {
 // Run 6:00 PM UTC every Monday through Friday
 schedule: events.Schedule.expression('cron(0 18 ? * MON-FRI *)'),
 });

 // Get an AWS CDK Function instance from the Pulumi Lambda ARN.
 const lambdaFn = lambda.Function.fromFunctionArn(this, 'lambdaFn', pulumicdk.asString(fn.arn));

 // Use the AWS CDK to add a Rule target to trigger the Function.
 rule.addTarget(new events_targets.LambdaFunction(lambdaFn));

 // Export the Lambda function's ARN as an output.
 this.ruleArn = this.asOutput(rule.ruleArn);

 this.synth();
 }
}

const stack = new LambdaStack('teststack');
export const ruleArn = stack.ruleArn;

Note how the pulumicdk.asString and asOutput functions are used to convert Pulumi Outputs to AWS CDK Tokens and vice-versa to map values out of and into Pulumi.

Building on AWS Native and the AWS Cloud Control API

Last year we released a new AWS Native Pulumi provider in preview. The AWS Native provider builds on the new AWS Cloud Control API, which offers direct provisioning support for the same resource model supported by AWS CloudFormation.

The new AWS CDK on Pulumi builds upon AWS Native by converting the CloudFormation resource tree into the corresponding resource definitions from the AWS Native provider. This provides a high fidelity mapping of each resource into Pulumi. The coverage provided by AWS Cloud Control API (and thus AWS Native) is not yet complete across the entire breadth of the CloudFormation resource model though, so for some resources, it may be necessary to map them to a resource definition to use via the AWS provider. This is handled via the remapCloudControlResource API which allows providing a custom mapping for a given CloudFormation resource type which is not yet supported in Pulumi AWS Native and AWS Cloud Control.

For example, to provide a mapping for the AWS::Lambda::Permission resource type, define a function like the following:

export function remapCloudControlResource(
 element: CfnElement,
 logicalId: string,
 typeName: string,
 rawProps: any,
 options: pulumi.ResourceOptions,
): pulumi.CustomResource | undefined {
 const props = pulumicdk.interop.normalize(rawProps);
 switch (typeName) {
 case 'AWS::Lambda::Permission':
 return new aws.lambda.Permission(
 logicalId,
 {
 action: props['action'],
 function: props['functionName'],
 principal: props['principal'],
 sourceArn: props['sourceArn'] ?? undefined,
 },
 options,
 );
 }

 return undefined;
}

In many cases, this is not needed, as AWS CDK on Pulumi offers builtin support for remapping many common resources. Over the coming months, as AWS Cloud Control API support expands to cover more of the CloudFormation resource model, these remappings will become less necessary in practice, with almost all resources being supported directly via Pulumi AWS Native.

Use AWS CDK constructs with Pulumi

Deploy your existing AWS CDK constructs through Pulumi, and combine them with resources from hundreds of cloud and SaaS providers in a single program.

Learn more

Using High-Level “L3” Constructs

AWS CDK offers a mix of relatively low level (unopinionated) constructs, as well as high level (more opinionated) constructs. With AWS CDK on Pulumi, we can use any of these. This includes libraries like aws-ecs-patterns and AWS Solutions Constructs.

For example, to deploy a load balanced and auto-scaled Fargate service to a dedicated VPC, we can use the following program:

import * as ec2 from 'aws-cdk-lib/aws-ec2';
import * as pulumi from '@pulumi/pulumi';
import * as pulumicdk from '@pulumi/cdk';
import * as ecs from 'aws-cdk-lib/aws-ecs';
import * as ecs_patterns from 'aws-cdk-lib/aws-ecs-patterns';
import { Construct } from 'constructs';
import { Stack, Duration, CfnOutput } from 'aws-cdk-lib';
import {remapCloudControlResource } from './adapter';

class FargateStack extends pulumicdk.Stack {

 loadBalancerDNS: pulumi.Output<string>;

 constructor(id: string, options?: pulumicdk.StackOptions) {
 super(id, { ...options, remapCloudControlResource });

 // Create VPC and Fargate Cluster
 const vpc = new ec2.Vpc(this, 'MyVpc', { maxAzs: 2 });
 const cluster = new ecs.Cluster(this, 'fargate-service-autoscaling', { vpc });

 // Create Fargate Service
 const fargateService = new ecs_patterns.NetworkLoadBalancedFargateService(this, 'sample-app', {
 cluster,
 taskImageOptions: {
 image: ecs.ContainerImage.fromRegistry("amazon/amazon-ecs-sample")
 },
 });

 // Setup AutoScaling policy
 const scaling = fargateService.service.autoScaleTaskCount({ maxCapacity: 2 });
 scaling.scaleOnCpuUtilization('CpuScaling', {
 targetUtilizationPercent: 50,
 scaleInCooldown: Duration.seconds(60),
 scaleOutCooldown: Duration.seconds(60)
 });

 this.loadBalancerDNS = this.asOutput(fargateService.loadBalancer.loadBalancerDnsName);

 // Finalize the stack and deploy its resources.
 this.synth();
 }
};

const stack = new FargateStack('fargatestack');
export const loadBalancerURL = stack.loadBalancerDNS;

Deploying this with pulumi up builds a few dozen AWS resources to construct the various building blocks necessary to capture this in raw AWS infrastructure:

> pulumi up

Updating (dev)

View Live: https://app.pulumi.com/lukehoban/pulumi-aws-cdk-fargate/dev/updates/1

 Type Name Status
 + pulumi:pulumi:Stack pulumi-aws-cdk-fargate-dev created
 + └─ cdk:index:Stack fargatestack created
 + └─ cdk:construct:FargateStack fargatestack/fargatestack created
 + ├─ cdk:construct:Cluster fargatestack/fargatestack/fargate-service-autoscaling created
 + │ └─ aws-native:ecs:Cluster fargateserviceautoscalingD107CF93 created
 + ├─ cdk:construct:NetworkLoadBalancedFargateService fargatestack/fargatestack/sample-app created
 + │ ├─ cdk:construct:FargateTaskDefinition fargatestack/fargatestack/sample-app/TaskDef created
 + │ │ ├─ cdk:construct:ContainerDefinition fargatestack/fargatestack/sample-app/TaskDef/web created
 + │ │ │ └─ cdk:construct:LogGroup fargatestack/fargatestack/sample-app/TaskDef/web/LogGroup created
 + │ │ │ └─ aws-native:logs:LogGroup sampleappTaskDefwebLogGroup34BE8C79 created
 + │ │ ├─ cdk:construct:Role fargatestack/fargatestack/sample-app/TaskDef/ExecutionRole created
 + │ │ │ ├─ cdk:construct:Policy fargatestack/fargatestack/sample-app/TaskDef/ExecutionRole/DefaultPolicy created
 + │ │ │ │ ├─ aws:iam:Policy sampleappTaskDefExecutionRoleDefaultPolicy0AD15374 created
 + │ │ │ │ └─ aws:iam:RolePolicyAttachment sampleappTaskDefExecutionRoleDefaultPolicy0AD15374-0 created
 + │ │ │ └─ aws-native:iam:Role sampleappTaskDefExecutionRoleAD6F4C40 created
 + │ │ ├─ cdk:construct:Role fargatestack/fargatestack/sample-app/TaskDef/TaskRole created
 + │ │ │ └─ aws-native:iam:Role sampleappTaskDefTaskRoleB530CAC0 created
 + │ │ └─ aws-native:ecs:TaskDefinition sampleappTaskDef6BF75736 created
 + │ ├─ cdk:construct:NetworkLoadBalancer fargatestack/fargatestack/sample-app/LB created
 + │ │ ├─ cdk:construct:NetworkListener fargatestack/fargatestack/sample-app/LB/PublicListener created
 + │ │ │ ├─ cdk:construct:NetworkTargetGroup fargatestack/fargatestack/sample-app/LB/PublicListener/ECSGroup created
 + │ │ │ │ └─ aws:lb:TargetGroup sampleappLBPublicListenerECSGroup525A567D created
 + │ │ │ └─ aws-native:elasticloadbalancingv2:Listener sampleappLBPublicListenerC4DF6480 created
 + │ │ └─ aws:lb:LoadBalancer sampleappLBBDE1D276 created
 + │ └─ cdk:construct:FargateService fargatestack/fargatestack/sample-app/Service created
 + │ ├─ cdk:construct:ScalableTaskCount fargatestack/fargatestack/sample-app/Service/TaskCount created
 + │ │ └─ cdk:construct:ScalableTarget fargatestack/fargatestack/sample-app/Service/TaskCount/Target created
 + │ │ ├─ cdk:construct:TargetTrackingScalingPolicy fargatestack/fargatestack/sample-app/Service/TaskCount/Target/CpuScaling created
 + │ │ │ └─ aws:appautoscaling:Policy sampleappServiceTaskCountTargetCpuScalingF4452F80 created
 + │ │ └─ aws:appautoscaling:Target sampleappServiceTaskCountTargetE827DC30 created
 + │ ├─ cdk:construct:SecurityGroup fargatestack/fargatestack/sample-app/Service/SecurityGroup created
 + │ │ └─ aws:ec2:SecurityGroup sampleappServiceSecurityGroup0ABF0D21 created
 + │ └─ aws-native:ecs:Service sampleappServiceE7504FDB created
 + └─ cdk:construct:Vpc fargatestack/fargatestack/MyVpc created
 + ├─ cdk:construct:PublicSubnet fargatestack/fargatestack/MyVpc/PublicSubnet2 created
 + │ ├─ aws:ec2:Eip MyVpcPublicSubnet2EIP8CCBA239 created
 + │ ├─ aws-native:ec2:Subnet MyVpcPublicSubnet2Subnet492B6BFB created
 + │ ├─ aws-native:ec2:RouteTable MyVpcPublicSubnet2RouteTable1DF17386 created
 + │ ├─ aws:ec2:NatGateway MyVpcPublicSubnet2NATGateway91BFBEC9 created
 + │ ├─ aws:ec2:Route MyVpcPublicSubnet2DefaultRoute052936F6 created
 + │ └─ aws-native:ec2:SubnetRouteTableAssociation MyVpcPublicSubnet2RouteTableAssociation227DE78D created
 + ├─ cdk:construct:PublicSubnet fargatestack/fargatestack/MyVpc/PublicSubnet1 created
 + │ ├─ aws:ec2:Eip MyVpcPublicSubnet1EIP096967CB created
 + │ ├─ aws-native:ec2:Subnet MyVpcPublicSubnet1SubnetF6608456 created
 + │ ├─ aws-native:ec2:RouteTable MyVpcPublicSubnet1RouteTableC46AB2F4 created
 + │ ├─ aws:ec2:NatGateway MyVpcPublicSubnet1NATGatewayAD3400C1 created
 + │ ├─ aws-native:ec2:SubnetRouteTableAssociation MyVpcPublicSubnet1RouteTableAssociation2ECEE1CB created
 + │ └─ aws:ec2:Route MyVpcPublicSubnet1DefaultRoute95FDF9EB created
 + ├─ cdk:construct:PrivateSubnet fargatestack/fargatestack/MyVpc/PrivateSubnet2 created
 + │ ├─ aws-native:ec2:RouteTable MyVpcPrivateSubnet2RouteTableCEDCEECE created
 + │ ├─ aws-native:ec2:Subnet MyVpcPrivateSubnet2Subnet0040C983 created
 + │ ├─ aws-native:ec2:SubnetRouteTableAssociation MyVpcPrivateSubnet2RouteTableAssociation86A610DA created
 + │ └─ aws:ec2:Route MyVpcPrivateSubnet2DefaultRoute9CE96294 created
 + ├─ cdk:construct:PrivateSubnet fargatestack/fargatestack/MyVpc/PrivateSubnet1 created
 + │ ├─ aws-native:ec2:RouteTable MyVpcPrivateSubnet1RouteTable8819E6E2 created
 + │ ├─ aws-native:ec2:Subnet MyVpcPrivateSubnet1Subnet5057CF7E created
 + │ ├─ aws-native:ec2:SubnetRouteTableAssociation MyVpcPrivateSubnet1RouteTableAssociation56D38C7E created
 + │ └─ aws:ec2:Route MyVpcPrivateSubnet1DefaultRouteA8CDE2FA created
 + ├─ aws-native:ec2:InternetGateway MyVpcIGW5C4A4F63 created
 + ├─ aws-native:ec2:VPC MyVpcF9F0CA6F created
 + └─ aws:ec2:InternetGatewayAttachment MyVpcVPCGW488ACE0D created

Outputs:
 loadBalancerURL: "sampleappLBBDE1D276-f412396-b0bebc132ad72c07.elb.us-west-2.amazonaws.com"

Resources:
 + 61 created

Duration: 4m51s

Future Directions

Currently AWS CDK on Pulumi is supported only for TypeScript users, due to how the AWS CDK synthesis process (implemented in TypeScript) must be invoked from within the Pulumi program. We are exploring ways to bring this support into other Pulumi (and CDK) languages as part of future updates to the library.

In the meantime, AWS CDK on Pulumi can be used within Component Packages implemented in TypeScript, and exposed to any Pulumi language (including newly supported Java and YAML.

Summary

Support for deploying AWS CDK constructs from within Pulumi programs opens up a wide range of new opportunities to interoperate between the AWS CDK and Pulumi ecosystems.For existing AWS CDK users, this provides new ways to deploy your CDK infrastructure and constructs. For existing Pulumi users, this opens up new higher-level libraries from the AWS CDK to use from within your Pulumi programs. We’re excited to be bringing these two communities closer together, and can’t wait to see what cloud builders can do with these two great cloud technologies together.

The AWS CDK on Pulumi project is open source at https://github.com/pulumi/pulumi-cdk and on NPM at https://www.npmjs.com/package/@pulumi/cdk. Get started today!

Pulumi Kubernetes Operator 1.0

Luke Hoban — Tue, 12 Oct 2021 00:00:00 +0000

Check out version 2.0 of the Pulumi Kubernetes Operator.

Last year we released the Pulumi Kubernetes Operator, a new cloud-native way to manage and deploy cloud infrastructure using Pulumi from within your Kubernetes environment. Since then, we’ve worked with many Pulumi users who have adopted the Pulumi Kubernetes Operator at increasingly larger scales and for a wide variety of use cases. Today, we’re excited to make the 1.0 release of the Pulumi Kubernetes Operator available.

The Pulumi Kubernetes Operator defines a Kubernetes Custom Resource called pulumi.com/v1/Stack, which represents a Pulumi stack. The Pulumi stack can be authored in any supported Pulumi language (TypeScript, Python, Go, .NET) and can deploy and manage cloud infrastructure in any supported cloud (AWS, Azure, GCP, Kubernetes and 60+ additional cloud and SaaS providers). The Pulumi Kubernetes Operator triggers cloud deployments based on changes to the Stack Custom Resource or the resources it tracks.

As a result, the Pulumi Kubernetes Operator enables users to specify the desired state of their cloud infrastructure using resources managed directly in their Kubernetes cluster, which trigger creation, update and deletion of the detailed cloud infrastructure they need.

For example, the following Kubernetes resource can be created to deploy a Pulumi program which provisions cloud infrastructure to manage NGINX provided in the target Git repo and commit, and future updates to the commit or config settings will deploy updates to the infrastructure.

apiVersion: pulumi.com/v1
kind: Stack
metadata:
 name: nginx-k8s-stack-production
spec:
 stack: acmecorp/nginx/production
 projectRepo: https://github.com/acmecorp/pulumi-nginx
 commit: 2b0889718d3e63feeb6079ccd5e4488d8601e353

The Pulumi Kubernetes Operator can also be used along with Pulumi’s Kubernetes provider, which allows Pulumi programs to deploy resources to Kubernetes, either in the same cluster as the operator, or in another cluster. This can be used to package up complex Kubernetes infrastructure into more abstracted units, to mix Kubernetes and cloud infrastructure into a single unit of deployment, to define how application workloads are deployed into a cluster, or for a wide variety of additional use cases. The Pulumi Kubernetes provider has full support for the Kubernetes API available in every Pulumi language, along with newly improved support for Helm.

We’re at KubeCon North America 2021 this week - come meet us at booth SU8 if you are there! We also have two Kuberentes-focused webinars coming up where you can learn more about working with Pulumi and Kubernetes: Provision, Manage and Scale Kubernetes on Any Cloud (Oct 14th) and From Zero to Production in Kubernetes (Oct 21st).

What’s New in Pulumi Kubernetes Operator 1.0?

The 1.0 release includes many significant enhancements since the initial launch, as well as scalability, performance and quality improvements.

GitOps workflows with branch tracking and private Git repos

The operator now supports specifying a branch instead of a commit in a target Git repo. When a branch is specified, the operator will track any changes to what that branch is tracking, and trigger deployments automatically when new code is pushed. This is a “pull-based” model, where the operator watches the Git repo for changes. This ensures that the operator can be run in a private network environment that is not reachable from the internet (or via GitHub webhooks), and that the operator is fully in control of how these updates are discovered and processed.

This opens up a ton of new use cases for the Pulumi Kubernetes Operator, allowing the declarative model to be pushed from being purely within the Kubernetes resource, to allowing the exact commit to deploy to be specified in a Git repo branch. This allows infrastructure configuration to be defined once for a given environment, and then actively updated via git push. In this mode, the Pulumi Kubernetes Operator can be used in place of traditional CI/CD integrations or other cloud infrastructure deployment workflows.

The same stack as above can be changed to track a branch by using the branch option:

apiVersion: pulumi.com/v1
kind: Stack
metadata:
 name: nginx-k8s-stack-production
spec:
 stack: acmecorp/nginx/production
 projectRepo: https://github.com/acmecorp/pulumi-nginx
 branch: refs/heads/production

This will watch the production branch of the repo for Git pushes, and then trigger updates to the production stack of the NGINX deployment based on the new code pushed to that branch.

To help identify what commit was last deployed when tracking a branch, the Operator publishes information about the last update under .status.lastUpdate, including the lastAttemptedCommit property and lastSuccessfulCommit property.

Along with support for branch tracking, the Operator also now supports access to private Git repositories, critical for deploying your production infrastructure via the Pulumi Kubernetes Operator.

Building on the Pulumi Automation API

A couple months after we released the Pulumi Kubernetes Operator, we also released the initial version of the Pulumi Automation API. The Automation API provides an SDK for embedding Pulumi deployments inside your own software and systems, allowing you to build your own interfaces for managing cloud infrastructure. The design of the Automation API was heavily influenced by our experience building the Pulumi Kubernetes Operator, which was a perfect example of one of these custom interfaces - the Pulumi Kubernetes Operator embeds Pulumi deployments inside Kubernetes, offering a new interface (Kubernetes Stack resources) for managing cloud infrastructure.

Since then, we’ve moved the Pulumi Kubernetes Operator over to use the Pulumi Automation API, significantly reducing the size of the operator implementation, building on a robust foundation shared with dozens of other projects in the Pulumi ecosystem, and benefiting from many improvements and features added to the Pulumi Automation API. Earlier this year, the Pulumi Automation API also reached GA as part of the Pulumi 3.0 release.

Support for any state and secrets management backend

Pulumi supports a wide variety of state management and secrets management backends. Since the initial release of the Pulumi kubernetes Operator, we have added support for using not just the Pulumi Service backends, but also all of the other open source backend options, such as S3, Azure Blob and GCS for state storage, and KMS, KeyVault and Vault for secrets management.

apiVersion: pulumi.com/v1
kind: Stack
metadata:
 name: s3-bucket-stack
spec:
 backend: s3://my-s3-backend
 stack: acmecorp/operator-dev
 projectRepo: https://github.com/acmecorp/pytest1
 initOnCreate: true
 branch: refs/head/master
 config:
 aws:region: us-west-2

Worry-free Upgrades

The Pulumi Kubernetes Operator manages cloud infrastructure deployments which can take seconds to tens of minutes to complete. Although Pulumi ensures these deployments can be continued if interrupted, we want to ensure as little disruption to critical cloud infrastructure deployments as possible via the Operator. As a result, we have added a few key features to ensure that the Pulumi Kubernetes Operator can be safely upgraded to new versions without disrupting any deployments that are in-flight.

This includes capabilities like Leader Election, Graceful Shutdown and more.

Prometheus Metrics

The Pulumi Kubernetes Operator automatically manages a lot of details of Pulumi deployments, and as a result, it’s critical to be able to get insight into the operation of the Operator. Previously, this was available either via operator logs, or via the Pulumi Service. As part of the 1.0 release, the Operator is also now exporting Prometheus-compatible metrics for stacks_active and stacks_failing. These can be used to drive reporting and other workflows inside the Kubernetes cluster.

$ kubectl port-forward service/pulumi-kubernetes-operator-metrics 8383:8383 &
$ curl -s localhost:8383/metrics | grep "stacks_active\|stacks_failing"
# HELP stacks_active Number of stacks currently tracked by the Pulumi Kubernetes Operator
# TYPE stacks_active gauge
stacks_active 2
# HELP stacks_failing Number of stacks currently registered where the last reconcile failed
# TYPE stacks_failing gauge
stacks_failing{name="stack-test-aws-s3-commit-change-5wuczx",namespace="default"} 1

Documentation

We’ve added new documentation and enhanced existing documentation to ensure it’s easy to get started with the Pulumi Kubernetes Operator.

Full Stack Custom Resource API documentation is now available. And guides for deploying the Pulumi Kubernetes Operator and creating Stack resources are available for use either from kubectl/YAML or using the Pulumi Kubernetes Provider.

What’s Next

The 1.0 release of the Pulumi Kubernetes Operator is an important milestone, but there is still lots more planned to further expand the capabilities and scalability of the Operator. Some upcoming areas we’ll be investing:

Conclusion

The Pulumi Kubernetes Operator is a great way to deploy and manage cloud infrastructure from within your Kubernetes environment. With the 1.0 release, we’ve added many significant enhancements, as well as scalability, performance and quality improvements. You can get started with the Pulumi Kubernetes Operator today. Enjoy!

Announcing the Pulumi AWS Native Provider, Powered by the AWS Cloud Control API

Luke Hoban — Thu, 30 Sep 2021 00:00:00 +0000

The AWS Native Provider was renamed as the “AWS Cloud Control Provider” in October 2024.

We are excited to announce the release of the new AWS Native provider for Pulumi, which is available today in preview. AWS is the most-used cloud provider across the Pulumi ecosystem, and with the new AWS Native provider, we are focused on delivering the best possible support for the AWS platform to all Pulumi users.

Pulumi Native Providers like AWS Native are a new type of Pulumi Package that give you the most complete and consistent interface for the modern cloud. Pulumi native providers bring the full power of the top cloud providers to the Pulumi Cloud Engineering Platform, with faster updates and more complete coverage than any other infrastructure as code offering.

The AWS Native provider offers same-day support for all new AWS features and releases covered by the newly released AWS Cloud Control API, which typically supports new AWS features on the day of launch. By building on the AWS Cloud Control API, the AWS Native provider offers a robust, reliable and well-defined resource model for AWS that’s available to Pulumi users in all Pulumi languages, including TypeScript, Python, Go and C#. By leveraging the AWS Cloud Control API, the AWS Native provider builds on the work done by service teams at AWS to define the resource model for their services. This ensures a rock solid provisioning lifecycle for resources deployed with the AWS Native provider.

This release also includes a new tool for migrating existing CloudFormation templates into Pulumi programs in your favorite language, powered by the new AWS Native provider and AWS Cloud Control API, as well as the ability to deploy any 3rd party resources in the CloudFormation Registry, including resources from Atlassian, Datadog, Densify, Dynatrace, Fortinet, New Relic, and Spot by NetApp.

To learn how AWS built the Cloud Control API, and many other great topics, watch the replay of Cloud Engineering Summit 2021 with speakers from each of the major cloud providers as well as 40+ leaders and practitioners from across the industry.

API Coverage

Resources available in the Pulumi AWS Native provider are based on the resources defined in the AWS CloudFormation Registry. Today, hundreds of AWS resources are available from this registry via Cloud Control API and the Pulumi AWS Native provider. This list will continue to grow as additional resources are added to the AWS Cloud Control API. In addition, all new features and services that are released by AWS will be supported in AWS Native, typically on the same day as the release.

The Pulumi AWS Native provider can be used in combination with the classic Pulumi AWS provider, as well as the 60+ additional Pulumi resource providers which cover a wide variety of other cloud and SaaS platforms. While in preview, the Pulumi AWS Native provider may not yet have support for every feature of AWS that you need in your Pulumi applications, but can always be used along with the classic Pulumi AWS provider to cover both existing use cases and brand new supported features.

In this example, we can see how the new AWS S3 Object Lambda feature can be used via the AWS Native provider, with access to the full API defined by the S3 team at AWS:

import * as pulumi from "@pulumi/pulumi";
import * as awsnative from "@pulumi/aws-native";

const bucket = new awsnative.s3.Bucket("source");

const accessPoint = new awsnative.s3.AccessPoint("ap", {
 bucket: bucket.id,
});

const objectlambda = new awsnative.s3objectlambda.AccessPoint("objectlambda-ap", {
 objectLambdaConfiguration: {
 supportingAccessPoint: accessPoint.arn,
 transformationConfigurations: [{
 actions: ["GetObject"],
 contentTransformation: {
 AwsLambda: {
 FunctionArn: fn.arn,
 },
 },
 }]
 }
});

import pulumi
import pulumi_aws_native as aws_native

my_bucket = aws_native.s3.Bucket("myBucket")

ap = aws_native.s3.AccessPoint("ap", bucket=my_bucket.id)

objectlambdaap = aws_native.s3objectlambda.AccessPoint("objectlambdaap", object_lambda_configuration=aws_native.s3objectlambda.AccessPointObjectLambdaConfigurationArgs(
 supporting_access_point=ap.arn,
 transformation_configurations=[aws_native.s3objectlambda.AccessPointTransformationConfigurationArgs(
 actions=["GetObject"],
 content_transformation={
 "AwsLambda": {
 "FunctionArn": fn.arn,
 },
 },
 )],
))

var bucket = new AwsNative.S3.Bucket("my-bucket");

var accessPoint = new AwsNative.S3.AccessPoint("ap", new AwsNative.S3.AccessPointArgs
{
 Bucket = bucket.Id
});

var objectLambda = new AwsNative.S3ObjectLambda.AccessPoint("objectlambda-ap", new AwsNative.S3ObjectLambda.AccessPointArgs
{
 ObjectLambdaConfiguration = new AwsNative.S3ObjectLambda.Inputs.AccessPointObjectLambdaConfigurationArgs
 {
 SupportingAccessPoint = accessPoint.Arn,
 TransformationConfigurations =
 {
 new AwsNative.S3ObjectLambda.Inputs.AccessPointTransformationConfigurationArgs
 {
 Actions = { "GetObject" },
 ContentTransformation = fn.Arn.Apply(arn => new Dictionary<string, object>
 {
 ["AwsLambda"] = new Dictionary<string, object>
 {
 ["FunctionArn"] = arn
 }
 }
 }
 }
 }
});

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 myBucket, err := s3.NewBucket(ctx, "myBucket", nil)
 if err != nil {
 return err
 }
 ap, err := s3.NewAccessPoint(ctx, "ap", &s3.AccessPointArgs{
 Bucket: myBucket.ID(),
 })
 if err != nil {
 return err
 }

 _, err = s3objectlambda.NewAccessPoint(ctx, "objectlambdaap", &s3objectlambda.AccessPointArgs{
 ObjectLambdaConfiguration: &s3objectlambda.AccessPointObjectLambdaConfigurationArgs{
 SupportingAccessPoint: ap.Arn,
 TransformationConfigurations: s3objectlambda.AccessPointTransformationConfigurationArray{
 &s3objectlambda.AccessPointTransformationConfigurationArgs{
 Actions: pulumi.StringArray{
 pulumi.String("GetObject"),
 },
 ContentTransformation: pulumi.Map{
 "AwsLambda": pulumi.Map{
 "FunctionArn": fn.Arn,
 },
 },
 },
 },
 },
 })
 if err != nil {
 return err
 }
 return nil
 })
}

Always Up-To-Date

The Pulumi AWS Native provider is built automatically from the resources and specifications provided by the CloudFormation Registry. Those specifications are updated frequently whenever new resources and features are shipped by service teams at AWS. The AWS Native provider is rebuilt and released nightly directly from the latest specifications, ensuring that Pulumi users have access to rich Pulumi SDKs for working with new features the same day they are added to Cloud Control API, typically on the day of launch.

Interoperability with CloudFormation

Although the AWS Native provider leverages the CloudFormation Registry and AWS Cloud Control API, it does not depend on deploying CloudFormation templates. You can write your programs in Pulumi, and Pulumi is responsible for managing the lifecycle and deployment of all resources in your stack. You get the same lightning-fast deployments you expect with Pulumi, along with the same rich infrastructure as code fundamentals like secrets, aliases, components, testing, policy, management console, and more. There is no CloudFormation stack artifact in the AWS console, since the resource definitions and deployment are handled entirely by Pulumi".

Along with the release of the new AWS Native provider, we’ve also released cf2pulumi, a tool for migrating your existing CloudFormation templates to Pulumi programs in your favorite languages. cf2pulumi is available as both a web-based converter at https://www.pulumi.com/cf2pulumi/, and as a CLI tool.

Access to 3rd Party Resources from the CloudFormation Registry

In addition to all of the AWS resources available in AWS Native, we are also making it possible to create any of the 3rd-party resources available in the CloudFormation Registry, including resources from Atlassian, MongoDB, Snyk, and Spot by NetApp. You can read more about the 3rd party resources available in the CloudFormation registry in the AWS launch post from earlier this year. The aws.ExtensionResource resource can be used to construct any 3rd party resource based on its resource type name. In the following example, we create a MongoDB Atlas project via the AWS Native provider:

var project = new aws.ExtensionResource("atlas-project", {
 type: "MongoDB::Atlas::Project",
 properties: {
 Name: "test",
 OrgId: "org123",
 },
});

project = ExtensionResource(
 "atlas-project", {
 type="MongoDB::Atlas::Project",
 properties={
 Name: "test",
 OrgId: "org123",
 }
)

var project = new AwsNative.ExtensionResource("atlas-project", new AwsNative.ExtensionResourceArgs
{
 Type = "MongoDB::Atlas::Project",
 Properties =
 {
 "Name": "test",
 "OrgId": "org123",
 }
});

_, err = aws.ExtensionResource(ctx, "atlas-project", &aws.ExtensionResourceArgs{
 Type: pulumi.String("MongoDB::Atlas::Project"),
 Properties: []map[string]interface{}{
 "Name": "test",
 "OrgId": "org123",
 },
})

Conclusion

Pulumi’s Native Providers offer the very best support for each of the major cloud providers. Since we launched our first native provider, users have been asking when they could get access to a truly native provider for AWS, with same-day support for new features, high quality, and the familiar APIs documented by AWS service teams. To deliver this, we’ve worked with the AWS Cloud Control API team to ensure that we can provide a great experience for AWS users of Pulumi, and we’re extremely excited by what this will enable for AWS developers. Today’s release is just a starting point. We are working on increased resource coverage, higher-level APIs, and integrations with additional AWS services that will be delivered over the coming months as we prepare AWS Native for general availability next year.

Get started with the AWS Native provider today:

Announcing New Usage-Based Pricing For Your Whole Team

Luke Hoban — Tue, 29 Jun 2021 00:00:00 +0000

This post is out-of-date. For the latest pricing and editions, please visit the Pulumi Pricing page.

Today we are launching Pulumi Team Edition, a new way for teams to adopt and use Pulumi and the Pulumi Service to collaborate on building, managing and deploying cloud infrastructure as code. Pulumi Team Edition is priced based on the number of cloud resources under management, with a generous free tier to ensure that teams can get up and running with Pulumi Team Edition at no cost.

Pulumi Enterprise Edition, which offers larger organizations advanced security, policy, access control, support and billing options, is also now available with usage-based pricing, including prepaid options with bulk discounts available.

These new offerings replace the previous Pulumi Team Starter, Team Pro and Enterprise editions, which were priced based on the number of users who had access to deploy and manage infrastructure. As we’ve worked with many customers to tackle their cloud modernization initiatives, we’ve heard consistently that teams would like to use infrastructure as code broadly across the organization. This is a core goal of Cloud Engineering, which empowers every developer and infrastructure practitioner in the team to be hands-on with cloud infrastructure. For some users, this might be every day, for others, it might be a couple times a month, but giving everyone in the organization access to manage their cloud infrastructure empowers organizations to move more quickly and collaborate more effectively. This new pricing model was designed to make it easy to give Pulumi to the entire team. Pulumi Team Edition supports up to 10 users, and Pulumi Enterprise Edition supports unlimited users.

Pricing based on the number of resources under management means that price scales naturally with the cloud infrastructure complexity being tamed, and aligns well with how your infrastructure is purchased. Pricing is based on Pulumi Credits, and a Pulumi Credit is one cloud resource under management for one hour.

New customers can sign up for Pulumi Team Edition starting today. Existing customers can contact us now if they are interested in moving to the new Team or Enterprise Editions.

The net result is three Pulumi editions available now:

Individual Edition: our Community Edition is being rebranded but remains unchanged, and is free for unlimited individual use.
Team Edition: up to 10 members can collaborate on projects with unlimited deployments, unlimited stacks and unlimited update history. This plan includes 150k free Pulumi Credits per month, letting you manage up to 200 cloud resources for free every month. After that, additional Pulumi Credits cost only $1 for every 2000 Credits.
Enterprise Edition: for large organizations that need unlimited users, unlimited teams, Role-Based Access Control, SAML/SSO, policy enforcement, audit logging, and 12x5 or 24x7 support for their production deployments.

Full details of the new editions are available on our pricing page.

You can try out Pulumi with a 14-day free trial, no credit card required. If you need a longer trial period, want to discuss potential proof of concept projects, or are interested in advanced capabilities, such as policy and audit logs, Training, or Support, please contact us.

We remain committed to fostering a vibrant open source community while also delivering a robust production service that teams of all sizes are choosing for their cloud needs. From startups, to medium-sized organizations, to the largest Fortune 500 customers, we truly believe that these new editions now offer a great option for everyone. If you are using our pure open source offering, nothing has changed. We have also recently added a new offer for open source projects and foundations, and provide discounts for early stage startups. To discuss either, please contact us.

We hope this change helps to get Pulumi into the hands of more of your team faster, accelerating your Cloud Engineering journey, and look forward to seeing the great things you and your team builds with Pulumi!

Announcing Pulumi 3.0

Luke Hoban — Tue, 20 Apr 2021 00:00:06 -0700

Today we’re excited to announce the availability of Pulumi 3.0, the next major version of the Pulumi open source project, and the foundation for Pulumi’s Cloud Engineering Platform.

Pulumi offers the most complete infrastructure as code platform for building, deploying and managing modern cloud infrastructure and applications. Pulumi enables cloud engineers to use familiar languages to describe their cloud infrastructure - bringing core software engineering tools and practices to bear on managing and getting the maximum value from their cloud platforms of choice - across dozens of cloud and SaaS providers.

Pulumi 3.0 includes dozens of significant new features and hundreds of improvements that build on this foundation. This release includes more than 200 contributions from over 150 members of the Pulumi community, and builds on feedback from working with thousands of Pulumi users and customers over the last year.

Some of the most exciting new features in this release include:

Pulumi Automation API: All the power of Pulumi, now available to embed within your own software and applications in Node.js, Python, Go and .NET - opening up a brand new wave of modern cloud infrastructure use cases, including self-service infrastructure portals, custom CLIs, and richer CI/CD workflows.
Native Pulumi Providers for Azure (GA) and Google Cloud (Preview): New providers with 100% platform coverage, same-day support for new features, and the exact same resource models defined by the cloud providers - the most complete and consistent way to manage cloud infrastructure in Azure and Google Cloud.
Pulumi Packages and Multi-Language Components: Take any Pulumi components written in your language of choice, and make them available as a Pulumi Package to users in all other Pulumi languages - bringing together each of the language ecosystems Pulumi supports into a single Pulumi component ecosystem.
Improved Go and Python SDKs: Updates to the core Go and Python SDKs make APIs simpler and more consistent, unlock more tooling support, and improve compile times - making it even easier to use all the benefits of your favorite programming language when building and managing cloud infrastructure.
New Integrations and Features in the Pulumi Service: Alongside Pulumi 3.0, users of the Pulumi Service also now have access to new dashboards for organization-level cloud infrastructure insights, a CI/CD Assistant for easy version control and CI/CD integration with just a few clicks, and new identity, access control, and auditing capabilities.

👉 Get started with Pulumi 3.0 today or check out the Pulumi 3.0 migration guide to update your existing Pulumi projects to Pulumi 3.0.

Pulumi Automation API

Pulumi’s Automation API was released in preview last year and is now GA as part of Pulumi 3.0. It provides an API for embedding Pulumi within your own applications, and supports all Pulumi languages including TypeScript/JavaScript, Python, Go, and C#. Instead of working with the pulumi CLI directly, or writing scripts to execute the CLI within your CI/CD environment, you can now build your own software solutions that build on top of Pulumi’s robust, desired state infrastructure as code deployment engine. It’s simple to embed Pulumi in your software-defined workflows with just a few lines of code.

This opens up so many exciting new opportunities for how your team can manage cloud infrastructure; we’ve seen amazing creativity as Pulumi’s customers and communities have created solutions on top of Automation API. In addition, Pulumi Automation API powers several other new features like the Pulumi Kubernetes Operator, the Pulumi GitHub Action v3, and Concourse CI support for Pulumi.

👉 Read more in the Pulumi Automation API blog post.

Native Providers for Azure and Google Cloud

Pulumi offers packages for managing cloud infrastructure in more than 50 cloud and SaaS applications, with more than a dozen new providers added since Pulumi 2.0 including partnerships with Civo, Venafi, Okta, Auth0, Equinix Metal, Spot Ocean for Azure, and Splunk.

For the largest and most widely used cloud providers, we’ve created something new: Pulumi native providers, which give you the most complete and consistent interface for the modern cloud. These native providers are automatically generated from cloud provider APIs and resource models that are maintained directly by cloud provider service teams. This enables native providers to offer 100% API coverage, same-day support for new features, the exact same API designed by the provider service team, and fewer bugs and reliability issues. This makes native providers the best way to manage infrastructure in the supported cloud platforms, combining the benefits of Pulumi Infrastructure as Code with the resource models and pace of delivery of the cloud provider themselves.

Along with Pulumi 3.0, the new Azure Native provider, originally released in preview last year, is now generally available as version 1.0. The Azure Native provider is built on top of the Azure Resource Manager REST API specifications developed and maintained by Azure service teams. It supports more than 2x the number of resources, and 3x the number of resource properties as the classic Azure provider. Azure Native also supports every version of each resource API for maximum compatibility with your existing infrastructure. New versions of the Azure Native provider with support for the latest features of Azure are published every night, ensuring immediate availability of new features and improvements from the Azure service teams.

Today, we’re also excited to be releasing a preview of the new Google Cloud Native provider. This native provider offers 100% coverage of the Google Cloud platform, with 70% more resource properties supported compared to the existing provider. It also supports every version of each supported resource, including alpha, beta, and GA Google Cloud offerings.

👉 Read more in the Pulumi Native Providers blog post and the Pulumi Google Native Provider blog post. And of course, we haven’t forgotten AWS - an AWS Native provider is expected later this year.

Pulumi Packages and Multi-Language Components

Pulumi Packages are the core technology that enables cloud infrastructure resource provisioning to be defined once, and made available to users in all languages supported by Pulumi. If you’ve ever used a cloud provider in Pulumi, you’ve used a Pulumi Package. With Pulumi 3.0, we are extending Pulumi Packages to also support building and packaging Pulumi Components: higher-level abstractions over cloud infrastructure patterns described as normal Pulumi programs. This makes it easy to take a Pulumi program that defines a collection of resources and turn it into something reusable: a Pulumi Package that can be published and used by all Pulumi users, across all languages supported by Pulumi.

One example of building and packaging Pulumi Components as a Pulumi Package is the Pulumi EKS package. We originally released a component for easily building and managing production-ready EKS clusters nearly two years ago, but it was only available for Node.js users of Pulumi as an npm package. This component has been incredibly popular, as it abstracts away a lot of complexity in all the moving pieces that need to be managed to stand up an EKS cluster across both AWS resources and in-cluster Kubernetes resources. Recently, we turned this component into a Pulumi Package, making it available to users in Python, Go, and .NET as well, with just a single implementation. Pulumi EKS is just the first example of applying this technology; there’s a lot more in the works here in the coming months!

import pulumi
import pulumi_eks as eks

cluster = eks.Cluster("my-cluster")

pulumi.export("kubeconfig", cluster.kubeconfig)

package main

import (
 "github.com/pulumi/pulumi-eks/sdk/go/eks"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 cluster, err := eks.NewCluster(ctx, "my-cluster", nil)
 if err != nil {
 return err
 }
 ctx.Export("kubeconfig", cluster.Kubeconfig)
 return nil
 })
}

import * as eks from "@pulumi/eks";

const cluster = new eks.Cluster("my-cluster");

export const kubeconfig = cluster.kubeconfig;

using Pulumi;
using Pulumi.Eks;

class MyStack : Stack
{
 [Output("kubeconfig")]
 public Output<object> Kubeconfig { get; set; }

 public MyStack()
 {
 var cluster = new Cluster("my-cluster");

 Kubeconfig = cluster.Kubeconfig;
 }
}

All Pulumi packages define a common Pulumi schema, which describes the resource model the package exposes and can be used to generate language-specific SDKs and documentation. There are several different types of Pulumi Package:

Native Pulumi Provider Package: Use the full features of the Pulumi resource model to create a provider for a new cloud platform. Examples: the kubernetes, azure-native, and google-native packages.
Bridged Provider Package: Take an existing resource provider from another supported ecosystem (like a Terraform provider), and bridge it to be exposed as a Pulumi Package. Examples:the aws, tls, and cloudflare packages.
Component Package: Write a Pulumi Component in your language of choice and expose it to users in all languages supported by Pulumi. Example: the eks package.

Together, these options provide a combination of breadth and depth for the kinds of resources and components that can be exposed to Pulumi developers via Pulumi Packages.

👉 Read more in the Pulumi Packages blog post.

Improved Go and Python SDKs

Pulumi’s language SDKs bring the power of cloud infrastructure management into your language of choice in a natural and productive way, enabling you to use all of your existing software engineering knowledge and tools for your cloud infrastructure projects as well. In Pulumi 3.0, we’ve made improvements to the Go and Python SDKs to improve developer productivity and enable additional tooling.

For Go users, Pulumi 3.0 reduces binary sizes by 4x and improves build times by up to 3x, leading to a much faster infrastructure development loop. Along with this, the Go APIs for working with Pulumi Output values have been simplified to provide a single way of transforming outputs instead of three different options. This makes the Pulumi Go SDK more consistent and easier to get started with. You can learn more about these changes in the Pulumi 3.0 migration guide.

For Python users, Pulumi 3.0 brings a new, more strongly typed way to provide inputs to cloud resources, ensuring that users get completion lists, type checking, documentation, and more within their IDE of choice. Along with this, some inconsistencies in input property naming have also been removed; now, all property names follow a consistent and predictable naming convention. You can learn more about these changes in the Pulumi 3.0 migration guide.

New Integrations and Features in the Pulumi Service

In addition to the Pulumi 3.0 open source release, we’re also making available new features for users of the Pulumi Service: the simple and easy to use default backend for all Pulumi users, with additional features for teams and enterprises.

Integrating Pulumi into CI/CD systems is a key step in realizing the full benefits of modern infrastructure as code, and we’ve worked to make this easier than ever. Since 2.0, we’ve added partner integrations with Spinnaker, GitHub Actions and CodeFresh as well as a new Pulumi Kubernetes Operator. In addition, we’ve introduced a new CI/CD Assistant to the service to help configure version control and CI/CD integration for your Pulumi projects with just a few clicks.

For enterprise users, we’ve added several enhancements to SAML SSO and SCIM support. Pulumi’s SSO support now works seamlessly with Okta (as well as Google Identity, Azure Active Directory and more), users and groups can now be synced via SCIM support in your SAML provider, and new Team Admin roles make team management simpler for large organizations.

Lastly, the Pulumi Service now offers a dashboard landing page which brings insights about your infrastructure front-and-center whether you are an individual user, or part of a large organization.

👉 Read more in the Pulumi CI/CD Assistant blog post and the SAML SSO and SCIM blog post.

Keep exploring

Pulumi 3.0 is a significant new milestone for the Pulumi project and lays the foundations for the Pulumi Cloud Engineering Platform with support for the very best of the cloud via new Pulumi Native Providers, support for a unified ecosystem of reusable infrastructure components with Pulumi Packages, and support for a new era of cloud infrastructure management scenarios enabled by Pulumi Automation API. Each of these is just a start: there’s so much more to come on top of these foundations and we’re excited to work with our customers and community on the next wave of innovation in Cloud Engineering!

To learn more about each of the new areas - check out these blog posts with more details:

You can get started with Pulumi 3.0 today, or check out the Pulumi 3.0 migration guide to update your existing Pulumi projects to Pulumi 3.0.

We can’t wait to see what you build with Pulumi 3.0!

Easily Create and Manage AWS EKS Kubernetes Clusters

Luke Hoban — Thu, 28 Jan 2021 00:00:00 +0000

Provisioning, managing, and monitoring a Kubernetes cluster is not easy. AWS now offers EKS to reduce that burden – but it’s still difficult to get up and running. Pulumi’s infrastructure as code SDKs can help! We can provision an entire EKS cluster with a single CLI command, thanks to the @pulumi/eks package. Let’s see how.

Updated January 2021 to include Python, .NET, and Go support.

Updated October 2024 to Version 3.0

To get started, download the free and open source Pulumi SDK, which includes the pulumi CLI we’ll use below. If you don’t have the AWS CLI installed and configured, set that up too. You’ll need AWS credentials to access your AWS account and provision resources.

From here, there are two ways to proceed:

Provision a new cluster with a single CLI command
Use infrastructure as code to manage clusters … including Kubernetes resources themselves!

Installing Amazon EKS with Pulumi

With the Pulumi CLI installed and configured, a single command will provision your cluster:

$ pulumi new https://github.com/pulumi/apps/eks

This template will walk you through configuring a few settings such as the region to deploy, the instance type, and number of cluster nodes.

The CLI will first show you a preview of the AWS and Kubernetes resources that will be deployed to create the cluster, before actually performing any operation. Once you have reviewed and selected “Yes,” the cluster will begin deploying. The terminal output will look like this:

Following these steps will provide a complete managed Amazon EKS cluster, configured similarly to the steps outlined in the Amazon EKS Getting Started guide, including optional support for Amazon EBS-backed StorageClasses and access to the Kubernetes Dashboard. Pulumi replaces all of these tedious, manual steps with a straightforward description of your desired state, and then makes it so!

Once the deployment is complete – it will take 10-20 minutes for AWS to provision these resources – you can grab the kubeconfig.json needed to work with this cluster and connect to it with kubectl:

$ pulumi stack output kubeconfig > kubeconfig.json
$ KUBECONFIG=./kubeconfig.json kubectl get nodes

Reusable AWS EKS component

The Amazon EKS installer we just saw is built on top of a reusable eks.Cluster component that is available in Pulumi’s infrastructure as code solution. In fact, the command above created an instance of this component with reasonable defaults. If you prefer, you can do it by hand and programmatically. Here are examples in each of Pulumi’s supported languages:

import * as eks from "@pulumi/eks";

// Create an EKS cluster.
const cluster = new eks.Cluster("cluster", {
 instanceType: "t2.medium",
 desiredCapacity: 2,
 minSize: 1,
 maxSize: 2,
});

// Export the cluster's kubeconfig.
export const kubeconfig = cluster.kubeconfig;

import pulumi
import pulumi_eks as eks

# Create an EKS cluster.
cluster = eks.Cluster(
 "cluster",
 instance_type="t2.medium",
 desired_capacity=2,
 min_size=1,
 max_size=2,
)

# Export the cluster's kubeconfig.
pulumi.export("kubeconfig", cluster.kubeconfig)

using System;
using System.Threading.Tasks;
using Pulumi;
using Pulumi.Eks.Cluster;

class EksStack : Stack
{
 public EksStack()
 {
 // Create an EKS cluster.
 var cluster = new Cluster("cluster", new ClusterArgs
 {
 InstanceType = "t2.medium",
 DesiredCapacity = 2,
 MinSize = 1,
 MaxSize = 2,
 });

 // Export the cluster's kubeconfig.
 this.Kubeconfig = cluster.Kubeconfig;
 }

 [Output("kubeconfig")]
 public Output<string> Kubeconfig { get; set; }
}

class Program
{
 static Task<int> Main(string[] args) => Deployment.RunAsync<EksStack>();
}

package main

import (
 "github.com/pulumi/pulumi-eks/sdk/go/eks/cluster"
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // Create an EKS cluster.
 cluster, err := cluster.NewCluster(ctx, "cluster",
 cluster.ClusterArgs{
 InstanceType: pulumi.String("t2.medium"),
 DesiredCapacity: pulumi.Int(2),
 MinSize: pulumi.Int(1),
 MaxSize: pulumi.Int(2),
 },
 )
 if err != nil {
 return err
 }

 // Export the cluster's kubeconfig.
 ctx.Export("kubeconfig", cluster.Kubeconfig)

 return nil
 })
}

Adding resources, and deploying Helm charts

You can extend or modify this program to change the details of the Amazon EKS cluster, or to install additional AWS or Kubernetes resources to associate with the cluster. For example, if applications running in the cluster need access to an Amazon S3 bucket, create a new Bucket resource:

const bucket = new aws.s3.Bucket("assets");

bucket = aws.s3.Bucket("assets")

var bucket = new Aws.S3.Bucket("bucket");

bucket, err := s3.NewBucket(ctx, "bucket", &s3.BucketArgs{})

Or, to install Kubernetes applications automatically into the Amazon EKS cluster (such as WordPress), add them using the Pulumi Kubernetes provider pointing at the newly provisioned EKS cluster:

const wordpress = new k8s.helm.v3.Chart("wpdev", {
 version: "9.6.0",
 chart: "wordpress",
 fetchOpts: {
 repo: "https://charts.bitnami.com/bitnami",
 },
}, { providers: { kubernetes: cluster.provider }});

wordpress = Chart(
 "wpdev",
 ChartOpts(
 chart="wordpress",
 version="9.6.0",
 fetch_opts=FetchOpts(
 repo="https://charts.bitnami.com/bitnami",
 ),
 ),
 opts=ResourceOptions(provider=cluster.provider),
)

var wordpress = new Chart("wpdev", new ChartArgs
{
 Chart = "wordpress",
 Version = "9.6.0",
 FetchOptions = new ChartFetchArgs
 {
 Repo = "https://charts.bitnami.com/bitnami"
 }
}, new CustomResourceOptions
{
 Provider = cluster.Provider,
});

wordpress, err := helm.NewChart(ctx, "wpdev", helm.ChartArgs{
 Chart: pulumi.String("wordpress"),
 Version: pulumi.String("9.6.0"),
 FetchArgs: helm.FetchArgs{
 repo: pulumi.String("https://charts.bitnami.com/bitnami"),
 },
}, pulumi.Provider(cluster.Provider))

Pulumi makes it possible to define the cluster, AWS resources, and Kubernetes objects needed to bootstrap your entire Kubernetes environment, all in one place. They can then be versioned together and exact replicas of the environment can be created independently for operations such as testing or disaster recovery.

As usual, by expressing these resources in code instead of in YAML, you get many software engineering benefits: improved tooling, the ability to refactor using your favorite IDE, the ability to create and share reusable components, and strong typing to validate correctness up front.

What’s happening under the hood?

To make this whole process buttery smooth, the Pulumi EKS component takes care of setting up all the pieces needed for an Amazon EKS cluster:

Creating an EKS Service Role
Optionally creating a new Virtual Private Cloud (VPC)
Creating an Amazon EKS cluster
Configuring a Kubernetes provider with access to the Amazon EKS cluster
Launching worker nodes in an Auto Scaling group to join the cluster
Installing a ConfigMap into the cluster to allow new Amazon EC2 worker nodes to join
Installing a StorageClass into the cluster to allow provisioning Amazon EBS-backed PersistentVolumes
Constructing the kubeconfig needed to access the cluster
Optionally installing additional Kubernetes YAML and/or Helm charts

This involves steps that provision resources in AWS, steps that provision resources in Kubernetes, and other custom computation to coordinate these steps. In addition, during the deployment Pulumi also provides complex coordination between these different steps.

Managing Pulumi Deployments

Once you have deployed your Amazon EKS cluster and Kubernetes resources to AWS, you can manage the deployment on app.pulumi.com, including deep links into the AWS console and Kubernetes dashboard to get real-time insights on your cluster.

All of the provisioned resources across AWS and Kubernetes are available from this single view, along with an audit history of deployments to this stack. The stack exports are also published in this same dashboard, so that key outputs, like your kubeconfig, are easily available and can be shared with other users in your organization.

Additional management features, including RBAC, CI/CD integrations, and rich resource graph visualization, are also available via the Pulumi app, including Team and Enterprise Editions for production teams of all sizes.

Get Started

We’ve seen how Pulumi takes care of the heavy lifting with AWS EKS so that you don’t have to. You can use Pulumi to easily deploy new clusters, managed AWS resources, and Kubernetes apps, all with the same powerful toolchain. Support is also available for Azure AKS, Google GKE, DigitalOcean’s Managed Kubernetes, and custom clusters (including Minikube).

For more information:

Getting Started with Amazon EKS Distro & Pulumi

Luke Hoban — Tue, 01 Dec 2020 11:30:00 -0800

As Kubernetes grows in popularity, the number of options for Kubernetes users continues to increase. Providers of managed Kubernetes offerings will often learn lessons about operating large numbers of clusters at scale; it’s increasingly common that they will contribute this knowledge back to the ecosystem, allowing those organizations who need more control and flexibility to reap the benefits.

With the announcement of the Amazon EKS Distro during AWS re:Invent, the Amazon EKS team has contributed back to the cloud-native community in a big way. In this post, we’ll take a brief look at what the Amazon EKS Distro is, explore why you might choose this over current managed service offerings and finally, explore how you can get started with the Amazon EKS Distro on day 1 using Pulumi.

Kubernetes is a complex, distributed system that consists of many components that operate together to create the Kubernetes experience. If you’re using a managed Kubernetes service, often these components are hidden away from you as a user to reduce the operational complexity and ease the usage. The managed service provider will take care of tricky problems like upgrades and taking care of patching and compliance.

The Amazon EKS Distro is the packaging of many of the components needed to run a Kubernetes cluster distributed in an opinionated way by the Amazon EKS team. Parts of a working Kubernetes cluster like the scheduler, API server and the backing database (etcd) have been built into Docker images based on Amazon Linux. The images have backported security fixes for each Kubernetes version and are drop-in replacements for the upstream Kubernetes distribution, meaning you get all the benefits of Kubernetes, with the added support of knowing you’re running a verified and supported version from the Amazon EKS team.

Why use the Amazon EKS Distro?

If you’re already a user of Amazon EKS, you’re already getting all of the benefits provided by the Amazon EKS Distro. However, many users aren’t utilizing the managed Kubernetes cluster provided by the Amazon EKS team, either through choice or because of external requirements. The release of the Amazon EKS Distro means that those organizations can get the benefits provided by EKS, such as secure Docker images, backported security fixes and a single upstream vendor, without having to operate with the constraints a managed service in AWS might provide.

Provisioning an Amazon EKS Distro Cluster with Pulumi

Installing an Amazon EKS Distro cluster involves bootstrapping a Kubernetes cluster but with the images provided by the Amazon EKS team. The simplest way to provision a Kubernetes cluster is using kops and the Amazon EKS Distro team provides instructions on how to replace the default Kubernetes components with the Amazon EKS Distro built parts.

It’s possible to get all of the benefits of provisioning a Kubernetes cluster with kops while also taking part in the Pulumi resource lifecycle. Pulumi offers support for running arbitrary commands using dynamic providers (available in the JavaScript, Typescript and Python SDKs) and you can register the kops commands used to create and destroy clusters as dynamic provider callbacks. Once your Kubernetes cluster has been created, you can provision Kubernetes resources using Pulumi’s Kubernetes provider, using the Pulumi resource model to set dependencies on the created cluster.

Creating an Amazon EKS Distro cluster with Pulumi

Using a Pulumi dynamic provider, you can define an Amazon EKS Distro cluster using just a few lines of code:

import * as eksdistro from "./eksdistro";

const store = new aws.s3.Bucket("kops-state-store");
const cluster = new eksdistro.Cluster("cluster", {
 name: "luke.cluster.pulumi-demos.net",
 state: pulumi.interpolate`s3://${store.id}`,

Behind the scenes, we’ve defined a Typescript dynamic provider, which renders a kops configuration file and renders it to disk using mustache. This kops configuration file references the Amazon EKS Distro images.

This dynamic provider registers callbacks for creating, updating and deleting the cluster and also registers the created cluster’s kubeconfig as a Pulumi output, which can be passed to Kubernetes resources in Pulumi.

import * as pulumi from "@pulumi/pulumi";
import * as cp from "child_process";
import * as path from "path";
import * as fs from "fs";
import * as mustache from "mustache";
import * as tmp from "tmp";

interface ClusterProviderArgs {
 // KOPS_STATE_STORE
 state: string;
 // KOPS_CLUSTER_NAME
 name: string;
}

const clusterYamlTemplate = fs.readFileSync(path.join(__dirname, "cluster.yaml")).toString();
const authenticatorYamlTemplate = fs.readFileSync(path.join(__dirname, "aws-iam-authenticator.yaml")).toString();

const clusterprovider: pulumi.dynamic.ResourceProvider = {
 async create(inputs: ClusterProviderArgs): Promise<pulumi.dynamic.CreateResult> {
 const clusterYaml = mustache.render(clusterYamlTemplate, {
 CLUSTER_NAME: inputs.name,
 STATE_STORE: inputs.state,
 });
 console.log("kops create")
 const createOut = cp.execSync(`kops create --name ${inputs.name} --state ${inputs.state} -f -`, { input: clusterYaml });
 console.log(createOut.toString());
 console.log("kops create secret")
 const createSecretOut = cp.execSync(`kops create secret --name ${inputs.name} --state ${inputs.state} sshpublickey admin -i ~/.ssh/id_rsa.pub`);
 console.log(createSecretOut.toString());

 let outs = { ...inputs };
 try {
 const updateRes = await this.update!(inputs.name, {}, inputs);
 outs = { ...outs, ...updateRes.outs};
 } catch (err) {
 console.log(err);
 }

 return {
 id: inputs.name,
 outs,
 };
 },
 async update(id: pulumi.ID, olds: any, inputs: any): Promise<pulumi.dynamic.UpdateResult> {
 const clusterYaml = mustache.render(clusterYamlTemplate, {
 CLUSTER_NAME: inputs.name,
 STATE_STORE: inputs.state,
 });
 const authenticatorYaml = mustache.render(authenticatorYamlTemplate, {
 CLUSTER_NAME: inputs.name,
 });

 console.log("kops update cluster")
 const updateOut = cp.execSync(`kops update cluster --name ${inputs.name} --state ${inputs.state} --yes`);
 console.log(updateOut.toString());

 console.log("kops export kubecfg")
 const kubeConfigName = tmp.tmpNameSync();
 const exportKubeconfigOut = cp.execSync(`kops export kubecfg --name ${inputs.name} --state ${inputs.state} --kubeconfig ${kubeConfigName}`);
 const kubeconfig = fs.readFileSync(kubeConfigName).toString();
 console.log(exportKubeconfigOut.toString());

 // Needed to allow cluster to come available and DNS to propagate
 try {
 console.log("kops validate cluster")
 const validateOut = cp.execSync(`kops validate cluster --wait 2m --name ${inputs.name} --state ${inputs.state}`);
 console.log(validateOut.toString());

 console.log("kubectl apply -f aws-iam-authenticator.yaml");
 const authApplyOut = cp.execSync(`kubectl apply -f -`, { input: authenticatorYaml });
 console.log(authApplyOut.toString());
 } catch (err) {
 console.log(err);
 }

 return {
 outs: {
 ...inputs,
 kubeconfig,
 },
 }
 },
 async delete(id: pulumi.ID, inputs: any) {
 const clusterYaml = mustache.render(clusterYamlTemplate, {
 CLUSTER_NAME: inputs.name,
 STATE_STORE: inputs.state,
 });
 console.log("kops delete")
 const deleteOut = cp.execSync(`kops delete --name ${inputs.name} --state ${inputs.state} --yes -f -`, { input: clusterYaml });
 console.log(deleteOut.toString());
 },
}

export interface ClusterArgs {
 // KOPS_STATE_STORE
 state: pulumi.Input<string>;
 // KOPS_CLUSTER_NAME
 name: pulumi.Input<string>;
 keeper?: pulumi.Input<number>;
}

export class Cluster extends pulumi.dynamic.Resource {
 kubeconfig!: pulumi.Output<string>;
 constructor(name: string, args: ClusterArgs, opts?: pulumi.CustomResourceOptions) {
 super(clusterprovider, name, {
 ... args,
 kubeconfig: undefined,
 }, opts);
 }
}

Once the cluster is created, we can build a Kubernetes provider and register resources like any other Kubernetes cluster:


const k8sProvider = new k8s.Provider("provider", { kubeconfig: cluster.kubeconfig });
const pod = new k8s.core.v1.Pod("mypod", {
 spec: {
 containers: [{ name: "echo", image: "k8s.gcr.io/echoserver:1.4" }],
 }
}, { provider: k8sProvider });
export const kubeconfig = cluster.kubeconfig;

The full code for defining an Amazon EKS Distro cluster using Pulumi can be found in our examples repo

Wrap up

The Amazon EKS Distro opens lots of avenues for users needing more flexibility and control over their Kubernetes clusters while still providing the security, compliance and reliability required to feel confident in the clusters you’re building. With Pulumi, you can bootstrap Amazon EKS Distro clusters and leverage Pulumi’s best-in-class Kubernetes resource management while still benefiting from the power of Pulumi’s modern programming languages. Give it a try!

Authoring CrossGuard Policy with Open Policy Agent (OPA)

Luke Hoban — Fri, 14 Aug 2020 00:00:00 +0000

We’re excited to announce the addition of Open Policy Agent (OPA) Rego language support to Pulumi’s CrossGuard policy-as-code framework. This enables Pulumi CrossGuard policy to be authored in either JavaScript/TypeScript/Python or in the popular Rego language using OPA.

Pulumi’s CrossGuard policy-as-code framework provides the ability to author, apply and enforce policy directly as part of your Pulumi deployments. With the new support for OPA Rego, CrossGuard supports a broad spectrum of policy authoring options, from expressive imperative languages to a popular industry-standard declarative policy language.

OPA-based rules for CrossGuard get all the core benefits of Pulumi’s policy-as-code framework - policies can be run on previews to get warnings about errors before you even deploy, policies can produce either advisory or mandatory recommendations allowing flexibility in flagging and enforcing policy violations, and policies can be applied and enforced across an entire organization through the Pulumi Service.

What is Pulumi CrossGuard?

Pulumi CrossGuard is a policy-as-code framework for enforcing policy across Pulumi infrastructure deployments on any cloud or in Kubernetes. CrossGuard can be used to enforce policy for compliance, security, best practices or cost control.

Pulumi’s web console offers tools to manage and enforce CrossGuard policy across your organization, ensuring that every infrastructure project is held to a high bar on policy compliance.

CrossGuard policy is enforced during previews, ensuring that policy violations are found during development, not just at deployment time. At deployment, policies are enforced on each resource immediately before changes to that resource are applied. As a result, any change that would violate policy is rejected before being applied to your target cloud or Kubernetes environments.

What is OPA?

Open Policy Agent (OPA) provides policy-based control for cloud native environments. OPA provides a single authoring language (Rego) and policy runtime, which can be used to enforce policy in a wide variety of environments including Kubernetes, Envoy, Terraform Kafka, SQL, Linux - and now Pulumi!

The Rego language is declarative, easy to use, and expressive - along with good integration in tools like Visual Studio Code.

Using OPA with Pulumi CrossGuard

To get started with OPA and CrossGuard, start by downloading and installing Pulumi, then get the OPA plugin with:

$ pulumi plugin install analyzer policy-opa v0.0.2
[analyzer plugin policy-opa-0.0.2] installing
Downloading plugin: 6.11 MiB / 6.11 MiB [===========================] 100.00% 0s
Moving plugin... done.

You are now ready to apply OPA-based policies to your Pulumi deployments. Create a folder for your policies named policy-kubernetes and place two files in it - PulumiPolicy.yaml and kubernetes.rego.

Your PulumiPolicy.yaml should look like this. Note that it specifies the runtime to use for the policy rules is opa.

description: A minimal Policy Pack for Kubernetes using OPA.
runtime: opa

Your kubernetes.rego should look like this.

package kubernetes

name = input.metadata.name

labels {
 input.metadata.labels["app.kubernetes.io/name"]
 input.metadata.labels["app.kubernetes.io/instance"]
 input.metadata.labels["app.kubernetes.io/version"]
 input.metadata.labels["app.kubernetes.io/component"]
 input.metadata.labels["app.kubernetes.io/part-of"]
 input.metadata.labels["app.kubernetes.io/managed-by"]
}

deny[msg] {
 input.kind = "Deployment"
 not labels
 msg = sprintf("%s must include Kubernetes recommended labels: https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/#labels ", [name])
}

The rule above uses the standard OPA Rego language to express a rule that enforces that any Deployment resource in Kubernetes must have a set of defined labels applied to it. All OPA rules used in Pulumi will be applied to an input object, which is a JSON representation of the resource inputs for the Pulumi resource being validated. In particular, for a Kubernetes Deployment resource, this will have top level kind, apiVersion, metadata, and spec fields which can be used to query aspects of the resource for policy enforcement.

To test our rule, we can create a new Pulumi Kubernetes project and deploy it with the policy enforced.

First we create a new project kubeapp in a sibling folder:

$ cd .. && mkdir kubeapp && cd kubeapp
$ pulumi new kubernetes-typescript

This will create a project that deploys a single Kubernetes Deployment (but without our required labels!).

import * as k8s from "@pulumi/kubernetes";
import * as kx from "@pulumi/kubernetesx";

const appLabels = { app: "nginx" };
const deployment = new k8s.apps.v1.Deployment("nginx", {
 spec: {
 selector: { matchLabels: appLabels },
 replicas: 1,
 template: {
 metadata: { labels: appLabels },
 spec: { containers: [{ name: "nginx", image: "nginx" }] }
 }
 }
});
export const name = deployment.metadata.name;

We can deploy this to any Kubernetes cluster (configure your kubectl context locally to point at the desired cluster) with our policy applied with:

$ pulumi up --policy-pack ../policy-kubernetes
Previewing update (dev):
Type Name Plan Info
+ pulumi:pulumi:Stack simple-kubernetes-dev create 1 error
+ └─ kubernetes:apps:Deployment nginx create
Diagnostics:
pulumi:pulumi:Stack (simple-kubernetes-dev):
error: preview failed
Policy Violations:
[mandatory] kubernetes v0.0.1 deny (nginx: kubernetes:apps/v1:Deployment)
nginx-me0llhgr must include Kubernetes recommended labels: https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/#labels

Note that the preview fails because the policy was violated, preventing me from deploying this resource to my cluster!

The policy defined above has a “mandatory” policy, defined in OPA with deny. We can also define “advisory” policies using warn rules. For example, if we changed our policy from deny to warn and rerun - we will see the preview succeed:

$ pulumi up --policy-pack ../policy-kubernetes
Previewing update (dev2):
Type Name Plan
+ pulumi:pulumi:Stack simple-kubernetes-dev2 create
+ └─ kubernetes:apps:Deployment nginx create
Policy Violations:
[advisory] kubernetes v0.0.1 warn (nginx: kubernetes:apps/v1:Deployment)
nginx-ylgvoqpk must include Kubernetes recommended labels: https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/#labels

In the previous examples, we manually applied our policy to our deployment with the --policy-pack option. But we can also publish our policy pack to the Pulumi Service and configure it to be enforced across our organization. To do, go back to the policy folder and run pulumi policy publish.

$ cd ../policy-kubernetes
$ pulumi policy publish
Obtaining policy metadata from policy plugin
Compressing policy pack
Uploading policy pack to Pulumi service
Publishing "kubernetes" to "lukehoban"
Published as version 2
Permalink: https://app.pulumi.com/lukehoban/policypacks/kubernetes/2

You can now navigate into the Pulumi Service and configure the new policy to be part of the “default” policy pack (or create a new one), and applied to all stacks in your Organization (or select specify stacks to enforce the policy on).

We can now run our deployment without the --policy-pack argument and our policy will still be enforced on this deployment (and any others done by any other users in our organization)!

$ pulumi up
Previewing update (dev2):
Installing policy pack kubernetes 2...
Finished installing policy pack
Type Name Plan
+ pulumi:pulumi:Stack simple-kubernetes-dev2 create
+ └─ kubernetes:apps:Deployment nginx create
Policy Violations:
[advisory] kubernetes v0.0.1 warn (nginx: kubernetes:apps/v1:Deployment)
nginx-q4ilsuqq must include Kubernetes recommended labels: https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/#labels
Do you want to perform this update? yes
Updating (dev2):
Type Name Status
+ pulumi:pulumi:Stack simple-kubernetes-dev2 created
+ └─ kubernetes:apps:Deployment nginx created
Policy Violations:
[advisory] kubernetes v0.0.1 warn (nginx: kubernetes:apps/v1:Deployment)
nginx-6ooayw4j must include Kubernetes recommended labels: https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/#labels
Outputs:
name: "nginx-6ooayw4j"
Permalink: https://app.pulumi.com/lukehoban/simple-kubernetes/dev2/updates/1

We can now navigate to the URL for our update and see that the policy was enforced. Note the badge showing that this deployment was protected by policy enforcement and the list of policy violations.

Using OPA Rego with Pulumi

OPA-based rules are easy to write for Pulumi. The input object represents each individual resource that will be deployed by Pulumi, and the rules are applied independently to each Pulumi resource when it is deployed. The schema for each resource is the same as the properties available when defining the resource. Critically, although we highlighted Kubernetes above, Pulumi’s resource model spans any cloud or Kubernetes, with providers for more than 50 Cloud and SaaS platforms available. And OPA-based rules can target any or all of these in a single deployment.

For example - the policy file above could be expanded to include:

# Enforce that AWS EC2 Instance public IP is not enabled
warn[msg] {
 input.associatePublicIpAddress
 msg := "Associating public IP addresses is discouraged"
}

# Enforce that Kubernetes Pod images must be published by `hooli.com`
deny[msg] {
 input.kind == "Pod"
 image := input.spec.containers[_].image
 not startswith(image, "hooli.com/")
 msg := sprintf("image '%v' comes from untrusted registry", [image])
}

With support in Visual Studio Code, authoring both your infrastructure and policy in a rich IDE using Pulumi is also possible.

Conclusion

OPA support in Pulumi CrossGuard enables authoring cloud infrastructure policy using a familiar and popular policy authoring language while leveraging Pulumi’s CrossGuard platform for policy enforcement from development to production. Along with existing support for TypeScript, JavaScript and Python - Pulumi CrossGuard offers a great breadth of authoring languages to meet the needs of any infrastructure team.

Check out the open source pulumi-policy-opa project to learn more about the OPA integration with CrossGuard. And get started with Pulumi for your modern cloud infrastructure and policy today.

Announcing New Pulumi Open Source Projects for Kubernetes

Luke Hoban — Wed, 12 Aug 2020 00:00:00 +0000

Today, we’re excited to announce several new open source projects that advance Pulumi’s Kubernetes support. These projects and features have been developed while helping leading cloud native engineering teams like Snowflake, Lemonade, and Mercedes-Benz go into production with Kubernetes, and include new deployment automation options, improved ecosystem integrations, and tools to make it easier than ever to adopt Pulumi for new and existing projects.

These new capabilities include:

A new in-cluster Pulumi Kubernetes Operator provides a Kubernetes-native GitOps experience for managing infrastructure on any cloud or Kubernetes cluster.
A kube2pulumi tool for converting YAML manifests to Pulumi Python, TypeScript, Go, or .NET brings Pulumi’s software engineering capabilities for Infrastructure-as-Code to bear on complex Kubernetes applications.
Support for authoring Open Policy Agent (OPA)-based policies in Pulumi’s CrossGuard policy-as-code framework and enforcing rules across an organization.
A crd2pulumi tool to create strongly-typed APIs for any Kubernetes CRD, making it easy to leverage the full ecosystem of API extensions in the Cloud Native community.

Along with additional significant new features launched in the last few months:

A Pulumi plugin for Spinnaker enabling teams to use the same cloud native continuous delivery platform for both application and infrastructure deployments.
Support for deploying Helm 3 and Kustomize as part of Pulumi programs using the core Pulumi Kubernetes library.

Whether you are designing your cloud native delivery platform, looking to enforce policy across cloud infrastructure and Kubernetes, or looking to bring robust software engineering practices to your Kubernetes workloads, these features continue to make Pulumi a better way to Kubernetes!

Pulumi + Kubernetes

Pulumi is an Infrastructure as Code platform designed from the ground up for the modern cloud. That means deep support for containers, serverless, and Kubernetes, using your favorite programming languages to define, test, and manage your cloud infrastructure. Pulumi supports easy provisioning of Kubernetes clusters in many different managed platforms, including AWS, Azure, GCP, Digital Ocean, Rancher, Linode, and Civo. Additionally, Pulumi can manage resources inside a Kubernetes cluster using the same powerful tools, replacing the need for brittle and unscalable YAML configuration. Best of all, Pulumi combines these into a single cohesive programming model across your cloud infrastructure in managed cloud services and in Kubernetes.

Kubernetes has been at the center of many Pulumi-enabled cloud modernization projects, including provisioning managed Kubernetes clusters, connecting them to supporting cloud infrastructure, enabling platform teams to configure and manage platforms on top of Kubernetes, and allowing teams to replace unmaintainable piles of YAML with code and software-based deployment processes. Hundreds of modern cloud engineering teams have been able to accelerate the move to the cloud and Kubernetes thanks to Pulumi.

Let’s take a look now at each of the new Pulumi superpowers for working with Kubernetes!

Pulumi Kubernetes Operator

Pulumi integrates into more than a dozen CI/CD and delivery systems, automating the delivery of cloud infrastructure changes to AWS, Azure, GCP, Kubernetes, and more. This includes traditional CI tools like Jenkins, managed CI/CD platforms like Azure DevOps, and cloud native continuous delivery platforms like Spinnaker (see below for more details!). At the same time, we have heard feedback from Pulumi users looking to automate their delivery processes inside Kubernetes using a GitOps approach to delivery - both for Kubernetes workloads and for cloud infrastructure in any Pulumi-supported Cloud Provider in the Registry.

We’re excited to announce the new pulumi-kubernetes-operator open source project which provides a new Kubernetes-native option for infrastructure delivery. The Pulumi Kubernetes Operator allows you to specify the cloud infrastructure to deploy using a new Stack CustomResource in your Kubernetes cluster. The operator continuously drives the state of your cloud resources to align with the specified Stack. Stack resources can point at any Pulumi program in any Git repository, and use that program to define the desired state of the infrastructure. These stacks provide the full flexibility of Pulumi, including the ability to enforce policy using CrossGuard policy-as-code coupled with the benefits of the declarative Kubernetes resource model. You can create these Stack resources in normal Kubernetes YAML, or use Pulumi to manage the Kubernetes resources as well!

Creating a Stack is simple: provide the stack name, project repo, and a commit (support for branches coming soon!), and the operator starts driving toward the specified state. Stacks can be used directly, or as part of higher-level abstractions to manage your infrastructure and applications. The [Pulumi Service](/docs/pulumi-cloud/ents these capabilities by providing visibility into the updates, resources under management, and policy enforcement of your stacks.

apiVersion: pulumi.com/v1alpha1
kind: Stack
metadata:
 name: s3-bucket-stack
spec:
 stack: acmecorp/s3-op-project/dev
 projectRepo: https://github.com/joeduffy/test-s3-op-project
 commit: cc5442870f1195216d6bc340c14f8ae7d28cf3e2
 config:
 aws:region: us-east-2
 initOnCreate: true
 accessTokenSecret: pulumi-api-secret
 envSecrets:
 - pulumi-aws-secrets

Pulumi Spinnaker Plugin

Many teams using Spinnaker have asked about automating the delivery of their cloud infrastructure changes using Pulumi through the same great continuous delivery platform that Spinnaker offers for application deployments. As launch partners for the new Spinnaker Plugin model announced in June, Pulumi and Spinnaker worked closely together to deliver complete support for Pulumi inside the Spinnaker continuous deployment platform.

You can check out the Pulumi Spinnaker integration in this presentation from Spinnaker Live and in the Spinnaker guide in the Pulumi docs.

YAML to Pulumi

Kubernetes YAML offers a simple lowest-common-denominator approach for authoring configuration to apply to a Kubernetes cluster. But it is cumbersome to manage and difficult to scale as the complexity of the Kubernetes infrastructure and workloads increases. Pulumi provides a compelling solution to manage that complexity - apply the software engineering benefits of your favorite programming language - including Python, TypeScript, .NET, and Go! Using these languages to manage Kubernetes infrastructure provides useful abstraction (reusable functions and classes), simple configuration (parameters and interfaces), strong typing and tooling (IDEs, error checking as you type), and testing (bring your favorite unit testing framework).

With today’s release of kube2pulumi, it’s easier than ever to get all of these benefits by migrating your existing Kubernetes YAML (or JSON) into Pulumi code in Python, TypeScript, .NET or Go. The complete Kubernetes API surface area is available from Pulumi, so the code generated by kube2pulumi supports every feature of Kubernetes you might be using today. Convert one resource or an entire directory of manifests at a time! Once converted, deploy to your cluster with a simple pulumi up.

Your existing YAML:

apiVersion: v1
kind: Namespace
metadata:
 name: foo

Becomes strongly-typed code in Pulumi:

import * as pulumi from "@pulumi/pulumi";
import * as kubernetes from "@pulumi/kubernetes";

const fooNamespace = new kubernetes.core.v1.Namespace("fooNamespace", {
 metadata: {
 name: "foo",
 },
});

import pulumi
import pulumi_kubernetes as kubernetes

foo_namespace = kubernetes.core.v1.Namespace("fooNamespace",
 metadata={
 "name": "foo",
 })

using Pulumi;
using Kubernetes = Pulumi.Kubernetes;

class MyStack : Stack
{
 public MyStack()
 {
 var fooNamespace = new Kubernetes.Core.V1.Namespace("fooNamespace", new Kubernetes.Types.Inputs.Core.V1.NamespaceArgs
 {
 Metadata = new Kubernetes.Types.Inputs.Meta.V1.ObjectMetaArgs
 {
 Name = "foo",
 },
 });
 }

}

package main

import (
 corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v2/go/kubernetes/core/v1"
 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v2/go/kubernetes/meta/v1"
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 _, err := corev1.NewNamespace(ctx, "fooNamespace", &corev1.NamespaceArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("foo"),
 },
 })
 if err != nil {
 return err
 }
 return nil
 })
}

And best of all - you can try it out yourself at pulumi.com/kube2pulumi to see how easy it is to bring your Kubernetes YAML to Pulumi!

Strongly-Typed Kubernetes CRDs

As the Kubernetes ecosystem grows, projects across the Cloud Native landscape build powerful new extensions on top of the core Kubernetes resource model via CustomResources. These Custom Resources extend the Kubernetes API to provide first-class support for everything from Service Meshes to Storage extensions to Observability infrastructure and much more (including the Pulumi Kubernetes Operator!). While Pulumi offers a strongly-typed API for 100% of the core Kubernetes API, so far it has only exposed these extensions via the weakly-typed and general-purpose CustomResource API in Kubernetes.

Today’s launch of the crd2pulumi tool brings the same rich support for strongly-typed APIs in TypeScript and Go (Python and .NET coming soon!) to all of these extensions. Users can generate a strongly-typed programming language API directly from the Custom Resource Definition’s OpenAPI spec, and regenerate it if and when that spec is updated. Any code using the CRD benefits from IDE tooling, strong typing, and the ease of creating new custom abstractions. These benefits become more pronounced as the complexity of the infrastructure managed by the custom resource increases. Many CRDs encourage design patterns involving fine-grained and frequently-updated resource definitions, and Pulumi’s modern infrastructure as code approach greatly simplifies managing an ever-growing ecosystem of new APIs.

For example, the commonly used cert-manager CRD extends Kubernetes with the ability to easily configure Certificate Authorities and request certificates. After generating API wrappers with crd2pulumi, users can interact with the Cert Manager API from code:

import * as certificates from "./certificates"

// Register the Certificate CRD.
new certificates.CertificateDefinition("certificate");

// Instantiate a Certificate resource.
new certificates.v1beta1.Certificate("example-cert", {
 metadata: {
 name: "example-com",
 },
 spec: {
 secretName: "example-com-tls",
 duration: "2160h",
 renewBefore: "360h",
 commonName: "example.com",
 dnsNames: [
 "example.com",
 "www.example.com",
 ],
 issuerRef: {
 name: "ca-issuer",
 kind: "Issuer",
 }
 }
});

Coming soon along with overall improved Python typing support!

Coming soon!

package main

import (
 certv1b1 "goexample/certificates/v1beta1"

 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v2/go/kubernetes/meta/v1"
 "github.com/pulumi/pulumi-kubernetes/sdk/v2/go/kubernetes/yaml"
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {

 // Register the Certificate CRD.
 _, err := yaml.NewConfigFile(ctx, "my-certificate-definition",
 &yaml.ConfigFileArgs{
 File: "certificate.yaml",
 },
 )
 if err != nil {
 return err
 }

 // Instantiate a Certificate resource.
 _, err = certv1b1.NewCertificate(ctx, "example-cert", &certv1b1.CertificateArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("example-com"),
 },
 Spec: certv1b1.CertificateSpecArgs{
 SecretName: pulumi.String("example-com-tls"),
 Duration: pulumi.String("2160h"),
 RenewBefore: pulumi.String("360h"),
 CommonName: pulumi.String("example.com"),
 DnsNames: pulumi.StringArray{
 pulumi.String("example.com"),
 pulumi.String("www.example.com"),
 },
 IssuerRef: certv1b1.CertificateSpecIssuerRefArgs{
 Name: pulumi.String("ca-issuer"),
 Kind: pulumi.String("Issuer"),
 },
 },
 })
 if err != nil {
 return err
 }

 return nil
 })
}

OPA support in Pulumi CrossGuard Policy-as-Code

Pulumi’s CrossGuard policy-as-code framework enforces policy across all your modern cloud infrastructure and/or in Kubernetes - using the full expressiveness of rich programming languages. It is deeply integrated into the deployment process to ensure that policy violations are identified before infrastructure is deployed instead of minutes or hours after changes are made.

Today’s release of Open Policy Agent (OPA) support in Pulumi policy-as-code extends the existing support for TypeScript and Python to include OPA’s .rego language and runtime for declarative policy specification. As an emerging standard for cloud native policy, OPA is already used across a wide variety of cloud native technologies, and can now also be leveraged to author policies to run in Pulumi’s policy model across any cloud or Kubernetes.

OPA-based rules can be authored using Pulumi’s rich resource model (top pane) and enforced as part of deployments to any infrastructure (bottom pane).

And as part of Pulumi’s support for organizational enforcement of policy, OPA-based policy can be enforced across all infrastructure deployments in your organization (including across different clouds and in Kubernetes clusters).

Helm 3 and Kustomize

Pulumi’s Kubernetes support has long included support for both defining Kubernetes resources directly in your Pulumi program, as well as using Pulumi to deploy Kubernetes resources from existing YAML or Helm. With a wealth of existing YAML and Helm content out in the ecosystem, these features have been used heavily by cloud engineering teams to deploy critical Kubernetes platform services and application logic into their Kubernetes clusters.

Over the last few months, we’ve extended this support in two key ways. We added support for Helm 3, giving Pulumi programs access to the new generation of Helm Charts. And we added support for Kustomize, so that the same kustomizations that are supported by kubectl can also be deployed natively with Pulumi. Together these two enhancement ensure that Pulumi users have full access to the incredible ecosystem of content available for Kubernetes.

Conclusion

We’re incredibly excited about today’s new features and the opportunity to deepen Pulumi’s support for Kubernetes and the Cloud Native ecosystem. The Pulumi Kubernetes Operator unlocks exciting new approaches to cloud native infrastructure delivery. The kube2pulumi tool makes it easier than ever to migrate from cumbersome YAML to expressive code. Strongly-typed CRDs bring the richness of Pulumi to the entire Cloud Native ecosystem of extensions to the Kubernetes platform. And OPA support in Pulumi policy-as-code brings cloud native policy authoring with OPA into the Pulumi Modern Infrastructure-as-Code world.

It’s never been a more exciting time to get started with Pulumi and Kubernetes for your next modern cloud project. Check out the new features or the Pulumi Kubernetes Getting Started today.

You can learn more about each of the new capabilities in their corresponding project documentation:

Deploy AWS Lambda with Elastic File System using Pulumi

Luke Hoban — Tue, 16 Jun 2020 00:00:00 +0000

Some of the code in this post is out of date. See the AWS guides for an updated overview and examples.

Ever since AWS Lambda was released in 2015, users have wanted persistent file storage beyond the small 512MB /tmp disk allocated to each Lambda function. The following year, Amazon launched EFS, offering a simple managed file system service for AWS, but initially only available to mount onto Amazon EC2 instances. Over the last few months, AWS has been extending access to EFS to all of the modern compute offerings. First EKS for Kubernetes, then ECS and Fargate for containers. Today, AWS announced that EFS is now also supported in Lambda, providing easy access to network file systems from your serverless functions.

In this post, we’ll show you how to get started using EFS with Lambda and Pulumi.

Why would you want to use EFS from Lambda? There are a few common reasons:

You need more local disk than the 512MB of /tmp space available to a single Lambda Function.
You want to share persistent storage across various functions and would prefer to use file system storage instead of blob storage or a database.
You want to share persistent storage between Lambda and other compute (like containers or ECS), and would prefer to use file storage over blob storage or a database.

Storing data to the file system is particularly natural for the kind of small functions common in Lambda - no need to use the AWS SDK - just call fs.writeFileSync() (or equivalent for your language of choice).

EFS + Lambda Concepts

To use an EFS file system from Lambda, we need a few things:

A VPC
An EFS File System in the VPC
EFS mount points in each of the subnets of our VPC
An EFS Access Point specifying how Lambda should mount the EFS volume
An AWS Lambda Function running in the same VPC and private subnets and referencing the EFS Access Point

Let’s walk through how to build the required infrastructure. We’ll use Pulumi to provision the necessary resources. You can check out the full source code in the Pulumi Examples.

Because EFS is tied to a VPC, we can only access it from a Lambda running inside a VPC. We should create a VPC with both public and private subnets to ensure the Lambda function can reach the internet (since Lambda functions inside VPCs are not assigned public IPs). We could use an existing VPC or create a new one for our application.

const vpc = new awsx.ec2.Vpc("vpc", { subnets: [{ type: "private" }, { type: "public" }] });
const subnetIds = await vpc.publicSubnetIds;

An EFS FileSystem is a regional service, but accessing it requires adding mount points to a VPC. We can add those to the private subnets where our Functions will run. Each mount target has a security group, and we can use the default security group to allow access only from other compute resources in the same security group in our VPC. We could also use an existing FileSystem and mount targets if they have already been provisioned for another application.

const filesystem = new aws.efs.FileSystem("filesystem");
const targets = [];
for (let i = 0; i < subnetIds.length; i++) {
 targets.push(new aws.efs.MountTarget(`fs-mount-${i}`, {
 fileSystemId: filesystem.id,
 subnetId: subnetIds[i],
 securityGroups: [vpc.vpc.defaultSecurityGroupId],
 }));
}

To use EFS from our Lambda Function, we need to tell Lambda how to mount the NFS volume. Specifically, we must specify the Unix user and group to use to enable access, and the location and permission scheme to use as the root folder of the mount. This is done using an EFS AccessPoint.

const ap = new aws.efs.AccessPoint("ap", {
 fileSystemId: filesystem.id,
 posixUser: { uid: 1000, gid: 1000 },
 rootDirectory: { path: "/www", creationInfo: { ownerGid: 1000, ownerUid: 1000, permissions: "755" } },
}, { dependsOn: targets });

Finally, each Lambda Function we want to run will need to be in our VPC, use our default security group, and mount the EFS file system via our AccessPoint. We can write a small function that creates Lambda Functions with this configuration and we can specify where we want to mount this into the Lambda Function, though we must mount it underneath /mnt/.

function efsvpcCallback(name: string, f: aws.lambda.Callback<awsx.apigateway.Request, awsx.apigateway.Response>) {
 return new aws.lambda.CallbackFunction(name, {
 policies: [aws.iam.ManagedPolicy.AWSLambdaVPCAccessExecutionRole, aws.iam.ManagedPolicy.LambdaFullAccess],
 vpcConfig: {
 subnetIds: vpc.privateSubnetIds,
 securityGroupIds: [vpc.vpc.defaultSecurityGroupId],
 },
 fileSystemConfig: { arn: ap.arn, localMountPath: "/mnt/storage" },
 callback: f,
 });
}

We now have our infrastructure all set up to start using Lambda with EFS. Let’s build an app!

Interact with EFS from an HTTP API

For a first simple Lambda application, we’ll expose an HTTP API that GETs files from, and POSTs files to, our EFS file system. We’ll implement this with API Gateway and two routes. Using the helper we defined above, each of these is just a few lines of code.

Note that we can use our normal Node.js fs.readFileSync and fs.writeFileSync APIs to interact with the EFS file system - no need for an AWS SDK to talk to EFS. This makes it easy to take existing software and libraries that can interact with the filesystem and run them inside Lambda.

const api = new awsx.apigateway.API("api", {
 routes: [
 {
 method: "GET", path: "/files/{filename+}", eventHandler: efsvpcCallback("getHandler", async (ev, ctx) => {
 try {
 const f = "/mnt/storage/" + ev.pathParameters!.filename;
 const data = fs.readFileSync(f)
 return {
 statusCode: 200,
 body: data.toString(),
 };
 } catch {
 return { statusCode: 500, body: "" }
 }
 }),
 },
 {
 method: "POST", path: "/files/{filename+}", eventHandler: efsvpcCallback("uploadHandler", async (ev, ctx) => {
 try {
 const f = "/mnt/storage/" + ev.pathParameters!.filename;
 const data = new Buffer(ev.body!, 'base64');
 fs.writeFileSync(f, data)
 return {
 statusCode: 200,
 body: "",
 };
 } catch {
 return { statusCode: 500, body: "" }
 }
 }),
 },
 ]
});

Once this is deployed, we can interact with the API to add files to our file system:

$ curl -X POST -d 'Hello world' $(pulumi stack output url)files/file.txt
$ curl -X GET $(pulumi stack output url)files/file.txt
Hello, world

Easy as that - our two Lambda functions are sharing data via EFS!

We can get more insight into what the file system looks like inside Lambda by exposing a REST endpoint to run commands inside the a Lambda function. This will allow us to run curl to run commands to populate and read the contents of the file system.

...
 {
 method: "POST", path: "/", eventHandler: efsvpcCallback("postHandler", async (ev, ctx) => {
 const cmd = new Buffer(ev.body!, 'base64').toString()
 const buf = cp.execSync(cmd);
 return {
 statusCode: 200,
 body: buf.toString(),
 };
 }),
 },
...

We can now interact with our EFS file system by just running shell commands. Since it’s a normal file system mount, all our standard Unix tools work as we expect. Note that the files are owned by uid/gid 1000/1000, and have the 755 permissions specified by our Access Point.

$ curl -X POST -d 'ls -la /mnt/storage' $(pulumi stack output url)
total 8
drwxr-xr-x 2 1000 1000 6144 Jun 10 05:37 .
drwxr-xr-x 3 root root 4096 Jun 10 05:35 ..
-rw-rw-r-- 1 1000 1000 21 Jun 10 05:37 file.txt

$ curl -X POST -d 'echo "<h1>Hello World</h1>" > /mnt/storage/index.html' $(pulumi stack output url)
$ curl -X POST -d 'ls -la /mnt/storage' $(pulumi stack output url)
total 12
drwxr-xr-x 2 1000 1000 6144 Jun 10 05:38 .
drwxr-xr-x 3 root root 4096 Jun 10 05:35 ..
-rw-rw-r-- 1 1000 1000 21 Jun 10 05:37 file.txt
-rw-rw-r-- 1 1000 1000 21 Jun 10 05:38 index.html

Now that we can read and write to our file system from Lambda, what about sharing it between Lambda and other compute services? We could mount our volume to EC2, EKS, ECS, or Fargate. Let’s try out Fargate by running a small nginx server that serves from the same file system we just used.

First, we create an EFS volume configuration that describes the volume we want to mount in our Fargate Task. This is similar to Lambda - we use the same AccessPoint and root directory.

const efsVolumeConfiguration: aws.types.input.ecs.TaskDefinitionVolumeEfsVolumeConfiguration = {
 fileSystemId: filesystem.id,
 authorizationConfig: { accessPointId: ap.id, },
 rootDirectory: "/www",
};

Then we can launch our nginx task. We use our EFS volume configuration to define a volume named efs in the task, and then mount that volume into the /usr/share/nginx/html path in the nginx container so it can serve content directly from the EFS file system. Note that we also have to explicitly set the platformVersion to 1.4.0 so that we can use EFS since 1.4.0 is not yet the default. We ensure that we are using the same security group that can access the file system and that we’ve opened up access to port 80 for our nginx server.

const nginx = new awsx.ecs.FargateService("nginx", {
 cluster: cluster,
 taskDefinitionArgs: {
 container: {
 image: "nginx",
 memory: 128,
 portMappings: [{ containerPort: 80, hostPort: 80, protocol: "tcp" }],
 mountPoints: [{ containerPath: "/usr/share/nginx/html", sourceVolume: "efs" }],
 },
 volumes: [{ name: "efs", efsVolumeConfiguration }],
 },
 securityGroups: [vpc.vpc.defaultSecurityGroupId, ...cluster.securityGroups],
 subnets: vpc.publicSubnetIds,
 platformVersion: "1.4.0",
});

Navigating to our Task’s public IP - we see the contents of the file we stored in our file system.

Conclusion

Support for EFS in Lambda opens up a lot of new options for managing persistent data inside serverless functions. And even more importantly, makes it easy to share data in a format that existing applications in VM instances or containers can natively work with.

In this post, we’ve shown how you can use Pulumi to manage Lambda + EFS. Pulumi makes it easy to provision and manage cloud infrastructure on any cloud, using programming languages you are familiar with, including TypeScript, Python, Go and .NET. This is a great fit for EFS and Lambda, which combine managed infrastructure with small serverless function code.

Check out the full Lambda + EFS example above in the Pulumi Examples, learn more about Lambda support for EFS in the AWS docs, and get started with Pulumi for AWS today.

Resource Oriented Documentation

Luke Hoban — Thu, 02 Apr 2020 11:34:07 -0500

Documentation in any product is super important, and an area where folks have shared a lot of feedback! We’ve heard you, and this week we took a major step in rolling out a brand new approach to resource documentation. We hope you like it as much as we do.

A New Format

In short, our docs are now resource-oriented. Each resource is presented on a single page for better viewing, discovery, and navigation. All content now supports switching between different languages, so that Pythonistas, Gophers, and .NET aficionados can now easily find code examples in their favorite language.

Pick your favorite language!

API docs for each language are still available and are deep-linked. In general, we will lead with this resource-oriented format throughout our content as a canonical, language-agnostic reference for all Pulumi resources.

We’ve rolled these out for AWS, Azure, and GCP, and other cloud providers, including Kubernetes.

We’d love your feedback on the new docs either in #docs or via the | buttons on the site.

Here are a few pointers:

Managing Kubernetes Infrastructure with .NET and Pulumi

Luke Hoban — Tue, 10 Dec 2019 00:00:00 +0000

Last month, we announced .NET support for Pulumi, including support for AWS, Azure, GCP, and many other clouds. One of the biggest questions we heard was about Kubernetes — “can I use Pulumi to manage Kubernetes infrastructure in C#, F#, and VB.NET as I can already in TypeScript and Python today?” With last week’s release of Pulumi.Kubernetes on NuGet, you can now also deploy Kubernetes infrastructure using your favorite .NET languages.

Using .NET to build our Kubernetes infrastructure offers several benefits:

Strong Typing: Unlike YAML, C# and F# offer a rich type system with quick feedback on potential errors.
Rich IDE Support: Use the rich features of IDEs like Visual Studio, Visual Studio Code, and Rider to develop your Kubernetes infrastructure—completion lists, refactoring, IntelliSense, and more.
Familiar Languages and APIs: Apply all the features of C#, F#, and VB.NET to your Kubernetes infrastructure—loops, variables, and the entire ecosystem of .NET Core libraries from System to everything in NuGet.
Components and Classes: Instead of copy/pasting pages of YAML between projects, .NET code can abstract common functionality into classes and libraries for code re-use and clean infrastructure design.

Together, these benefits provide a more familiar experience for working with Kubernetes than using YAML or Helm (a mix of YAML and Go templates) for .NET developers.

Tour of Kubernetes with .NET

We can see a simple example in practice - deploying an Nginx pod into a Kubernetes cluster.

var pod = new Pod("pod", new PodArgs
{
 Spec = new PodSpecArgs
 {
 Containers =
 {
 new ContainerArgs
 {
 Name = "nginx",
 Image = "nginx",
 },
 },
 },
});

We can deploy this with pulumi up.

$ pulumi up
Previewing update (luke-dev):
Type Name Plan
+ pulumi:pulumi:Stack basic_dotnet_kubernetes-luke-dev create
+ └─ kubernetes:core:Pod pod create
Resources:
+ 2 to create
Do you want to perform this update? yes
Updating (luke-dev):
Type Name Status
+ pulumi:pulumi:Stack basic_dotnet_kubernetes-luke-dev created
+ └─ kubernetes:core:Pod pod created
Resources:
+ 2 created
Duration: 11s

As always, Pulumi programs describe the desired state of our infrastructure, instead of an imperative process to construct that infrastructure. If we change our program, Pulumi will compute the minimal delta to apply to our Kubernetes cluster to transition to this new desired state. This almost feels like using Edit-and-Continue on our deployed Kubernetes resources—modifying our Kubernetes resources in place inside the cluster.

We can add a label to our Pod:

 var pod = new Pod("pod", new PodArgs
 {
+ Metadata = new ObjectMetaArgs
+ {
+ Labels = { { "app", "nginx" } },
+ },
 Spec = new PodSpecArgs
 {
 Containers =
 {
 new ContainerArgs
 {
 Name = "nginx",
 Image = "nginx",
 },
 },
 },
});

And then see that this updates the pod in place with this new label.

$ pulumi up
Previewing update (luke-dev):
Type Name Plan Info
pulumi:pulumi:Stack basic_dotnet_kubernetes-luke-dev
~ └─ kubernetes:core:Pod pod update [diff: ~metadata]
Resources:
~ 1 to update
1 unchanged
Do you want to perform this update? details
pulumi:pulumi:Stack: (same)
[urn=urn:pulumi:luke-dev::basic_dotnet_kubernetes::pulumi:pulumi:Stack::basic_dotnet_kubernetes-luke-dev]
~ kubernetes:core/v1:Pod: (update)
[id=default/pod-wyjq31t3]
[urn=urn:pulumi:luke-dev::basic_dotnet_kubernetes::kubernetes:core/v1:Pod::pod]
[provider=urn:pulumi:luke-dev::basic_dotnet_kubernetes::pulumi:providers:kubernetes::default_1_4_0::127c0c24-8b12-4010-a544-5f5390f91e4e]
~ metadata: {
~ labels: {
+ app: "nginx"
}
}
Do you want to perform this update? yes
Updating (luke-dev):
Type Name Status Info
pulumi:pulumi:Stack basic_dotnet_kubernetes-luke-dev
~ └─ kubernetes:core:Pod pod updated [diff: ~metadata]
Resources:
~ 1 updated
1 unchanged
Duration: 3s

The real benefits of .NET come when we extract common code into a reusable component. We can do that to create a new component like a ServiceDeployment, which includes both a Kubernetes Service and Deployment using opinionated defaults. With this, we can describe entire Kubernetes applications (100s of lines of YAML), in a short and semantically meaningful snippet of C#:

var config = new Config();
var isMiniKube = config.GetBoolean("isMiniKube") ?? false;

var redisMaster = new ServiceDeployment("redis-master", new ServiceDeploymentArgs
{
 Image = "k8s.gcr.io/redis:e2e",
 Ports = { 6379 },
});

var redisReplica = new ServiceDeployment("redis-slave", new ServiceDeploymentArgs
{
 Image = "gcr.io/google_samples/gb-redisslave:v1",
 Ports = { 6379 },
});

var frontend = new ServiceDeployment("frontend", new ServiceDeploymentArgs
{
 Replicas = 3,
 Image = "gcr.io/google-samples/gb-frontend:v4",
 Ports = { 80 },
 AllocateIPAddress = true,
 ServiceType = isMiniKube ? "ClusterIP" : "LoadBalancer",
});

return new Dictionary<string, object?>{
 { "frontendIp", frontend.IpAddress },
};

You can check out the implementation of this ServiceDeployment component in the Guestbook example in the Pulumi Examples repo on GitHub.

Building Docker Images for Kubernetes with .NET

In the examples so far, we have specified the Docker image to deploy as part of our Kubernetes Deployments by referring to an image already in the DockerHub or Google Container Registry. But what if we wanted to push our own custom Docker image built from our own application’s source code, and use that in our Kubernetes Pod or Deployment? This is easy to do as well with the Pulumi.Docker package. For example, we can deploy a customized Docker image derived from Nginx with the following:

var image = new Image("nginx", new ImageArgs
{
 ImageName = "my-username/my-nginx",
 Build = "./app",
});

var pod = new Pod("pod", new PodArgs
{
 Spec = new PodSpecArgs
 {
 Containers =
 {
 new ContainerArgs
 {
 Name = "nginx",
 Image = image.ImageName,
 },
 },
 },
});

Instead of using the default nginx image on DockerHub, we can use our own Dockerfile from the app folder in our Pulumi project, for instance, including the following in our Dockerfile to deploy some customized files in the app/content folder to our Nginx server:

FROM nginx
COPY content /usr/share/nginx/html

When we deploy this with Pulumi, the Docker file will be build locally, pushed to DockerHub, then the image name in DockerHub referenced from the Pod in Kubernetes. All of this happens automatically, allowing you to seamlessly deploy and version both your application code and infrastructure together with a simple pulumi up.

If we wanted to push to another Docker container registry (like ACR, GCR, ECR or others), we can easily do that too using additional parameters on the Pulumi.Docker.ImageArgs class.

Cloud + Kubernetes with .NET

Because Pulumi lets you work with both Kubernetes and cloud (AWS, Azure, GCP, and more), you can also create and manage the infrastructure that builds both a managed Kubernetes cluster as well as deploying applications and services into the cluster. Using a single familiar programming model, we have access to all these various cloud technologies at our fingertips.

For example, we can deploy a managed Kubernetes cluster on DigitalOcean, and then deploy a Pod into it:

var cluster = new KubernetesCluster("do-cluster", new KubernetesClusterArgs
{
 Region = "sfo2",
 Version = "latest",
 NodePool = new KubernetesClusterNodePoolArgs
 {
 Name = "default",
 Size = "s-2vcpu-2gb",
 NodeCount = nodeCount,
 },
});

var k8sProvider = new Pulumi.Kubernetes.Provider("do-k8s", new Pulumi.Kubernetes.ProviderArgs
{
 KubeConfig = cluster.KubeConfigs.Apply(array => array[0].RawConfig)
});

var app = new Pulumi.Kubernetes.Apps.V1.Deployment("do-app-dep", new DeploymentArgs
{
 Spec = new DeploymentSpecArgs
 {
 Selector = new LabelSelectorArgs
 {
 MatchLabels = new InputMap<string>
 {
 {"app", "app-nginx"},
 },
 },
 Replicas = appReplicaCount,
 Template = new PodTemplateSpecArgs
 {
 Metadata = new ObjectMetaArgs
 {
 Labels = new InputMap<string>
 {
 {"app", "app-nginx"},
 },
 },
 Spec = new PodSpecArgs
 {
 Containers = new ContainerArgs()
 {
 Name = "nginx",
 Image = "nginx",
 },
 },
 },
 },
},
new CustomResourceOptions { Provider = k8sProvider });

var appService = new Pulumi.Kubernetes.Core.V1.Service("do-app-svc", new ServiceArgs {
 Spec = new ServiceSpecArgs() {
 Type = "LoadBalancer",
 Selector = app.Spec.Apply(spec => spec.Template.Metadata.Labels),
 Ports = new ServicePortArgs {
 Port = 80,
 }
 }
}, new CustomResourceOptions {
 Provider = k8sProvider
});

var ingressIp = appService.Status.Apply(status => status.LoadBalancer.Ingress[0].Ip);

if (!string.IsNullOrWhiteSpace(domainName)) {
 var domain = new Domain("do-domain", new DomainArgs() {
 Name = domainName,
 IpAddress = ingressIp,
 });

 var cnameRecord = new DnsRecord("do-domain-cname", new DnsRecordArgs {
 Domain = domain.Name,
 Type = "CNAME",
 Name = "www",
 Value = "@",
 });
}

Note that this example deploys resources first into DigitalOcean (a Kubernetes Cluster), then into Kubernetes itself (Deployment and Service via a Pulumi.Kubernetes.Provider configured to connect to the Kubernetes cluster in Digital Ocean), then optionally also more resources in DigitalOcean dependent on the Kubernetes Service (Domain and DnsRecord). This example is a complex infrastructure deployment, all in a few dozen lines of declarative and strongly typed C# code.

Check out the full DigitalOcean Kubernetes Cluster in C# example for more details.

Conclusion

Kubernetes support is one of several significant new additions to the Pulumi .NET support, and many more improvements are in progress over the coming weeks. Get started with Kubernetes and .NET today, and let us know what you think!

Pulumi Watch: Fast Inner Loop Development for Infrastructure

Luke Hoban — Mon, 02 Dec 2019 00:00:00 +0000

A big part of our vision with Pulumi is to bring application developers and infrastructure teams closer together in the cloud. That includes both providing infrastructure teams with better software engineering tools, as well as providing developers with easier access to cloud infrastructure. We are often inspired by looking at great software engineering experiences in other development stacks and applying them to the cloud infrastructure space. Whether it be general-purpose languages and rich IDEs, testing and package management, or components and rich APIs, at Pulumi, we’ve repeatedly applied successful development tools and practices to the challenges of building and scaling modern cloud infrastructure.

As we look across other areas of successful application development, one of the most critical drivers of creativity and productivity is the speed of iteration. When developers can make a change and see or experience the results nearly instantaneously, they can explore solutions quicker, can take more creative risks (and thus learn more), and can ultimately iterate more quickly toward their desired result. From fast build times as a driving force in the design of the Go language, to the DevTools in Chrome, to Edit-and-Continue in Visual Studio, to the watch modes on tools like TypeScript, this idea of enabling quick or near-instantaneous feedback as code changes are made is a driving force behind many of the most successful developer experiences across a wide variety of software development environments.

Introducing Pulumi Watch

In Pulumi 1.5, we released a preview of a new pulumi watch command. This command provides a mode for developing cloud infrastructure that speeds up the rate of iteration and allows cloud developers to focus on their code and infrastructure, instead of on the mechanics of their deployments. With pulumi watch, cloud infrastructure feels like it’s truly at your fingertips.

The video above shows an example of working with some S3 resources. Every time I hit “save” in my editor, pulumi watch automatically deploys any changes to my infrastructure. If I add a new BucketObject resources and hit save, the object appears in S3. If I add a loop around it and hit save, 10 new objects appear and the original object is removed. And if I add an event handler, the 5 Lambda and IAM resources needed are deployed in seconds and are hooked up to the bucket. You don’t have to switch back and forth between an editor and a terminal, and you don’t have to go through heavyweight infrastructure deployment tools like Terraform or wait on slow external services like CloudFormation or CI/CD. Instead, changes appear directly in the cloud within seconds of hitting enter. You can try things out, experiment, learn, and rapidly iterate on your cloud infrastructure.

This new watch mode isn’t a replacement for a standard pulumi preview/pulumi update. For production infrastructure deployments, it is critical to understand precisely what changes will happen before applying them, to integrate with CI/CD systems, to be able to do targeted updates, and to apply policy to guard the production environment against misconfiguration. But that same level of care isn’t necessarily the right way to approach building new features in a development stack. And it’s in these development environments that pulumi watch opens up a new experience for infrastructure development. Although they are shorter periods relative to the full life of an infrastructure deployment, they are the majority of the time that creativity and creation is applied to cloud infrastructure, and thus represent a key opportunity to empower developers.

Not all cloud resources can be deployed or updated in seconds - but many of the most popular modern cloud primitives are built on platforms that enable and encourage quick iteration - and in particular both serverless and Kubernetes. Watch mode can be used during the development of any infrastructure, but can really shine in these areas where the cloud platforms natively support fast iteration.

Kubernetes Infrastructure + Applications

For Kubernetes, pulumi watch allows us to deploy changes to our cluster automatically as we author our Kubernetes resources. Change the number of replicas for our Deployment and hit save and the new pods scale-out. Add a Service to expose the deployment and public endpoint is available immediately. This pairs beautifully with the recently released Pulumi Kubernetes Extensions (kx) library, which allows you to build Kubernetes applications with just a few lines of code - avoiding a lot of the boilerplate that slows down typical Kubernetes development when working with YAML. We can see this combination in action in the video below:

Both of the previous videos also highlight another interesting element of modern cloud infrastructure development. In both the S3 example and the Kubernetes example, the Pulumi program deployed not just the infrastructure, but also the application code and assets. By authoring and delivering both infrastructure and application code together, we remove another aspect of the silos that separate developers from infrastructure, and in doing so, unlock even faster iteration. This is a pattern that Pulumi enables in a wide variety of unique ways, from Lambdas as lambdas, to build-and-push of Docker images, to deploying assets for static sites.

From Zero to Cloud Application in 90 Seconds

We can bring these ideas together, and show another example of what’s possible with watch mode using a serverless application example. As with so many things these days, a tweet inspired this example:

I want a website that does one thing: displays the number of times it’s been viewed. What is the shortest number of steps you can think of to get it running in the cloud?

— Slava Akhmechet (@spakhm)

Indeed, such simple tasks are often surprisingly complex and cumbersome with today’s infrastructure tools. But with pulumi watch and the Pulumi Cloud Framework, we can go from nothing to having this application deployed in AWS (or Azure!) in under 90 seconds.

Conclusion

When using pulumi watch, working with cloud infrastructure feels so different - it feels malleable, it feels accessible, and it feels like it’s right at your fingertips. By building on the same robust incremental deployment infrastructure as the rest of Pulumi, you get the same access to any cloud, and to a variety of languages and all the other benefits of Pulumi’s modern infrastructure as code tooling. Pulumi watch triggers a deploy on any save - including both of files in the Pulumi program itself, as well as other assets that the Pulumi program may reference like application source code or static assets. Pulumi watch also displays runtime logs for supported platforms (Lambda, ECS, and Cloudwatch currently, more coming soon), and can be paired with another new feature, Pulumi Query, to get customized status about the runtime environment during rapid iteration.

The pulumi watch command is available in preview now when the PULUMI_EXPERIMENTAL environment variable is set to true. We are just getting started - and see many more opportunities to tighten the iteration cycle further, to support more logging capabilities, to integrate more deeply with query, and to be applied to more modern infrastructure scenarios.

Give it a try today and drop us a note about where you’d like to see this go for Pulumi 2.0!

Adopting Existing Cloud Resources into Pulumi

Luke Hoban — Thu, 15 Aug 2019 00:00:00 +0000

See Migrating From Terraform to Pulumi for updated documentation on migrating from Terraform.

Most cloud infrastructure projects involve working with existing cloud resources — either building on top of existing resources or adopting existing resources under management with a new and more robust infrastructure provisioning solution.

Whether you are adopting resources that were deployed manually using your cloud provider’s console or CLI — or migrating existing infrastructure from tools like Terraform or CloudFormation — Pulumi makes it easy to adopt and manage your existing resources.

When working with existing resources, there are typically two scenarios:

You need to reference existing resources to use as inputs to new resources in Pulumi
You need to adopt existing resources under management so they can be updated/managed in Pulumi

We’ll review referencing existing resources, and then dive deeper into how you can adopt existing resources with Pulumi.

Referencing Existing Resources

For referencing existing resources, Pulumi offers several tools.

The .get methods available on every resource let you get all the details for a resource from the cloud provider based just on its id.
The StackReference resource lets you reference outputs of another stack for use as inputs to a stack, which is very useful for organizing projects and stacks.
terraform.state.RemoteStateReference(), aws.cloudformation.getStack() and azure.core.SubscriptionTemplateDeployment.get() let you reference outputs from existing Terraform, CloudFormation and ARM deployments respectively.

Together, these make it easy to reference existing infrastructure regardless of how it was provisioned.

Adopting Existing Resources

For adopting existing resources, Pulumi offers the import resource option to request that a resource defined in your Pulumi program adopt an existing resource in the cloud provider instead of creating a new one. In keeping with its focus on infrastructure as code, Pulumi lets you specify this import behavior inside the Pulumi code for your infrastructure deployment, instead of outside of it in some manual workflow. In its simplest form, it looks like this:

const myVpc = new aws.ec2.Vpc("my-vpc", {
 cidrBlock: "10.11.0.0/16",
}, { import: "vpc-02cd0cd2d807f340c"});

When Pulumi first sees a resource (in this case my-vpc), if it has an import option set, it will adopt the existing resource specified by that import ID from the cloud provider under management instead of creating a new resource.

We can see this when we deploy our Pulumi program with pulumi up:

$ pulumi up
Previewing update (dev):
Type Name Plan
pulumi:pulumi:Stack importtestaws-dev
= └─ aws:ec2:Vpc my-vpc import
Resources:
= 1 to import
1 unchanged

Because the import is provided in code, it can be configured in several different ways. You can look up import ids from Pulumi config instead of hard-coding them into programs, construct ids from predictable names based on program parameters, or even conditionally add the import: <id> property depending on whether one wants to deploy a stack in “adoption” or “creation” mode.

Import can be used for a wide variety of adoption scenarios, from importing a single resource to migrating an entire stack from an existing tool like Terraform. You can even automate an entire migration process across dozens of instances of infrastructure deployment. For small numbers of resources, you can just paste in individual resource IDs. For larger conversions, you can create a mapping of cloud resource ids (in JSON, CSV, or any other format) and load that into your Pulumi program with something like import: idMapping[name]. If you are importing or migrating dozens of stacks, you can even select between which of these mappings to use via a Pulumi config setting.

Walkthrough of Adopting Existing Infrastructure to Pulumi

To see one of these scenarios in action end-to-end, let’s walk through the process of deploying some “existing” infrastructure outside of Pulumi, and then adopting it under management of a Pulumi program. You can also check out the video below which walks through the same scenario.

We’ll use Terraform to provision our “existing” infrastructure. These resources could have been provisioned manually in the console, or using platform-native infrastructure-as-code solutions like CloudFormation, ARM or Kubernetes YAML.

We’ll also use TypeScript, though of course, the process is similar regardless of the language you choose.

1. Deploy “existing” infrastructure

For our “existing” infrastructure, we’ll use a small Azure deployment of a Virtual Machine and corresponding Resource Group and Networking resources. We’ll deploy this example using terraform. Save the code in that example to main.tf and run terraform init and then terraform apply to deploy it.

2. Create a Pulumi program

Next, we’ll create a Pulumi program to describe the same infrastructure that exists in Azure. We could write this by hand, but since we used Terraform to define the original infrastructure, we also have the option to use tf2pulumi to automatically convert our .tf files to Pulumi.

As of Pulumi CLI v3.71.0, tf2pulumi has been replaced with pulumi convert --from terraform. Converting Full Terraform Programs to Pulumi blog has more details.

$ pulumi new azure-typescript --force
$ tf2pulumi

We now have a Pulumi program in index.ts that describes the same infrastructure:

import * as pulumi from "@pulumi/pulumi";
import * as azure from "@pulumi/azure";

const config = new pulumi.Config();
const prefix = config.get("prefix") || "tfvmex";

const mainResourceGroup = new azure.core.ResourceGroup("main", {
 location: "West US 2",
 name: `${prefix}-resources`,
});
const mainVirtualNetwork = new azure.network.VirtualNetwork("main", {
 addressSpaces: ["10.0.0.0/16"],
 location: mainResourceGroup.location,
 name: `${prefix}-network`,
 resourceGroupName: mainResourceGroup.name,
});

...

If we try to run a pulumi up now to deploy this though, we see that as we expect, we will try to deploy a new copy of the infrastructure:

$ pulumi up
Previewing update (dev2):
Type Name Plan
+ pulumi:pulumi:Stack importazure-dev2 create
+ ├─ azure:core:ResourceGroup main create
+ ├─ azure:network:VirtualNetwork main create
+ ├─ azure:network:Subnet internal create
+ ├─ azure:network:NetworkInterface main create
+ └─ azure:compute:VirtualMachine main create
Resources:
+ 6 to create

Instead of creating new resources though, we want to adopt the existing ones.

3.Add `import` IDs

As we saw earlier, to import existing resources, we simply add import: <id> to the resource options of a resource defined in Pulumi. That resource will then be imported instead of created.

We can try this for one resource first. We can look in our terraform.tfstate file or in the Azure Portal and see that the id of the Resource Group is /subscriptions/0282681f-7a9e-424b-80b2-96babd57a8a1/resourceGroups/tfvmex-resources. So let’s use that:

const mainResourceGroup = new azure.core.ResourceGroup("main", {
 location: "West US 2",
 name: `${prefix}-resources`,
}, { import: "/subscriptions/0282681f-7a9e-424b-80b2-96babd57a8a1/resourceGroups/tfvmex-resources" });

4. Adopt resources

Now our preview shows that it will import the ResourceGroup instead:

$ pulumi up
Previewing update (dev2):
Type Name Plan
+ pulumi:pulumi:Stack importazure-dev2 create
= └─ azure:core:ResourceGroup main import
Resources:
+ 1 to create
= 1 to import
2 changes

We can complete this process, adding import: <id> to all resources, and then import them all to Pulumi.

Updating (dev2):
Type Name Status
+ pulumi:pulumi:Stack importazure-dev2 created
= ├─ azure:core:ResourceGroup main imported
= ├─ azure:network:VirtualNetwork main imported
= ├─ azure:network:Subnet internal imported
= ├─ azure:network:NetworkInterface main imported
= └─ azure:compute:VirtualMachine main imported
Resources:
+ 1 created
= 5 imported
6 changes
Duration: 8s

As noted earlier, if we were doing this at larger scale, instead of putting resource IDs inline in the code, we could load the IDs from a file or from Pulumi config. These IDs could be automatically extracted from our .tfstate file or from queries against our cloud provider.

5. Managing the resources with Pulumi

Once all of the resources are adopted into Pulumi, we can manage them by making incremental changes in Pulumi, and can discontinue using the existing .tf files to manage this set of infrastructure. For example, if we want to add a new tag to our Virtual Machine, we can use the code below. We can also now remove the import property if we want, or else leave it intact so long as the id of the resource remains the same. (If in the future we need to make a change that replaces one of these imported resources, Pulumi will require that the id property is removed, as it will no longer be correct.)

const mainVirtualMachine = new azure.compute.VirtualMachine("main", {
 location: mainResourceGroup.location,
 name: `${prefix}-vm`,
 networkInterfaceIds: [mainNetworkInterface.id],
 osProfile: {
 // Note: This comes directly from the original Terraform example, but
 // you would typically replace this with secret config in Pulumi, which
 // you can do after importing these resources into Pulumi.
 adminPassword: "Password1234!",
 adminUsername: "testadmin",
 computerName: "hostname",
 },
 osProfileLinuxConfig: {
 disablePasswordAuthentication: false,
 },
 resourceGroupName: mainResourceGroup.name,
 storageImageReference: {
 offer: "UbuntuServer",
 publisher: "Canonical",
 sku: "16.04-LTS",
 version: "latest",
 },
 storageOsDisk: {
 caching: "ReadWrite",
 createOption: "FromImage",
 managedDiskType: "Standard_LRS",
 name: "myosdisk1",
 },
 tags: {
 environment: "staging",
+ managedBy: "Pulumi",
 },
 vmSize: "Standard_DS1_v2",
});

Deploying this, Pulumi shows we will update the existing resource to add the new tag:

 ~ tags: {
+ managedBy: "Pulumi"
}

Since the resources are now being managed by Pulumi, we can use the Pulumi Service to see them:

And then navigate to the Virtual Machine in the Azure Portal and see the newly added managedBy: Pulumi tag.

The result is that we have migrated our existing infrastructure into Pulumi and can now use all the rich features of Pulumi to manage it going forward!

Conclusion

With Pulumi, you can easily work with existing cloud infrastructure, either referencing the existing infrastructure as inputs to new cloud infrastructure defined in Pulumi, or adopting existing infrastructure under management with Pulumi.

Get started with Pulumi today!

Introducing Pulumi Crosswalk for AWS: The Easiest Way to AWS

Luke Hoban — Mon, 10 Jun 2019 00:00:00 +0000

Some of the code in this post is out of date. See the AWS guides for an updated overview and examples.

Amazon Web Services provides an incredible platform for developers to build cloud-native applications, and is used by millions of customers of all sizes. The building block services that AWS offers enable teams to offload undifferentiated heavy-lifting to AWS. To maximally benefit from these services though, cloud engineering teams must learn how to compose all of these building blocks together to build and deliver their own applications. Today, this is still too hard. Getting from your laptop to a production-ready AWS deployment frequently takes days or weeks instead of minutes or hours. And AWS building block services frequently leave you to re-implement (and re-discover) best-practices instead of providing these as smart defaults.

Pulumi Crosswalk for AWS is a new open source library of infrastructure-as-code components that make it easier to get from zero to production on AWS, easier to adopt AWS best practices by default, and easier to evolve your AWS infrastructure as your application needs mature.

Crosswalk for AWS offers higher-level components for a wide range of the AWS platform - from serverless (Lambda, API Gateway and more) to containers (ECS, Fargate) to Kubernetes (EKS) and across networking (VPCs, SecurityGroups), monitoring (Cloudwatch Dashboards and Alarms) and load balancing (ALB and NLB). Because Crosswalk builds on the primitive building blocks of the AWS platform, you can use 100% of what AWS offers, and can seamlessly move between high-level components and low-level platform primitives, mixing-and-matching as needed.

In some of these domains, there have been successful tools that have helped developers more easily work with one specific aspect of AWS - Serverless Framework for serverless, Docker Compose for ECS, eksctl for EKS. But these tools have traditionally been limited to one specific kind of compute, limiting what users can accomplish, fragmenting development of complete cloud solutions, and leading to sharp cliffs when application needs evolve. Crosswalk for AWS offers the same kind of simpler and more productive developer interface as these tools, but applied uniformly across the entire AWS platform. Simple things are simple, and arbitrarily complex things are possible.

Whether you are starting a new SaaS project at a startup, migrating an enterprise application to cloud-native infrastructure, or delivering a new service as part of an established cloud application, Crosswalk for AWS provides the easiest way to get from zero to production, and to then evolve with your projects’ needs. You can get started with Crosswalk for AWS today!

Note: Crosswalk for AWS projects are authored using the Pulumi infrastructure-as-code tools. Pulumi allows you to define your infrastructure using code. As your infrastructure code evolves, Pulumi makes the minimal required changes to your cloud resources, ensuring that your infrastructure is updated without downtime (if possible).

From Zero to Production And Beyond with Crosswalk for AWS

Serverless (Lambda, API Gateway)

When getting started with a new cloud project, the starting point is often just serving a simple API on the internet. With just a few lines of code and tens of seconds to deploy, Crosswalk for AWS gives us our API.

import * as awsx from "@pulumi/awsx";

const api = new awsx.apigateway.API("api", {
 routes: [{
 method: "GET",
 path: "/",
 eventHandler: async (ev, ctx) => {
 return {
 statusCode: 200,
 body: JSON.stringify(ev),
 };
 }
 }],
})

export const endpoint = api.url;

By building on top of great AWS building blocks like Lambda, API Gateway, IAM and more, we avoid needing to worry about infrastructure, pay nearly zero fixed costs, and gain the ability to iterate quickly. With just a few more lines of code, we can wire up to our own domain, add authorization, provision a database, or chain together a more complex event-driven application.

Containers (ECS, Fargate)

As our needs grow, we might find we need to deploy containers - either custom applications or existing Docker images. Again, just a few lines of code gets our container running in production in AWS.

import * as awsx from "@pulumi/awsx";

// Create the cluster
const cluster = new awsx.ecs.Cluster("cluster");

// Configure a Load Balancer
const alb = new awsx.lb.ApplicationLoadBalancer("lb", {
 securityGroups: cluster.securityGroups,
});
const web = alb.createListener("web", { port: 80, external: true });

// Define the scaled-out Fargate service
const appService = new awsx.ecs.FargateService("app-svc", {
 cluster,
 taskDefinitionArgs: {
 container: {
 image: awsx.ecs.Image.fromPath("app", "./app"),
 portMappings: [ web ],
 },
 },
 desiredCount: 5,
});

export const url = web.endpoint.hostname;

Using ECS, Fargate, ECR and ELB, we get a robust production-ready container deployment - horizontally scaled out, load-balanced, and integrated with a private image repository. With just a few more lines of code, we can add autoscaling, customize our ECS cluster, or wire through advanced container configuration (volumes, environment variables, and more).

Networking (VPC)

To adopt best practices security posture, we may want to move our cluster into a custom VPC, and run our containers in private subnets. We can define and configure our VPC, and then redeploy our containers into this new VPC with ease:

import * as awsx from "@pulumi/awsx";

const vpc = new awsx.ec2.Vpc("vpc", {
 numberOfAvailabilityZones: 3,
});

const cluster = new awsx.ecs.Cluster("cluster", {
 vpc: vpc,
});

// ... same as before ...

The Vpc component builds on best-practices Virtual Private Cloud design patterns based on AWS guidance and documentation. By building in simple defaults for routing, subnet structure, NATs and multi-AZ you can get started quickly without having to re-discover these best practices. And as your needs grow, you can deeply customize and evolve your VPC structure - defining custom CIDR blocks, customizing Internet and NAT Gateways or additional additional private subnets.

Kubernetes (EKS)

As our project continues to evolve, we may want to move our container workloads into Kubernetes using EKS. We can stand up an EKS cluster with just a few lines of code, then customize our cluster and deploy Kubernetes workloads into our cluster.

import * as awsx from "@pulumi/awsx";
import * as eks from "@pulumi/eks";

// Create a VPC for our cluster.
const vpc = new awsx.ec2.Vpc("vpc");

// Create the EKS cluster itself.
const cluster = new eks.Cluster("cluster", {
 vpcId: vpc.id,
 subnetIds: vpc.privateSubnetIds,
 instanceType: "m5.large",
 desiredCapacity: 2,
 minSize: 1,
 maxSize: 2,
 deployDashboard: true,
});

// Export the cluster's kubeconfig.
export const kubeconfig = cluster.kubeconfig;

Building on EKS we get all the benefits of managed Kubernetes paired with the platform capabilities of AWS. With Crosswalk for AWS, we can get started quickly, and then evolve to take advantage of all of these platform features, like customized node groups, private networking, and even deploying Kubernetes YAML and Helm charts to our cluster.

Monitoring (CloudWatch)

Getting our infrastructure up and running in AWS is great, but we also want to be able to monitor it. We can add a few lines to any of the solutions above to define metrics, dashboards, alerting and more to monitor our AWS infrastructure.

import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";

// Create a bucket and a function to log all new object created events
const bucket = new aws.s3.Bucket("b");
const subscription = bucket.onObjectCreated("newObject", async (ev) => {
 console.log(JSON.stringify(ev));
});

// Create a CloudWatch Dashobard for our functions invocations
const dashboardName = "funcDashboard";
const dashboard = new awsx.cloudwatch.Dashboard(dashboardName, {
 widgets: [
 new awsx.cloudwatch.LineGraphMetricWidget({
 title: "Lambda invocations",
 width: 14,
 metrics: awsx.lambda.metrics.invocations({
 function: subscription.func, statistic: "Sum", period: 60, }), }), ], });

// Export the URL of the dashboard in the AWS console
export const dashboardUrl =
 `https://${aws.config.region}.console.aws.amazon.com/cloudwatch/home?` +
 `region=${aws.config.region}#dashboards:name=${dashboardName}`;

With CloudWatch being deeply integrated into all AWS services, we can easily build up robust logging, alerting and dashboarding solutions across our AWS infrastructure. With just a few more lines of code we can get logs for our functions and containers, create alarms when we cross critical thresholds, and create rich dashboard layouts all of which will version along with our infrastructure!

Early Feedback

We’ve been working on pieces of Crosswalk for AWS since before we launched Pulumi, and we’re excited to be launching Crosswalk for AWS today to bring these together for all AWS users. Over the last few months, users have built on Crosswalk for AWS for all of the scenarios above and more.

Tableau has used Crosswalk for AWS to accelerate their Kubernetes migration on AWS, building on the networking and EKS capabilities of Crosswalk for AWS.

“We’ve been happily using Pulumi’s EKS support for more than three months now. Our team was looking for an end-to-end solution to tame the complexity of Kubernetes on AWS and ensure we adhere to AWS best practices. Pulumi’s Crosswalk effort has equipped our team to scale far better, our delivery is now automated, and we can now deliver new application and infrastructure features with much faster turn-around, sometimes in just a few hours. Pulumi is a key contributor to our team’s improvement in productivity.” - Pankaj Dhingra, Sr Director, Cloud Engineering, Tableau

Compute Software has used Crosswalk for AWS to go from zero to production, and then to evolve their infrastructure in AWS, all without needing a dedicated cloud infrastructure engineering team.

“We use Crosswalk for AWS with Fargate. It’s been instrumental in getting us into production. Deploying to the cloud is necessarily complex. Crosswalk streamlines the process by exposing the controls you care about, always allowing you to dive deeper when needed.” - Kenny Williams, Co-Founder, Compute Software

We’re excited by the success these teams and many more have already had with Crosswalk for AWS, but even more excited about the potential to truly democratize access to the AWS platform and unlock highly productive cloud development for all kinds of engineering organizations.

What’s Next?

We’re still just scratching the surface of what’s possible with Crosswalk. We’re continuing to evolve the open source projects to simplify additional features of the AWS platform. And we’re also working to bring the same approach to other cloud platforms, including Azure, GCP and Kubernetes.

We believe that making it easier for cloud engineers to develop and deploy natively in the cloud will unlock creativity and empower developers to take advantage of more of the rich capabilities of AWS. We can’t wait to see what you build with Crosswalk!

Crosswalk for AWS is free and open source, and you can get started today.

For more on Crosswalk for AWS see:

Getting Started on Google Cloud Platform with Pulumi

Luke Hoban — Tue, 09 Apr 2019 00:00:00 +0000

Google Cloud is one of the most exciting cloud platforms available today, with a breadth of powerful infrastructure services from Google Container Engine (GKE) and Google Cloud Functions to Cloud Firestore and Cloud Spanner.

Pulumi is the most productive tooling available today for teams building cloud applications and infrastructure, in your favorite languages. Add them together, and teams can easily take maximum advantage of Google Cloud Platform’s rich features, productively, with a combined platform that makes it easy to collaborate, share, and reuse.

In this post, we’ll highlight some of the top features of Pulumi that make it a joy to work with Pulumi + Google Cloud:

Infrastructure as code for the Full Google Cloud Platform
Working with Kubernetes and Google Container Engine (GKE)
Super Simple Serverless with Google Cloud Functions
Continuous Delivery with Google Cloud Build
Managing Deployment with the Pulumi Service

Infrastructure as Code for the Full Google Cloud Platform

Pulumi lets you define and deploy infrastructure as code using your favorite programming languages, including JavaScript, Python, and Go. Pulumi provides access to the full Google Cloud Platform – every resource (including beta resources!) is available directly in your programming language and IDE. This delivers the software engineering benefits of your programming language, combined with the reliable and repeatable deployments of infrastructure as code.

In this example, we define a new network, firewall, and compute instance, using Python:

import pulumi
from pulumi_gcp import compute

disk = {
    'initializeParams': {
        'image': "centos-cloud/centos-7-v20190116"
    }
}

addr = compute.address.Address(resource_name='poc')

network = compute.Network("network")
network_interface = [
    {
        'network': network.id,
        'accessConfigs': [{'nat_ip': addr.address}],
    }
]

firewall = compute.Firewall("firewall", network=network.self_link, allows=[{
    'protocol': "tcp",
    'ports': ["22", "80"]
}])

instance = compute.Instance('poc', name='poc', boot_disk=disk, machine_type="f1-micro",
                            network_interfaces=network_interface)

pulumi.export('external_ip', addr.address)

Infrastructure can be deployed and updated using the pulumi CLI, with each update driving toward the desired state declared by the Pulumi program. Before each update, you can see a preview of the changes that will be made to your target infrastructure, and only after confirming will it proceed (with a full audit history):

$ pulumi update
Previewing update (luke):
     Type Name Plan Info
     pulumi:pulumi:Stack gcp-instance-nginx-luke
 ~ ├─ gcp:compute:Firewall firewall update [diff: ~allows]
 +- └─ gcp:compute:Instance poc replace [diff: ~bootDisk,name]
Resources:
    ~ 1 to update
    +-1 to replace
    2 changes. 3 unchanged
Do you want to perform this update? Yes
...

Learn more about working with Pulumi and Google Cloud infrastructure in this GCE Web Server tutorial.

Working with Kubernetes and Google Container Engine (GKE)

Pulumi also makes it easy to deploy resources to Kubernetes with access to the full Kubernetes API, across all object types and versions, using the same programming language and IDE experience. You no longer have to switch tooling at the boundary between GCP and Kubernetes. Instead, Pulumi gives you a single programming model for both.

In addition to the preview and auditing mentioned above, the Pulumi CLI also provides rich status and repeatable deployments to Kubernetes through its support for dependency-based orchestration of deployments:

And if you need a Kubernetes cluster, spinning one up with GKE using Pulumi is a breeze. Best of all you have easy access to the full breadth of configurability you might need: networking configuration, multiple NodePools, or clusters in multiple regions/zones. And you can easily add additional GCP infrastructure like managed databases, DNS configuration, or other resources needed to have a fully functional Kubernetes compute environment.

This example spins up a GKE cluster and uses Pulumi’s configuration system for easy parameterization:

from pulumi import Config
from pulumi_gcp.container import Cluster

# Read in some configurable settings for our cluster:
config = Config(None)

# nodeCount is the number of cluster nodes to provision. Defaults to 3 if unspecified.
NODE_COUNT = config.get('node_count') or 3
# nodeMachineType is the machine type to use for cluster nodes. Defaults to n1-standard-1 if unspecified.
# See https://cloud.google.com/compute/docs/machine-types for more details on available machine types.
NODE_MACHINE_TYPE = config.get('node_machine_type') or 'n1-standard-1'
# username is the admin username for the cluster.
USERNAME = config.get('username') or 'admin'
# password is the password for the admin user in the cluster.
PASSWORD = config.require('password')

# Now, actually create the GKE cluster.
k8s_cluster = Cluster('gke-cluster',
    initial_node_count=NODE_COUNT,
    node_version='latest',
    min_master_version='latest',
    master_auth={ 'username': USERNAME, 'password': PASSWORD },
    node_config={
        'machine_type': NODE_MACHINE_TYPE,
        'oauth_scopes': [
            'https://www.googleapis.com/auth/compute',
            'https://www.googleapis.com/auth/devstorage.read_only',
            'https://www.googleapis.com/auth/logging.write',
            'https://www.googleapis.com/auth/monitoring'
        ],
    },
)

Learn more about using Pulumi with Kubernetes and GKE in this Hello GKE tutorial.

Super Simple Serverless with Google Cloud Functions

Serverless has never been simpler than it is with Pulumi + Google Cloud Functions. Thanks to Pulumi’s “magic functions”, we can write callbacks in our language of choice, using a natural event-driven style, and Pulumi will handle the rest.

For example, want to serve a simple HTTP API with no fixed costs? It’s just a few lines of code:

import * as gcp from "@pulumi/gcp";

let greeting = new gcp.cloudfunctions.HttpCallbackFunction("greeting", (req, res) => {
    res.send(`Greetings from ${req.body.name || 'Google Cloud Functions'}!`);
});

export const url = greeting.httpsTriggerUrl;

Or perhaps a pubsub topic that runs some custom code on every message received:

// Create a PubSub Topic
let requests = new gcp.pubsub.Topic("requests");
// Print out a log message for every message on the Topic
requests.onMessagePublished("newMessage", (data) => {
    console.log(Buffer.from(data.data, "base64").toString());
});

And for quick inner loop development, the pulumi logs command supports pulling logs for your Google Cloud Functions from StackDriver, aggregating all compute logs in your program into one unified stream:

$ pulumi logs
Collecting logs for stack dev since 2019-04-08T12:02:48.000-07:00.
 2019-04-08T12:37:34.000-07:00[ newMessage-9d41c7a] Function execution started
 2019-04-08T12:37:34.000-07:00[ newMessage-9d41c7a] {"hello": "world"}
 2019-04-08T12:37:34.000-07:00[ newMessage-9d41c7a] Function execution took 70 ms, finished with status: 'ok'

Learn more about Pulumi support for Google Cloud Functions in this Simple Serverless Slackbot example.

Continuous Delivery with Google Cloud Build

Because Pulumi is infrastructure-as-code, the projects you deploy with Pulumi are robust and repeatable. Spinning up a testing or staging environment is simple, and continuously deploying infrastructure changes from commits and pull requests to your code is as easy as adding a few lines to your CI/CD system.

For teams working with Google Cloud Build, Pulumi deployments can be integrated into your continuous delivery with just a few steps. And for teams working with GitHub, Pulumi’s GitHub App gives you rich feedback on what changes will be deployed as part of a Pull Request, so you can see and approve infrastructure changes before they happen:

Learn more about integrating Pulumi with CI/CD in our continuous delivery documentation.

Managing Deployment with the Pulumi Service

In addition to the pulumi CLI for deploying and updating GCP infrastructure, Pulumi is always connected to the Pulumi Service, which provides a rich interface for viewing the history of your stacks and deployments, management of access controls for teams, and details of all resources under management.

With deep links into the Google Cloud Console, Pulumi provides quick access to all resources in your project.

And because Pulumi works with both Google Cloud and Kubernetes resources at the same time, it’s just as easy to deep link into resources in the Kubernetes console as well!

You can check out the Pulumi Service at app.pulumi.com and create your first Pulumi GCP project with just a few clicks, with an array of New Project options to get you up and running quickly.

Next Steps

Here at Pulumi, we love working with GCP and Kubernetes, and we are constantly working on ways to make Pulumi the most productive (and most fun!) way to program the Google Cloud Platform.

Pulumi is free and open source. You can get started with Pulumi today.

Here are a few resources to learn more about working with Pulumi and GCP:

Unified Logs with pulumi logs

Luke Hoban — Tue, 02 Apr 2019 00:00:00 +0000

Pulumi makes developing and deploying rich serverless and container-based applications a breeze. But how do you monitor and observe those applications while they are being developed and once they are deployed? There are many great answers: from the built-in capabilities of the underlying cloud services (Lambda, ECS, Kubernetes, and more), to great 3rd party solutions like IOpipe and Epsagon which we highlighted recently on this blog.

The Pulumi CLI provides another way to do logging, without requiring the additional setup of these existing solutions and seamlessly integrated into your Pulumi development workflow. The pulumi logs command provides a great first place to start for understanding your Pulumi application’s behavior. Especially during development, this command provides direct insight into the behavior of your application, bringing together logs across all of the different forms of compute you are using - from code running in serverless functions to containers to VMs.

Let’s take a quick look at pulumi logs and some of the ways it can be used as part of the inner loop of your Pulumi development.

Note: Currently pulumi logs supports only logs for applications running in AWS. In the near future it will be expanded to Kubernetes, Azure, GCP and additional cloud platforms.

Logging for Inner-Loop Development

When developing locally, it is common to use logging as an interactive part of the development process. With pulumi logs, we can bring some of these same approaches into the inner loop of the development process even for applications being actively developed/deployed in the cloud.

Let’s take a simple example:

import * as aws from '@pulumi/aws'
const bucket = new aws.s3.Bucket("mybucket");
bucket.onObjectCreated("newObject", async (ev) => {
    console.log(JSON.stringify(ev));
});
export const bucketName = bucket.id;

We can deploy this app, copy an object into the newly provisioned bucket, and then see the logs with the following:

$ pulumi up --skip-preview
 
$ aws s3 cp somefile.txt s3://$(pulumi stack output bucketName)
 
$ pulumi logs -f
Collecting logs for stack dev since
 
 2019-03-28T21:10:05.667-07:00[ newObject] {"Records":[{"eventVersion":"2.1","eventSource":"aws:s3","awsRegion":"us-west-2","eventTime":"2019-03-29T04:10:05.318Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"AWS:AIDAIB2I7NPKMQ2QAKXZ6"},"requestParameters":{"sourceIPAddress":"24.35.94.249"},"responseElements":{"x-amz-request-id":"F2D7DF26693AEFEE","x-amz-id-2":"o2leqNNwCVXTI8jfvUGk5KE7w6CiKsJYP/Q4nT0fHqZPrqwaVlSP0lNhspv5NiLAAwutCzJwlLc="},"s3":{"s3SchemaVersion":"1.0","configurationId":"tf-s3-lambda-20190328222809603300000002","bucket":{"name":"mybucket-97f10e0","ownerIdentity":{"principalId":"A3FZDZNSWOKQHV"},"arn":"arn:aws:s3:::mybucket-97f10e0"},"object":{"key":"out.txt","size":36000,"eTag":"818ac0cf0273f7e40e673d9892405e31","sequencer":"005C9D9A9D311D5C11"}}}]}

I can use this to quickly see what kind of values I get back on an S3 Bucket Object event, using live data running in the cloud. But I can do this whole thing in just a couple of seconds - from deployment to logs. This lets me iterate on my application live in the cloud with ease.

If there are errors in my code, I find out about them immediately:

bucket.onObjectCreated("newObject", async (ev, ctx) => {
    throw new Error("oops");
});

Results in:

$ pulumi logs --since 1m
Collecting logs for stack dev since 2019-03-28T20:17:28.000-07:00.
 
 2019-03-28T21:17:01.269-07:00[ newObject] {"errorMessage":"oops","errorType":"Error","stackTrace":["/var/task/__index.js:26:11","Generator.next (<anonymous>)","/var/task/__index.js:12:71","new Promise (<anonymous>)","Object.<anonymous> (/var/task/__index.js:8:12)","Object.__f1 [as __awaiter] (/var/task/__index.js:17:34)","/var/task/__index.js:24:21","__f0 (/var/task/__index.js:30:34)"]}

Aggregating Logs

As applications grow to span multiple forms of compute – such as micro services, lambdas, containers, and VMs – it becomes important to be able to correlate logs across services.

One great example of this is the video thumbnailer example which implements this simple architecture involving two buckets, a lambda event, and a long-running Fargate container task.

The Pulumi program deploys several components, using different types of cloud resources. But using Pulumi logs we can see all logs in a single, unified view.

From the image above, you can see the onNewVideo Lambda function, then triggering the ffmpegThumbTask Fargate container, and then copying a file back into S3 triggering the onNewThumbtail Lambda function. So you don’t need to jump around between different systems for viewing logs, you can get a consistent view of what’s going on in your cloud application!

Underneath the hood, these logs live in different and uncorrelated CloudWatch LogGroups, spread across an AWS account. But with pulumi logs we can get the application-centric view we need to understand the overall behavior of our application architecture.

Controlling Log Rendering

The pulumi logs command supports a few options for controlling what gets logged.

The --since flag retrieves logs only since a given relative or absolute time. For example, pulumi logs --since 1m will get logs just since 1 minute ago - useful for limiting logs to only what’s happened very recently - especially during inner-loop development. And to get logs since a specific point in time (such as the timestamp of a known event from a previous execution of pulumi logs) you can use pulumi logs --since 2019-03-28T21:17:01.269-07:00.

A couple of the examples above used the --follow flag to follow the log output until cancelled. This is useful when waiting for log output to come in during debugging.

And for applications with many resources, it is often important to scope results down to one or more specific resources, which can be done with the --resource flag. For example, logs for just the Fargate task above could be retrieved with pulumi logs --resource ffmpegThumbTask.

Working Seamlessly with Your Cloud Provider

Like many other Pulumi features, pulumi logs works by building on top of the great building blocks provided by the underlying cloud provider services. Instead of adding new runtime logging infrastructure to your application, pulumi logs lets you easily extract out logs directly from your existing infrastructure. But because Pulumi understands the natural application scope for what resources are part of a single application, it can bring together the logs from all of those resources into one place, instead of having to think about individual resources.

Behind the scenes, pulumi logs makes the same calls to CloudWatch Logs or other backend log providers that you would make yourself via the aws CLI or AWS Console. But you don’t have to think about the physical identities of the cloud resources themselves - Pulumi takes care of finding the resources with supported logs, collecting those logs based on the requested filters, and then presenting them in unified logical view.

What’s Next?

We’ve seen Pulumi users working with Lambda, ECS and other AWS compute offerings become heavy users of pulumi logs during development. But supporting a broad set of cloud providers is key to our mission at Pulumi - so we really want to bring these benefits to all Pulumi users. We’ll be expanding pulumi logs to Kubernetes, Azure and GCP in the near future - with easy support for plugging in other cloud providers and cloud resources as sources for log data.

Even better, because Pulumi allows you to work with multiple cloud providers at once - like AWS and Kubernetes, or GCP and Azure, or even two different AWS regions - pulumi logs will be able to aggregate logs across multiple cloud providers and offer a seamless application-centric view of your logs.

Users of the Pulumi Service at app.pulumi.com also would love to have access to these logs at the click of a button - and this is something we’re also excited to enable in the near future.

You can take pulumi for a spin today by checking out the Getting Started Guide guide or some of the Pulumi Examples on GitHub. Then join us in the Pulumi Community Slack to chat about where you want to see us go (and where you’d like to help out!) with pulumi logs and other features of the Pulumi open source project in the coming weeks and months.

Epsagon: Define, Deploy and Monitor Serverless Applications

Luke Hoban — Thu, 29 Nov 2018 00:00:00 +0000

Pulumi makes it incredibly easy to use serverless functions within your cloud infrastructure and applications - an AWS Lambda is as simple as writing a JavaScript lambda!

const bucket = new aws.s3.Bucket("my-bucket");
bucket.onObjectCreated("onNewObject", async (ev) => console.log(ev));

By making it so easy to introduce serverless functions into cloud infrastructure, Pulumi programs often incorporate many Lambdas, all wired together as part of a larger set of infrastructure and application code.

Epsagon is a serverless monitoring solution that lets users observe entire serverless applications composed of many functions and other AWS infrastructure, instead of just looking at individual functions one at a time. This is a great fit for Pulumi applications!

Pulumi users can use the @pulumi/epsagon package to automatically get the benefits of Epsagon monitoring for all of the serverless functions defined in their Pulumi applications. By just adding two lines of code - every AWS Lambda created from a JavaScript callback (like in the example above) will automatically get instrumented with Epsagon’s monitoring library.

const epsagon = require("@pulumi/epsagon");
epsagon.install(pulumi, { appName: "my-example" });

Monitoring a Serverless Pulumi Application

I have a small Pulumi application I used to compare the latency of various Lambda event sources. In about 200 lines of code, it does the following:

Provisions required infrastructure for various AWS event sources
Hooks up Lambda event handlers to this infrastructure
Describes the code to run in those event handlers (using JavaScript callbacks)
Creates shared infrastructure for storing test results, and uses it inside the event handlers (via a generic test harness)
Creates and exposes a REST API for kicking off tests and viewing graphs of test results

Each test case looks like this (you can see more details in the README):

// Test for SNS Topic
const topic = new aws.sns.Topic("my-topic");
function sendTopicValue(val: number) {
return (new aws.sdk.SNS()).publish({
TopicArn: topic.arn.get(),
Message: val.toString(),
}).promise();
}
const topicSubscription = topic.onEvent("ontopicevent", (ev, ctx) => {
return harness("sns", Number(ev.Records[0].Sns.Message), sendTopicValue);
});

All up, it creates 6 Lambda functions connected to a variety of event sources.

I add the two lines of code above to add Epsagon, and set a config secret for the Epsagon API token:

pulumi config set –secret epsagon:token <my-epsagon-token>

Then, after kicking off a test using the REST API exposed by my Pulumi application, I go over to the Epsagon console for my project to see what happened. First, I can get a birds-eye view of my overall architecture. This provides a great visualization of how my resources are related at runtime. Each event source triggers a lambda attached to it, which both triggers that same event source again, and also logs to a central datastore that keeps track of statistics.

I can also drill in and see details about each piece of this, and what performance looked like across all of these.

And finally, I can see what chain of events in my infrastructure led to an operation occurring in DynamoDB - in this case, a REST API was handled by a Lambda, which invoked a Table scan.

Conclusion

Together, Pulumi and Epsagon make it easy to build truly serverless applications – not just a single Lambda, but a whole application made up of many small event driven callbacks that run on-demand. Pulumi makes it easy to author and deploy these kinds of architectures, and Epsagon makes it easy to monitor them. Users get an application-centric view across their full serverless application infrastructure.

Checkout the quickstart guides for Pulumi and Epsagon now!

Announcing Pulumi 0.15

Luke Hoban — Wed, 15 Aug 2018 00:00:00 +0000

This blog post refers to an outdated version of Pulumi (0.15). For current features and capabilities, please refer to the latest documentation and current release notes.

Just over a month ago we publicly launched Pulumi, a new cloud native development platform. The response has been overwhelming and we’ve been hard at work responding to your feedback ever since.

Today, we are excited to release Pulumi 0.15 and make it available to download. This release includes improvements across the entire Pulumi development experience. Pulumi supports more platforms (Kubernetes and OpenStack, is faster (Parallelism, simpler (native TypeScript support), richer (serverless frameworks for Azure and Google Cloud), and is more deeply integrated into the application lifecycle (GitHub App for CI/CD integration).

In this post, we’ll take a quick tour of these new features. Stay tuned for follow up blog posts to dive deeper into individual topics.

Kubernetes

Pulumi can now deploy and manage Kubernetes resources using the same familiar programming model supported for AWS, Azure, and Google Cloud Platform. It’s also possible to mix resources from multiple clouds in the same program, such as using an S3 Bucket or RDS database from your Kubernetes-managed containers.

This release includes dozens of features that make it simple and productive to create and manage Kubernetes applications and resources and to perform production-grade deployments. The @pulumi/kubernetes package provides access to the full Kubernetes API, ensuring you can deploy any Kubernetes application with Pulumi.

New support for new helm.v2.Chart(...) allows deploying existing Helm charts via Pulumi, ensuring you can coordinate deployment of charts as part of a larger Pulumi deployment.

Complex Kubernetes resources (like Deployments and Services) also provide rich status updates so that you have complete visibility into the state of a deployment while it is in progress, including incremental progress reporting, and significant work was put into making it easy to diagnose and robustly recover from deployment failures.

The Pulumi Kubernetes provider can be used to deploy to any Kubernetes cluster, whether cloud hosted (EKS, AKS, GKE) or local (Minikube, Docker for Mac) or on-premise. And because Pulumi is already great at deploying cloud resources, the Pulumi Kubernetes provider can be combined with, for example, the Pulumi Azure provider to deploy and manage both the cluster and Kubernetes resources that should be installed into the cluster.

import * as azure from "@pulumi/azure";
import * as k8s from "@pulumi/kubernetes";
import * as helm from "@pulumi/kubernetes/helm";

// Create an Azure Kubernetes Service cluster
const resourceGroup = new azure.core.ResourceGroup("aks", { location: "West US" });
const kubernetesService = new azure.containerservice.KubernetesCluster("kubernetes", {
/* ... */
});

// Create a Pulumi Kubernetes provider configured to deploy to the AKS cluster above
export const azk8s = new k8s.Provider("azk8s", {
 kubeconfig: kubernetesService.kubeConfigRaw,
});

// Deploy a Helm chart into the cluster
const kibana = new helm.v2.Chart("kibana", {
 repo: "stable",
 chart: "kibana",
 version: "0.8.0",
 values: { service: { type: "LoadBalancer" } },
}, { providers: { kubernetes: azk8s } });

Check out the Kubernetes overview docs, the API documentation and the pulumi-kubernetes GitHub project for additional details.

Parallelism

Pulumi now runs deployments in parallel by default. To accomplish this, the Pulumi engine tracks all dependencies across cloud resources in your program, maximizing the amount of work that can proceed in parallel, while also ensuring all resource are created, updated, replaced, or destroyed in the correct order.

In some extreme cases, this new parallelism support can improve performance by 2-10x, but even more importantly, parallelism provides a performance boost for practically every Pulumi deployment.

First Class Providers

Pulumi programs can now create multiple instances of a resource provider, and can initialize an instance of a resources provider using outputs of other resources.

For example, a Pulumi program can now deploy resources into multiple AWS regions by creating providers targeting each region. This can be used for resources that must be deployed in a certain region (e.g. Lambda@Edge or ACM certificates), or just for managing multi-region deployments from a single Pulumi program.

This also enables scenarios where a Pulumi program deploys some base infrastructure, and then wants to manage resources deployed on top of that using a different provider. For example, deploy a GKE cluster, and then configure a Kubernetes provider to deploy Kubernetes resources on top of the cluster. Or even then deploy OpenFaaS into the Kubernetes cluster, and use the Pulumi OpenFaaS provider to deploy functions.

You might have already noticed, however we already used this feature in the Kubernetes example earlier!

Native TypeScript Support

Pulumi has supported JavaScript since inception, and as a result, it’s been possible to use Pulumi via TypeScript to get the benefits of earlier error checking and richer tooling for cloud infrastructure and application development. However, TypeScript code had to previously be manually compiled to JavaScript prior to doing a Pulumi deployment, which added some friction for users who wanted to use TypeScript with Pulumi.

In this release, Pulumi now natively supports TypeScript, so you do not need to run tsc explicitly before deploying. (We often forget to do this too!) Simply run pulumi up, and the program will be recompiled on the fly before running it.

New templates are now available via pulumi new which use the new, simpler, TypeScript support.

Using Imports from Runtime Functions

Pulumi supports creating cloud functions (Lambdas, Azure Functions, Google Cloud Functions) using JavaScript callbacks in your Pulumi application. This provides a nice and simple way to write the little pieces of code that you need to integrate into your cloud application and version with your infrastructure using real functions.

We heard feedback from early adopters who wanted to be able to import NPM packages in their Pulumi program more easily, and then use them within one of these cloud functions. We’ve enabled this now, so that the common way of structuring code works as expected, and so that writing idiomatic JavaScript or TypeScript “just works.”

For example, the code below uses the axios NPM package to make an HTTP request inside an AWS Lambda invoked by an AWS API Gateway (in just a few lines of code!).

import * as axios from "axios";
import * as cloud from "@pulumi/cloud-aws";

const api = new cloud.API("api");
api.get("/", async (req, res) => {
 const statusText = (await axios.default.get("https://www.pulumi.com")).statusText;
 res.write(`GET https://www.pulumi.com/ == ${statusText}`).end();
});

export const url = api.publish().url;

OpenStack

Thanks to a contribution by Fraser Waters (@Frassle), Pulumi now also supports OpenStack. This allows users targeting an OpenStack API from their public or private cloud provider to use Pulumi to author and manage infrastructure deployments.

For example, a VM can be deployed to OVH with just the following:

const os = require("@pulumi/openstack");
const instance = new os.compute.Instance("test", {
 flavorName: "s1-2",
 imageName: "Ubuntu 16.04",
});

export const instanceIP = instance.accessIpV4;

Check out the API documentation and the pulumi-openstack GitHub project for additional details. Huge thanks to Fraser for his work on this!

Serverless Functions in Google Cloud and Azure

Particularly with serverless functions being real functions, we had a lot of interest in broadening our existing support beyond just AWS serverless. We are happy to announce better serverless functionality for Google Cloud and Azure.

On Azure, the new @pulumi/azure-serverless package makes it easy to work with serverless functions, and has initial support for hooking up to Blob storage event sources:

import * as azure from "@pulumi/azure";
import * as serverless from "@pulumi/azure-serverless";

const storageAccount = new azure.storage.Account("images-container", { /* ... */ });
serverless.storage.onBlobEvent("newImage", storageAccount, (context, blob) => {
 context.log(context);
 context.log(blob);
 context.done();
}, { containerName: "folder", filterSuffix: ".png" });

export const storageAccountName = storageAccount.name;

On Google Cloud, the new gcp.serverless.Function provides an easy way to create a Google Cloud Function from a JavaScript callback in a Pulumi program. Thanks to Mikhail Shilkov (@mikhailshilkov) for contributing this feature!

import * as gcp from "@pulumi/gcp";
let f = new gcp.serverless.Function("f", {}, (req, res) => {
 res.send(`Hello ${req.body.name || 'World'}!`);
});

export const url = f.function.httpsTriggerUrl;

GitHub App for CI/CD Integration

Pulumi already works with your favorite CI/CD systems to accomplish automated and continuous deployments of cloud infrastructure and applications. This is how Pulumi deploys and manages our own infrastructure that runs <pulumi.com>, and is how our most engaged users adopt Pulumi in their own teams.

Our mission is to make Pulumi the easiest way to get code to the cloud. To that end, we are launching a new Pulumi GitHub App which bridges the gap between GitHub (source code, pull requests) and Pulumi (cloud resources, stack updates). By installing the Pulumi GitHub App into your GitHub organization , and then running Pulumi as part of your existing CI/CD process, you will receive integrated stack updates and previews in your pull requests. This allows you to see the potential impact a change would have on your cloud infrastructure before merging the code, and collaborate with your teammates when deploying application and infrastructure changes.

The Pulumi GitHub App is still in preview as we work to support more CI systems and extend its capabilities. For information on how to install it and configure it with your CI system, please read the documentation.

Summary

We’re excited about all the new features in this release and the new scenarios they enable for the Pulumi community . If you are new to Pulumi, download the tools and get started today, or join us in Slack. A big thanks to all the users and contributors who have helped shape this release – we can’t wait to see what you build next !

Special Thanks

Fraser Waters (@Frassle)
Mikhail Shilkov (@mikhailshilkov)
James Nugent (@jen20)
Thomas Schersach (@Tirke)
JT (@Jtango18)
Itay Maman (@imaman)

How we use Pulumi to build Pulumi

Luke Hoban — Tue, 26 Jun 2018 00:00:00 +0000

Here at Pulumi we are (perhaps unsurprisingly!) huge fans of using Pulumi to manage our cloud infrastructure and services. We author our infrastructure in strongly-typed programming languages, which allows us to to benefit from rich tooling - documenting and factoring our infrastructure using the same software engineering practices we apply to our application code. This also allows us to create reusable abstractions which accelerate our ability to deliver new features and services, and our ability to standardize and refactor infrastructure patterns across our services with relative ease.

Like other users, we use Pulumi at a variety of levels of abstraction. We use Pulumi for raw infrastructure provisioning, defining the core networking layer for our AWS-based backend infrastructure. And we use Pulumi to define how our application services are deployed into ECS using just a few lines of code. Pulumi hosts and manages static content for www.pulumi.com and get.pulumi.com. We use Pulumi to define the CloudWatch dashboards connected to our infrastructure. And for monitoring, Pulumi defines metrics and notifications/alarms in PagerDuty and Slack.

Best of all, we’ve been able to take things we’ve learned from these use cases, and others we’ve worked with beta users on over the last few months (thank you!), and factor common patterns out into reusable libraries like @pulumi/aws-infra and @pulumi/cloud for ourselves and others to build upon.

In this post, we’ll do a deeper dive into each of these use cases, highlighting unique aspects of how we use Pulumi itself, and some of our engineering processes around how we integrate Pulumi into the rest of our toolchain.

Infrastructure

Our backend infrastructure that supports app.pulumi.com and the Pulumi CLI is fairly typical of a cloud service/application building on AWS. Those who have done this before will be familiar with the combination of AWS VPC, EC2, ASG, ALB, ECS, S3, Aurora, CloudFront and Route53 that powers the core of many AWS-based web applications.

We started off building this infrastructure last year - before we even had completed support for managing AWS resources in Pulumi! As a result, we brought this up initially using CloudFormation. Moving this to Pulumi was an amazing experience. We immediately brought more engineering rigour to the codebase, factoring and decomposing functionality in clearer ways, extracting out useful patterns into named functions, using loops and conditionals instead of copy/paste, and getting completion lists and on-the-fly error checking from within our development tools.

Equally important, authoring our infrastructure in Pulumi gave us the ability to factor out reusable abstractions. We found that our own VPC definition was something that many early customers we worked with also needed - so we factored it out into new awsinfra.Network. We found that standing up an ECS cluster was another big pile of AWS resources that are commonly needed, but can be configured with a much smaller space of configuration values in the common case, so we factored out new awsinfra.Cluster.

Defining infrastructure in Pulumi made it easy to stand up many instances of our service, including per-developer stacks, with just pulumi stack init my-developer-stack and pulumi update. The ability to build reusable components which can be multi-instantiated into the same target cloud is critical here, and something that Pulumi takes care of by default. Within minutes, a developer can have an isolated copy of our full infrastructure stood up, which can be used to test out and quickly iterate on changes. This includes not just the basic infrastructure, but also provisioning of DNS entries in Route53 and creation of custom CloudWatch dashboards (see more on this below).

We have three shared environments - development, staging and production - each an isolated environment with separate configuration. For these shared environments we drive deployments of the Pulumi programs that define the infrastructure above via our CI/CD pipeline. Pulumi can be integrated into any existing CI/CD pipeline or toolchain, via direct use of pulumi preview and pulumi update as part of build and deployment stages. This allows us to reliably deliver a known desired state for each target environment, and to promote changes between environments with just a git merge.

API and App Services

Most of our backend application services are developed in Go, and we deploy them onto our production infrastructure as container images. However, instead of building separate CI pipelines for our container images and the infrastructure they need to function correctly (ECS Service, Application Load Balancer, ECR Repository, etc.), we drive the deployment of both infrastructure and application code via a single Pulumi program. This allows our application code to version along with its supporting infrastructure - when we need to expose a new port for example, we can make the changes to both application code and load balancer as part of a single commit, and promote it through CI as an atomic action. We have found this to significantly simplify the delivery of containerized services on AWS, as well as making it easy to stand up new services into our existing shared networking and cluster infrastructure.

Again, we found this to be a common pattern as we worked with other early customers, and factored out a library to simplify deployment of containerized services into a production deployment on AWS as part of the cloud.Service API in @pulumi/cloud. Here’s an example of using an application service defined in the ./airflow-container folder (with a Dockerfile) into an ECS cluster. This one declaration will provision a scaled-out ECS Service for running the container images, a ECR repository to store the image, optionally a load balancer if ports were exposed, and each pulumi update will build the container image, push it to the private repository, and turn-over each service task to use the new image. All of that workflow complexity is abstracted away behind a simple API and a single CLI command.

Metrics, Alarms and Dashboards

We also use Pulumi to define a few less traditional parts of our infrastructure - including our metrics, dashboards and alarming infrastructure on top of AWS CloudWatch.

Defining and versioning this along with our infrastructure has significant benefits: new features are built with metrics, dashboards and alarms lit up from the get-go, developers get a full set of dashboards as part of their personal isolated development environments, and promotions from development to staging to production can verify not just physical infrastructure, but also that alarms and dashboards are behaving as expected prior to pushing code to production. Here’s a sample of our CloudWatch dashboards:

We build up AWS resources representing CloudWatch dashboards using a compositional framework shown below. As this example shows, bringing software engineering approaches to infrastructure as code can lead to new approaches to how code is authored and factored - and ultimately to the creation of higher level frameworks for building domain-specific cloud infrastructure - in this case, CloudWatch dashboards.

The same metrics that power these dashboards are also used to power notifications and alarms. In these cases, we want to define both the infrastructure for the alarm - the SNS topics and registration of those topics with CloudWatch - as well as the small piece of code that fires off Pagerduty and Slack notifications on notifications being pushed to that SNS topic - all in one codebase. This ensures that we can version these together, and that we can deploy both consistently as part of updates. To do this, we include the event handlers for these infrastructure events directly inline in our Pulumi programs. Note in the code snippet below how we define the infrastructure for a Topic, and then conditionally wire up event handlers on that topic, including the code to execute at runtime on these events directly in our Pulumi program. This code can even reach out and reference configuration used to provision our Pulumi deployment - key to enabling this to be deployed seamlessly into a variety of different development environments.

Static Content

We serve three properties as static content from S3, each described using Pulumi. One of these is https://get.pulumi.com which hosts downloads of our pulumi CLI installer. It’s a very simple program, and even better - it’s open source on GitHub. It combines AWS S3, CloudFront and Route53.

This infrastructure is deployed and managed using the Pulumi Service at app.pulumi.com, allowing us to track the history of deployments and changes to the infrastructure and quickly jump in to the resources defined by this stack in the AWS console.

Conclusion

Treating cloud infrastructure as software has changed how we think about infrastructure engineering. Using Pulumi for both traditional cloud infrastructure as well as to get application code and supporting service infrastructure into production provides a simple workflow and the ability to create simple high-level APIs to make new services easy to define and deploy.

Try out Pulumi yourself today!

Pulumi Blog: Luke Hoban

Announcing the General Availability of Pulumi ESC - Welcome to a New Era of Secrets Management

Challenges in Secrets Management

Introducing Pulumi ESC: A Modern Approach to Environments, Secrets, and Configuration

Centralize and Simplify Secrets Management:

Boost Developer Velocity and Security:

Built for the Modern Cloud, Open by Design:

Customer Stories

Beyond Preview: The Evolution of Pulumi ESC

Pricing

Conclusion

Introducing: Support For Using Any Terraform Provider with Pulumi

Walkthrough

Why Support All Terraform Providers?

How Does This Work?

What’s Next?

Introducing Pulumi Copilot: Intelligent Cloud Management

Pulumi Copilot Key Features

Access any data in Pulumi Cloud

Pulumi IaC Authoring and Deployment

Access cloud metadata from the clouds themselves

Walkthrough of Pulumi Copilot

Future of Pulumi Copilot

Taking Action

Pulumi CLI Integration

Pulumi Docs Website Integration

Cloud CLI Skills

Conclusion

Secrets and Configuration for Software Engineers: Pulumi ESC Versioning, SDKs and More

What is Pulumi ESC?

ESC Versioning

ESC SDKs for Python, TypeScript/JavaScript and Go

Environments as Code with IaC

Conclusion

The Present and (Near) Future of AI and Infrastructure as Code

AI transforming Infrastructure as Code

Today

Near Future

Infrastructure as Code transforming AI

Today

Near Future

Conclusion

Deploy Cloud Infrastructure in 30 Seconds with Pulumi AI

Hope Pulumi AI Works with pulumi new

Deploying Cloud Infrastructure with Pulumi AI

Step-by-Step Guide: Deploying with Pulumi AI

FAQs: Pulumi AI and Cloud Deployment

What is Pulumi AI?

How does pulumi new –ai work?

Which cloud providers does Pulumi AI support?

Can I modify the AI-generated infrastructure before deployment?

Where can I provide feedback on Pulumi AI?

Looking Ahead: The Future of Pulumi AI

Pulumi for Platform Teams: New Features for Developer Portals, Policy and Deployments

Organization Templates and New Project Wizard

Backstage Plugin

Compliance Ready Policies

Remediation Policies

Deployments GA

Conclusion

Introducing Pulumi ESC: Easy and Secure Environments, Secrets and Configuration

Pulumi ESC: How It Works & Key Features

Why Use Pulumi ESC for Cloud Secrets & Configuration?

Pulumi ESC Overview

Creating and Managing Environments, Secrets and Configuration

Using Pulumi ESC with Pulumi IaC

The Future of Pulumi ESC

Application-level SDKs

Syncing Configuration to External Systems

Dynamic Configuration Provider Ecosystem

Versioning Environments

Easy-to-Use Key/Value Interface in Pulumi Cloud

Conclusion

Pulumi Insights and AI in the Pulumi CLI

Pulumi Resource Search in the CLI

Pulumi AI in the CLI

Conclusion

Pulumi Kubernetes 4.0: Even More Kubernetes-Native

Server-Side Apply, Upserts and Patch Resources

Improved diff experience

Hope Pulumi AI Works with `pulumi new`

Interactivity using the `pulumi-ai` CLI

Full support for `pulumi convert`

New `str` Package

Pulumi Kubernetes Operator Inline `Program` Support