Pulumi Blog: Abhinav Gupta

Aligning Projects between Service and DIY Backend

Justin Van Patten — Mon, 03 Apr 2023 00:00:00 +0000

At Pulumi, our goal is to offer the best Infrastructure as Code experience for all cloud developers. From the very beginning, we’ve believed that the best IaC experience is made possible by combining a great open source SDK and CLI with a great backend management service. This is why we built Pulumi Cloud, a rich management platform for your Infrastructure as Code, which includes a forever free option for individuals, a generous free tier for teams, and critical tools for enterprises to manage IaC at scale.

Over the last few years, we’ve continued to expand the features of the Pulumi Service - with Deployments, Audit Logs, SAML SSO and SCIM, Teams, Stack Transfers, Favorites, Organization and Team Access Tokens and much more.

While the majority of Pulumi users do choose to use the Pulumi Service, we also know that there are good reasons why some organizations would prefer to use Pulumi IaC alone without the Pulumi Service. And so we support and continue to invest in enabling a variety of additional backends that allow the Pulumi CLI to be used with state stored in the local filesystem or in cloud storage like S3, Azure Blob Storage, or Google Cloud Storage.

Historically the Pulumi Service backend and the DIY backend have differed in their handling of “projects”. The Pulumi Service stores state for a Pulumi stack in a separate namespace per project. The DIY backends have historically stored all stacks in a single namespace across all projects. This inconsistency has been a common source of confusion for users getting started with Pulumi when using the file storage backends.

Today, we are aligning how projects are managed across all backends, adding Project-Scoped Stacks support to DIY backends.

Background

This section goes over some of the background necessary to understand project-scoping and why it’s important. Feel free to skip ahead to the next section if you’re already familiar with this.

Projects and stacks

Pulumi projects are a unit of organization. A project holds the code and configuration for your infrastructure, and can be used to deploy multiple stacks. A stack is a single, isolated instance of your infrastructure.

For example, you might have an “app” project for your application, deploying to the “test” and “prod” stacks.

What’s a state backend?

Each stack tracks metadata about your infrastructure so it can manage your cloud resources. This metadata is called state and it is stored in a backend of your choosing: Service or DIY.

The Service backend is hosted at app.pulumi.com or on your own server. It provides the best combination of usability, safety, and security for most users.

The DIY backend stores state in a JSON file, deferring to you to manage persistence of this information. You can store it in Amazon S3, Azure Blob Store, Google Cloud Storage, a system such as Minio or Ceph with an API compatible with Amazon S3, or on your local filesystem. Using this backend trades reliability and convenience of the Service backend for additional control over where the information is stored.

If you’re unsure of which backend to use, see Deciding On a State Backend.

Project-scoped stacks

Depending on your needs, you can have several Pulumi projects. For example, you might have a “net” project to set up an Amazon VPC, a “k8s” project to set up a Kubernetes cluster deployed to it, and an “app” project to build and deploy your application to the cluster. Each of these projects will have their own stacks, e.g. “test” and “prod”.

The Pulumi Service backend scopes stacks to the project they belong to, so we can re-use the names “test” and “prod” across different projects.

This was not the case for the DIY backend, where stack names were global across all projects. To work around this, users resorted to qualifying stack names in the form <project>-<stack> when using the DIY backend.

We’re pleased to announce that such workarounds are no longer necessary!

Project-scoped stacks in DIY backends

With the latest release of Pulumi, we’ve added support for project-scoped stacks to the DIY backend. Stacks created in new or empty DIY backends will automatically use the new project-scoped layout.

With this change, you can now use stack names like “test” and “prod” across your projects without risk of conflict—the same way you do with the Service backend.

Upgrading existing backends

Only stacks created in new or empty DIY backends will scope stacks to projects. Existing backends will continue to operate as they do today.

If you have an existing DIY backend that you’d like to upgrade to use project-scoped stacks, you can do so with the new pulumi state upgrade command, which upgrades all stacks in the backend to use the new project-scoped layout.

$ pulumi state upgrade
This will upgrade the current backend to the latest supported version.
Older versions of Pulumi will not be able to read the new format.
Are you sure you want to proceed?
Please confirm that this is what you'd like to do by typing `yes`:

Once stacks in a DIY backend are upgraded, older versions of the Pulumi CLI will be unable to access those stacks. This change cannot be reverted.

Cleaning up old stack names

If you previously qualified stack names with the project name to avoid conflicts, you can use the pulumi stack rename command to clean up these names after the upgrade.

$ pulumi stack rename --stack app-test test

Manage stacks across your projects

Organize your infrastructure into projects and reuse stack names like test and prod across them. Store state in Pulumi Cloud or a self-managed backend, whichever fits your team.

Get started

Referencing stacks in the CLI

Many commands in the Pulumi CLI expect a stack name to be specified. These commands accept a fully-qualified stack name in the form:

<org>/<project>/<stack>

Where, <org> is the organization name, <project> the project name, and <stack> the stack name.

They also accept one of the following shorthand forms if we’re inside the project directory:

<org>/<stack>
<stack>

For instance, given a stack “prod” inside the “app” project, the following are all valid ways to refer to it:

my-org/app/prod
my-org/prod // if inside the app directory
prod // if inside the app directory

With the latest release of Pulumi, the above is true for both the Service backend and the DIY backend.

Referencing DIY project-scoped stacks

All projects in a DIY backend are placed under a virtual organization named “organization”. This value is constant and cannot be changed.

Repeating the above examples with a DIY backend, we can use any of the following forms to refer to the same stack:

organization/app/prod
organization/prod // if inside the app directory
prod // if inside the app directory

Availability

Support for project-scoped stacks in DIY backends is now available in Pulumi v3.61.0. Give it a try and share your thoughts with us on the Pulumi Community Slack!

Streamline access to stack outputs with OutputDetails

Abhinav Gupta — Fri, 24 Mar 2023 00:00:00 +0000

You can use the new StackReference.getOutputDetails method to get output values from other stacks directly—without calling Output.apply.

What are Stack References?

If you already understand stack references, skip on to Introducing OutputDetails.

Stack references are a useful feature of Pulumi that you can use to access output values from one stack in another.

For instance, let’s say you have a ’network’ stack that provisions an AWS VPC, and an ‘app’ stack that deploys your application on an EC2 instance in that VPC. The app stack can reference the output of the network stack to obtain the VPC ID, which it needs to configure the EC2 instance.

const network = new pulumi.StackReference("acmecorp/network/other");
const vpcId = network.getOutput("vpcId");

The getOutput method above returns an Output value that holds the result. You can call Output.apply to perform transformations on the output value. As an example, if you need to extract a subnet ID from a JSON string in the output, you can use the following code:

const subnetConfig = network.getOutput("subnetConfig");
const subnetId = subnetConfig.apply(output => JSON.parse(output).subnetId);

This approach works but is limited because the output value obtained from the apply transformation is also an Output value. You cannot access or manipulate this value directly, and you must continue to use apply repeatedly to consume the value.

As an aid in resolving this problem, we have introduced OutputDetails. These provide direct access to the output of one stack from another stack.

Introducing `OutputDetails`

We’ve added a new method, StackReference.getOutputDetails, to allow direct access to output values from stack references. This method can be used as follows:

const vpcIdDetails = await network.getOutputDetails("vpcId");
const vpcId = vpcIdDetails.value;

Unlike the getOutput method, the resulting value is a plain value, not an Output value. As a result, there is no need to call apply repeatedly to transform or consume the value.

Revisiting the previous example of extracting a value from a JSON string, getOutputDetails can be used as shown below:

const subnetConfig = (await network.getOutputDetails("subnetConfig")).value;
const subnetId = JSON.parse(subnetConfig).subnetId;
console.log(`Using subnet ID ${subnetId}`);

What about secrets?

getOutputDetails can also be used to access [secret outputs generated by the referenced stack. For example, if the referenced stack sets up a database and produces the password as a stack output, you can access it as follows:

const passwordDetails = await dbStack.getOutputDetails("databasePassword");
const password = passwordDetails.secretValue;

Can we do this for all `Output` values?

Unfortunately, it is not feasible to do this to every Output value. Stack references are unique in that the referenced stack must already have deployed for the reference to be resolved. Therefore, when the stack reference is created, the outputs of the referenced stack are available immediately. This is not the case for other Output values. Their values only become available as pulumi update runs, so we cannot access them in the same way.

Availability

This feature is available in all stable SDKs for Pulumi, starting from v3.55.0. Give it a spin and share your thoughts with us on Slack!

Pulumi Blog: Abhinav Gupta

Aligning Projects between Service and DIY Backend

Background

Projects and stacks

What’s a state backend?

Project-scoped stacks

Project-scoped stacks in DIY backends

Upgrading existing backends

Cleaning up old stack names

Referencing stacks in the CLI

Referencing DIY project-scoped stacks

Availability

Streamline access to stack outputs with OutputDetails

What are Stack References?

Introducing OutputDetails

What about secrets?

Can we do this for all Output values?

Availability

Introducing `OutputDetails`

Can we do this for all `Output` values?