State and Backends
Pulumi stores its own copy of the current state of your infrastructure. This is often simply called state, and is stored in transactional snapshots we call checkpoints. A checkpoint is recorded by Pulumi at various points so that it can operate reliably, whether that means diffing goal state versus current state during an update, recovering from failure, or destroying resources accurately to clean up afterwards, for example. Because state is critical to how Pulumi operates, this page gives an overview of the options.
Pulumi supports multiple so-called backends for storing this state. There are two primary kinds:
- Web: a backend accessed by the CLI through REST API calls
- Filesystem: a backend stored locally on your computer’s filesystem
By default, the CLI uses a web backend hosted at https://app.pulumi.com. This is a free service offered by Pulumi that also offers advanced tiers for team and enterprise features. It is possible to opt-in to the filesystem backend.
We think that using the web backend and the CLI together provides the right combination of usability, safety, and security for most users. But the filesystem backend provides more flexibility for those who need it.
Web Backend Features
The web backend stores all checkpoint state securely, and the CLI always communicates directly with your cloud provider using client-side authentication. All state is encrypted in transit and at rest.
Although backends are primarily responsible for storing state reliably, the web backend also supports additional features that the filesystem backend does not. In particular, the web backend supports keeping track of full deployment history (for auditing and rollback purposes), and supports concurrent locking so that you don’t accidentally corrupt your infrastructure state when using Pulumi in a team environment. It also supports advanced policy and RBAC.
The app.pulumi.com architecture can be visualized as follows:
The Pulumi Enterprise product offers self-hosting options for the web backend, if you wish to use these features without depending on app.pulumi.com. The Enterprise web architecture looks like the following:
For more information on Pulumi Enterprise, please contact us.
pulumi login command lets you log into a backend. By default, anytime you try to do something that requires
stacks or state, you will be prompted to log in.
To the Web Backend
The web backend login process involves using access tokens. The prompt looks like the following:
$ pulumi login Manage your Pulumi stacks by logging in. Run `pulumi login --help` for alternative login options. Enter your access token from https://app.pulumi.com/account or hit <ENTER> to log in using your browser :
If you hit
<ENTER> as instructed, a web browser will pop up, and will interact with the service to generate a new
access token. If this is your first time using the service, you will need to authenticate.
If you wish to get a token manually, or view your generated tokens, you may go to https://app.pulumi.com/account/tokens. This page will show you all past tokens, when they were last used, and allow you to revoke them:
After logging in, state will automatically get persisted with the service, and from time to time, you will see a helpful URL to your update or stack pages. You can always go there to see a full history of updates.
To log into a privately hosted version of Pulumi Enterprise, simply add its URL to the command:
$ pulumi login https://pulumi.acmecorp.com
Everything else works the same, except that URLs will target your private service instead of app.pulumi.com.
To the Filesystem Backend
The filesystem backend allows you to store state locally to your machine. Because it is stored locally, it’s up to you to back it up, share it across machines, and synchronize access to it properly in a team environment. We built the web backend to solve all of these problems “out of the box,” but we understand that some users will want more control.
To opt into using the filesystem backend, simply pass the
--local flag when logging in:
$ pulumi login --local Logged into my-machine as myuser (file://~)
This will store all stack checkpoints underneath my home directory, in the
.pulumi directory, as JSON files.
If you’d like to control where these checkpoints get stored, you may instead log into a
<path> is the full path to the target directory under which files will get stored. For instance, to store
/app/data/.pulumi/ instead, you can run this command:
$ pulumi login file:///app/data Logged into my-machine as myuser (file:///app/data)
If you use a relative path (e.g.
file://./einstein), Pulumi will always make it relative to the current working directory!
pulumi login --local is simply syntactic sugar for
pulumi login file://~.
The precise JSON format these checkpoint files use is not documented, but is defined in source code here if you’d like to understand their contents. Note that
this is the same JSON format used by the
pulumi stack export and
pulumi stack import commands.
If you lose the checkpoint for your stack, Pulumi will be unable to manage any existing resources! Additionally, since Pulumi will believe your stack is empty, the next update will attempt to re-create all of the resources in your stack.
Some commands may behave slightly differently when using the local endpoint. For example, when connected to pulumi.com,
pulumi up ensures there are no other updates in flight for a given stack, something that doesn’t happen with the
local endpoint. Secrets are also managed using a key encrypted with a passphrase and stored in
Pulumi.<stack-name>.yaml. This requires you to enter the passphrase when you preview, update or delete your stack.
If you want to collaborate with another person, you’ll need to share this passphrase with them as well.
Going Back to the Web Backend
Some users initially try out Pulumi using the filesystem backend and then move to the web backend when it’s clear they will benefit from the features that it delivers (particularly when operationalizing their usage).
To use the web backend, just run
pulumi login again, and you’ll be back to using app.pulumi.com. If you have any
existing stacks, however, you’ll need to migrate them. It’s easiest to just plan on recreating them.
If you’d like to migrate your stacks from the filesystem to web, however, you can do the following. Suppose the stack “my-app-production” has been managed with a local checkpoint file, and you want to migrate it to pulumi.com. If you are currently logged in to the local endpoint, run the following commands:
$ pulumi stack select my-app-production # switch to the stack we want to export $ pulumi stack export --file my-app-production.checkpoint.json # export the stack's checkpoint to a local file $ pulumi login $ pulumi stack init my-app-production # create a new stack with the same name on pulumi.com $ pulumi stack import --file my-app-production.checkpoint.json # import the new existing checkpoint into pulumi.com
In addition, if you have any encrypted configuration in your stack, you’ll need to re-run
pulumi config set --secret <key> <value> because pulumi.com uses a different key to encrypt your secrets than the
When a secret value is provided via secret configuration (by passing –secret to pulumi config set) or created inside your program by using
Output.secret (Python) the value is encrypted with a key managed by the backend you are connected to. When using the local backend, this key is derived from a passphrase you set when creating your stack, and when using the web backend, it is handled by a key managed by the service.
For new stacks managed with the web backend, you may choose to use the passphrase based key instead. Pass
--secrets-provider passphrase when you create the stack (either via
pulumi new or
pulumi stack init). You’ll be prompted to choose a passphrase and future operations like
destroy will require you enter this passphrase before the operation will run.
When using the filesystem backend, you must use the passphrase based secrets provider.