Troubleshooting Pulumi CI/CD
In order to understand the errors encountered during an automated pipeline execution, it is important to understand the steps involved in a typical CI configuration for Pulumi regardless of the CI service you are using. The type of failure you experience is likely related to one of these steps.
In order to run a Pulumi command, the following are required:
- A Pulumi access token for the account you wish to use. Create a token by logging in with the appropriate account.
- A stack that you would like to update the automated pipeline.
- Pulumi CLI available in the system
- Build tools (more on this below) based on the runtime of your Pulumi app.
- Dependencies for your Pulumi app.
- The right cloud provider credentials.
Pulumi Access Token
Pulumi has the smarts to know when it is run inside of a CI service by detecting the environment configuration. When this happens,
Pulumi automatically executes a
pulumi login command using the non-interactive mechanism.
- Ensure that the access token is saved in an environment variable called
- Most services support marking environment variables as a secret. Your Pulumi Access Token is indeed a sensitive value and as such should be saved as a secret env var.
- Ensure that the environment variable is actually accessible to the job that is running the
- Several CI systems have the concept of restricting access to environment secrets to specific branches. So make sure that your branch and the job
pulumicommand can access the env var.
Learn about stacks and their configuration.
A stack represents a specific configuration state for your infrastructure resources. For a typical CI pipeline, the stack must have been created
beforehand using the
pulumi stack init command and in the appropriate organization.
Ensure that the account represented by the token you are using has access to the stack.
This can lead to 404s being returned from the Pulumi Cloud because the token is invalid for any number of reasons. One way to check this is, that you are able to navigate to the stack in the Pulumi Cloud using the same account you are using in CI.
Ensure that you use the fully-qualified stack name when passing the stack name to
A fully-qualified stack name is of the format
<org_name>/<project_name>/<stack_name>. For example, for a stack called
productionin an org called
slack-bot, its FQDN is
pulumi/slack-bot/production. Using an FQDN, thought optional, in your automated pipelines removes any ambiguity as to which stack is used, even though Pulumi determines the correct stack by automatically taking into account the project name in your
Pulumi.yamlfile. It makes it explicit for anyone in your team looking at the CI configuration.
Ensure that you are running the
pulumicommand from the folder containing the
Pulumi.yamlfile that contains your Pulumi project’s name.
If your stack has configuration, then the stack configuration file must also be co-located with the project file.
For example, for a stack named
Pulumi.production.yamlfile must exist alongside the
Pulumi.yaml. If your Pulumi app is in a different folder, you can use the
--cwdflag with almost every
pulumicommand. Learn more about the global flags.
Installing the Pulumi CLI
Depending on the CI service, there may be a few ways to install the Pulumi CLI. The following CI systems have native extensions that provide
an easy-to-use mechanism for installing and running the various
- Azure Pipelines Task Extension - Azure Guide
Pulumi CLI is now pre-installed on GitHub Actions runners. However, if you need to install a specific version, you can always use one of the aforementioned actions.
- CircleCI Orb - CircleCI Guide
- Octopus Deploy Step Template - Octopus Deploy Guide
- Spinnaker Plugin - Spinnaker Guide
If you are using a CI system that does not have a native extension for installing the CLI, you can always run an inline script step to install the CLI manually.
Ensure that the
pulumiCLI is available in the environment across multiple steps.
In most cases, the official Pulumi installation script will try to add
pulumito the system
This problem is especially prevalent for users using a Docker container-based pipeline. With most Docker container pipelines, the state of the container does not persist across steps, so you cannot split-out the installation step of the Pulumi CLI into another step and expect to use the modified environment in the following step. There are exceptions to this. Some CI services allow you to create a reusable template step that you can inject as a pre-cursor to other steps. This is a way to reduce repeating the installation step every time you would like to run
Pulumi invokes the build tool that corresponds to your Pulumi project’s runtime. If your pipeline is missing one of these tools,
Pulumi cannot run your infrastructure app. For example, if your Pulumi project uses the
nodejs runtime, then making sure that a valid
Node installed along with
yarn depending on whichever package manager you are using to manage dependencies.
Ensure that your Pulumi project’s build tools are installed on your CI runner’s build agent.
Many CI services offer built-in mechanisms to acquire language runtimes of a specific version.
For example, GitHub Actions offers the
actions/setup-node action that can install
yarn. Similarly, there is a GitHub Action for installing
the runtime for Python, Go, as well as .NET.
If you are using a Docker container-based build, consider using one of Pulumi’s SDK images for your respective language.
Regardless of the language you pick for your Pulumi app, you will have dependencies on at least one other package or library. You must ensure that you
restore these dependencies before running a
pulumi command. In the case of
Go runtimes, those dependencies are automatically restored for you
when you run
pulumi preview or
pulumi update --yes.
pythonruntimes, add a step prior to running any
pulumicommands to restore the dependencies.
goruntimes, the dependencies are restored for you automatically when you run
pulumi update --yes.
- There is an exception to restoring dependencies automatically for
.NETwhen you use a private package feed. You must ensure that the package(s) from the private feed are accessible or you can use a pre-built binary with Pulumi to avoid rebuilding your
Note that if you do choose to use a pre-built binary, you will need to install the necessary Pulumi plugins manually using
pulumi plugin install.
You might be caching the library dependencies but not the Pulumi plugins. Some services offer dependency caching by capturing a specific folder and restoring that folder when your pipeline executes. However, note that Pulumi dependencies have a post-install step that also pulls-down a plugin binary from our CDN. So be sure to cache the plugins path as well.
If in doubt about problems encountered during execution, clear out all caches and restore dependencies from scratch.
Note that depending on the number of providers you use in your Pulumi app, the cache size can get big. Some CI services have restrictions on the max size of the cache.
Cloud Provider Credentials
Cloud provider credentials are another common point of failure. The class of errors related to cloud provider credentials can include:
- Incorrect credentials (wrong account, mismatched keys, etc.)
- Keys with a strict access scope that don’t have access to creating specific resources
Make sure that the cloud provider credentials are made accessible to
pulumiusing a valid mechanism. In almost all cases, specifying the credentials through environment variables will work.
Some cloud providers also support authentication via their respective CLIs. For example, the Azure provider supports authentication via the Azure CLI. But if the Azure CLI is not properly authenticated (account doesn’t have the right role, subscription etc.), then Pulumi will not be able to acquire the necessary credentials.
Ensure that the step with the
pulumicommand can actually access to the environment variables containing the credentials.
If your CI service has features that allow you to restrict which pipelines or steps in a pipeline can access the secrets, then ensure that your pipeline/steps can access them. Many CI services allow you to group steps in the form of
stages. If you are using one of those, make sure that the
stagein which you are running the
pulumicommand has the expected environment variables. Sometimes build agent environments get reset across jobs, stages etc. depending on how the CI service you are using works.
Check that there are not typos in the name of the environment variables.
Refer to the cloud providers in the Registry docs and the respective setup pages for each to find the correct environment variables to use.
Still need help?
If the above tips don’t help solve your particular issue, then there is always the power of the strong, knowledgeable Pulumi Slack Community. Signup is free and quick. In addition to other community members, various members of the Pulumi team themselves hang out in the Slack channels and would be happy to help you.