Troubleshooting Pulumi CI/CD
In order to understand the errors encountered during an automated pipeline execution, it is important to understand the steps involved in a typical CI configuration for Pulumi regardless of the CI service you are using. The type of failure you experience is likely related to one of these steps.
Overall Requirements
In order to run a Pulumi command, the following are required:
- A Pulumi access token for the account you wish to use. Create a token by logging in with the appropriate account.
- A stack that you would like to update the automated pipeline.
- Pulumi CLI available in the system
PATH
. - Build tools (more on this below) based on the runtime of your Pulumi app.
- Dependencies for your Pulumi app.
- The right cloud provider credentials.
Pulumi Access Token
Pulumi has the smarts to know when it is run inside of a CI service by detecting the environment configuration. When this happens,
Pulumi automatically executes a pulumi login
command using the non-interactive mechanism.
Tips
- Ensure that the access token is saved in an environment variable called
PULUMI_ACCESS_TOKEN
.- Most services support marking environment variables as a secret. Your Pulumi Access Token is indeed a sensitive value and as such should be saved as a secret env var.
- Ensure that the environment variable is actually accessible to the job that is running the
pulumi
command. - Several CI systems have the concept of restricting access to environment secrets to specific branches. So make sure that your branch and the job
running the
pulumi
command can access the env var.
Stack Name
Learn about stacks and their configuration.
A stack represents a specific configuration state for your infrastructure resources. For a typical CI pipeline, the stack must have been created
beforehand using the pulumi stack init
command and in the appropriate organization.
Tips
Ensure that the account represented by the token you are using has access to the stack.
This can lead to 404s being returned from the Pulumi Cloud because the token is invalid for any number of reasons. One way to check this is, that you are able to navigate to the stack in the Pulumi Cloud using the same account you are using in CI.
Ensure that you use the fully-qualified stack name when passing the stack name to
pulumi
commands.A fully-qualified stack name is of the format
<org_name>/<project_name>/<stack_name>
. For example, for a stack calledproduction
in an org calledpulumi
and projectslack-bot
, its FQDN ispulumi/slack-bot/production
. Using an FQDN, thought optional, in your automated pipelines removes any ambiguity as to which stack is used, even though Pulumi determines the correct stack by automatically taking into account the project name in yourPulumi.yaml
file. It makes it explicit for anyone in your team looking at the CI configuration.Ensure that you are running the
pulumi
command from the folder containing thePulumi.yaml
file that contains your Pulumi project’s name.If your stack has configuration, then the stack configuration file must also be co-located with the project file.
For example, for a stack named
production
, thePulumi.production.yaml
file must exist alongside thePulumi.yaml
. If your Pulumi app is in a different folder, you can use the--cwd
flag with almost everypulumi
command. Learn more about the global flags.
Installing the Pulumi CLI
Depending on the CI service, there may be a few ways to install the Pulumi CLI. The following CI systems have native extensions that provide
an easy-to-use mechanism for installing and running the various pulumi
commands.
- Azure Pipelines Task Extension - Azure Guide
- GitHub Actions - JavaScript Action, Docker Action - GitHub Actions Guide
Pulumi CLI is now pre-installed on GitHub Actions runners. However, if you need to install a specific version, you can always use one of the aforementioned actions.
- CircleCI Orb - CircleCI Guide
- Octopus Deploy Step Template - Octopus Deploy Guide
- Spinnaker Plugin - Spinnaker Guide
If you are using a CI system that does not have a native extension for installing the CLI, you can always run an inline script step to install the CLI manually.
Tips
Ensure that the
pulumi
CLI is available in the environment across multiple steps.In most cases, the official Pulumi installation script will try to add
pulumi
to the systemPATH
automatically.This problem is especially prevalent for users using a Docker container-based pipeline. With most Docker container pipelines, the state of the container does not persist across steps, so you cannot split-out the installation step of the Pulumi CLI into another step and expect to use the modified environment in the following step. There are exceptions to this. Some CI services allow you to create a reusable template step that you can inject as a pre-cursor to other steps. This is a way to reduce repeating the installation step every time you would like to run
pulumi
command.
Build Tools
Pulumi invokes the build tool that corresponds to your Pulumi project’s runtime. If your pipeline is missing one of these tools,
Pulumi cannot run your infrastructure app. For example, if your Pulumi project uses the nodejs
runtime, then making sure that a valid
version of Node
installed along with npm
or yarn
depending on whichever package manager you are using to manage dependencies.
Tips
Ensure that your Pulumi project’s build tools are installed on your CI runner’s build agent.
Many CI services offer built-in mechanisms to acquire language runtimes of a specific version.
For example, GitHub Actions offers the actions/setup-node
action that can install node
, npm
, and yarn
. Similarly, there is a GitHub Action for installing
the runtime for Python, Go, as well as .NET.
If you are using a Docker container-based build, consider using one of Pulumi’s SDK images for your respective language.
Restoring Dependencies
Regardless of the language you pick for your Pulumi app, you will have dependencies on at least one other package or library. You must ensure that you
restore these dependencies before running a pulumi
command. In the case of .NET
and Go
runtimes, those dependencies are automatically restored for you
when you run pulumi preview
or pulumi update --yes
.
Tips
- For
nodejs
andpython
runtimes, add a step prior to running anypulumi
commands to restore the dependencies. - For
dotnet
andgo
runtimes, the dependencies are restored for you automatically when you runpulumi preview
orpulumi update --yes
. - There is an exception to restoring dependencies automatically for
.NET
when you use a private package feed. You must ensure that the package(s) from the private feed are accessible or you can use a pre-built binary with Pulumi to avoid rebuilding your.NET
solution again.
Note that if you do choose to use a pre-built binary, you will need to install the necessary Pulumi plugins manually using
pulumi plugin install
.
You might be caching the library dependencies but not the Pulumi plugins. Some services offer dependency caching by capturing a specific folder and restoring that folder when your pipeline executes. However, note that Pulumi dependencies have a post-install step that also pulls-down a plugin binary from our CDN. So be sure to cache the plugins path as well.
If in doubt about problems encountered during execution, clear out all caches and restore dependencies from scratch.
Note that depending on the number of providers you use in your Pulumi app, the cache size can get big. Some CI services have restrictions on the max size of the cache.
Cloud Provider Credentials
Cloud provider credentials are another common point of failure. The class of errors related to cloud provider credentials can include:
- Incorrect credentials (wrong account, mismatched keys, etc.)
- Keys with a strict access scope that don’t have access to creating specific resources
Tips
Make sure that the cloud provider credentials are made accessible to
pulumi
using a valid mechanism. In almost all cases, specifying the credentials through environment variables will work.Some cloud providers also support authentication via their respective CLIs. For example, the Azure provider supports authentication via the Azure CLI. But if the Azure CLI is not properly authenticated (account doesn’t have the right role, subscription etc.), then Pulumi will not be able to acquire the necessary credentials.
Ensure that the step with the
pulumi
command can actually access to the environment variables containing the credentials.If your CI service has features that allow you to restrict which pipelines or steps in a pipeline can access the secrets, then ensure that your pipeline/steps can access them. Many CI services allow you to group steps in the form of
jobs
orstages
. If you are using one of those, make sure that thejob
orstage
in which you are running thepulumi
command has the expected environment variables. Sometimes build agent environments get reset across jobs, stages etc. depending on how the CI service you are using works.Check that there are not typos in the name of the environment variables.
Refer to the cloud providers in the Registry docs and the respective setup pages for each to find the correct environment variables to use.
Still need help?
If the above tips don’t help solve your particular issue, then there is always the power of the strong, knowledgeable Pulumi Slack Community. Signup is free and quick. In addition to other community members, various members of the Pulumi team themselves hang out in the Slack channels and would be happy to help you.
Thank you for your feedback!
If you have a question about how to use Pulumi, reach out in Community Slack.
Open an issue on GitHub to report a problem or suggest an improvement.