Pulumi v3.254.0 introduces automatic logging: every operation is logged in an encrypted log file that can optionally be shared with the Pulumi team for inspection. No more re-running commands just to get logs to the Pulumi team for debugging; instead you can share existing logs securely.
You might have been in a situation where pulumi hit an error for an unexpected reason, or did something that was not quite right. Currently the process for trying to resolve that is to try and reproduce the error, ideally now with logging enabled. Sometimes the error doesn’t reproduce, or the state pulumi was in at the time of the error doesn’t exist anymore. And even if the issue reproduces it’s a bit of a hassle to do all this again, just to get logs to Pulumi employees who can do something with them. There’s also no great mechanism to send the potentially sensitive log file.
How it works
From pulumi v3.254.0 onward, we automatically produce log files for every operation and store them in $PULUMI_HOME/logs. These log files are encrypted on disk, using the relevant stack’s secret manager, whenever it is available, as they still contain secrets at this point. The final file consists of gzip’d chunks that are encrypted using AES256-GCM. Log files are gzip’d when no secrets manager is available, as no secrets from property values can be in the log at that point.
Note that the logs are rotated out after 7 days, or after the log directory has reached 500 MB, removing the oldest logs first. This way logs will never fill up your disk, but will still be available after running pulumi commands. These defaults can be overridden with the PULUMI_LOG_ROTATION_MAX_AGE_DAYS and PULUMI_LOG_ROTATION_MAX_TOTAL_MB environment variables.
Locally these logs can be decrypted using pulumi logs decrypt. For this to work the same stack’s secret manager as was used for the command needs to be available.
Sharing logs
Previously there was no good way to securely share the logs with us. Users were always forced to find a way to send the logs to us on their own. With the latest pulumi version, we introduce the pulumi logs share command. This will automatically create a key, safely stored on the server side, and re-encrypt the log with that key, redacting all the secrets by default.
This key can then be accessed by Pulumi employees and Pulumi employees only via an internal tool to decrypt the log. Again we encrypt the log using AES256-GCM. Given this encryption the log can be shared over unsafe channels, and still be secure, even if it’s posted on a GitHub issue.
And while we strive to keep the Pulumi experience as issue free as possible, this should drastically simplify the debugging experience when it is still necessary.
