Pulumi API

The Pulumi API is one of the components required for self-hosting Pulumi in your organization’s environment. It provides the necessary APIs for both the CLI and the Console.

Prerequisites

  • Provide a server or virtual machine to install and run the Pulumi components (see Minimum System Requirements below).
  • Provide a persistent volume for the service to store checkpoint objects.
  • Provider a persistent volume for the MySQL data (optional if you are providing your own DB.)
  • If you are providing your own DB instance, ensure that it is accessible within the same Docker network that the service and the UI containers will be running in.
    • The default DB endpoint is pulumi-db:3306. If you wish to change this, set PULUMI_LOCAL_DATABASE_NAME and PULUMI_LOCAL_DATABASE_PORT accordingly (see Script Variables.)
    • If you do not create this network prior to running run-ee.sh, it will create only a bridged network on your local host. Ensure that the DB can be accessed by the API service container.
  • Provide an external load balancer with TLS termination.

Minimum System Requirements

TypePropertiesNotes
Compute2 CPU cores w/ 8 GB memory
Storage20GB SSDFor MySQL data.

Note: By default, the installation uses a single data path via PULUMI_DATA_PATH to map both the SQL data volume and the object storage path. Specify a volume for the db container as required.
Storage200GB SSDFor Object Storage.

Note: By default, the installation uses a single data path via PULUMI_DATA_PATH to map both the SQL data volume and the object storage path. A dedicated path can be set via env var PULUMI_LOCAL_OBJECTS for the api container.

Note: The storage recommendations for the Object Storage can be lesser than 200GB depending on your organization size and the expected usage.

What’s In The Container?

The API service is a Go-based application. This is a single binary application that has all of the dependencies it needs in order to run.

Server

This container runs an HTTP server which provides the APIs needed by the Console and the CLI. By default, this container will serve using port 8080 over standard HTTP. If TLS is configured the server will instead serve over port 8443 using HTTPS.

Environment Variables for Core Infrastructure

The core infrastructure components to successfully run the API service are the database, object storage, and encryption services. Depending on your requirements, you can configure additional (optional) identity services as well as enhanced security between the API and the database.

Variable NameDescription
PULUMI_LICENSE_KEYThe license key value. A JWT string.

Note: Be sure to enclose the value in single-quotes.
PULUMI_DATABASE_ENDPOINTThe database server endpoint in the format host:port. This should be a MySQL 5.6 server.
PULUMI_DATABASE_NAMEThe name of the database on the database server.
PULUMI_API_DOMAINThe internet or network-local domain using which the API service can be reached, e.g. pulumiapi.acmecorp.com. Default is localhost:8080.
PULUMI_CONSOLE_DOMAINThe internet or network-local domain using which the Console can be reached, e.g. pulumiconsole.acmecorp.com. Default is localhost:3000.

Object storage

The service saves the state of every stack, that uses it as the backend, to a persistent object storage. If your org uses policy packs, then anytime a user publishes a policy pack, it gets uploaded to the same object storage service as well. Both the state (checkpoint) and the policy pack must use different buckets on the same object storage service.

Local storage

Variable NameDescription
PULUMI_LOCAL_OBJECTSFolder path for persisting state for stacks. Ensure that this path is highly available, and backed-up regularly. Only use this if you want the service to persist objects to a local path.
PULUMI_POLICY_PACK_LOCAL_HTTP_OBJECTSFolder path for persisting published policy packs. Ensure that this path is highly available, and backed-up regularly. Only use this if you want the service to persist policy packs to a local path.

AWS S3

Variable NameDescription
PULUMI_CHECKPOINT_BLOB_STORAGE_ENDPOINTThe storage endpoint for persisting stack state/checkpoint. The value takes the format: s3://<bucket-name>. The s3:// scheme also supports query-params. See the GoCloud docs for an example of the query-params.
PULUMI_POLICY_PACK_BLOB_STORAGE_ENDPOINTThe storage endpoint for persisting published policy packs. The value takes the format: s3://<bucket-name>. Similar to PULUMI_CHECKPOINT_BLOB_STORAGE_ENDPOINT, this also supports query-params.

Azure Storage

Variable NameDescription
PULUMI_CHECKPOINT_BLOB_STORAGE_ENDPOINTThe storage endpoint for persisting stack state/checkpoint. The value takes the format: azblob://<blob-container>.
PULUMI_POLICY_PACK_BLOB_STORAGE_ENDPOINTThe storage endpoint for persisting published policy packs. The value takes the format: azblob://<blob-container>.
AZURE_STORAGE_ACCOUNTThe name of the Azure storage account where the blob containers for checkpoint and policy pack have been created. See Cloud Provider Authentication for additional configuration options for Azure.
AZURE_STORAGE_KEY(Optional) The primary or secondary storage key for the storage account. You only need to specify this if you are not using Azure MSI.

Encryption services

The service supports using a master key available in a local-path or in a remote key management service. You only need to configure one of the support services.

Local keys

Variable NameDescription
PULUMI_LOCAL_KEYSFolder path that contains a random 32-byte key. Ensure that this path is highly available, and backed-up regularly.

AWS KMS

Variable NameDescription
PULUMI_KMS_KEYARN for the AWS KMS customer master key resource.

Azure KeyVault

Variable NameDescription
PULUMI_AZURE_KV_URIAzure KeyVault URI. For example, https://<vault-name>.vault.azure.net.
PULUMI_AZURE_KV_KEY_NAMEThe name of the key in KeyVault. The key must be an RSA key type. We recommend a key size of 2048 for most cases. The key operations must support Encrypt and Decrypt. Otherwise, the service will fail to start.
PULUMI_AZURE_KV_KEY_VERSIONThe version of the key that the service should use. Note: All previous versions of the key must remain enabled.

Cloud Provider Authentication

These settings are required if you are running the Pulumi Service on one of these clouds or using one of their services.

AWS

For more information about authenticating with AWS services, see the AWS SDK docs.

Variable NameDescription
AWS_REGIONThe region where the dependent AWS services used by Pulumi Service have been created. For example, the KMS key must exist in this region. Similarly, if you are using RDS, then there must be a writable cluster in this region.
AWS_ACCESS_KEY_IDThe AWS access key ID.
AWS_SECRET_ACCESS_KEYThe AWS secret key.
AWS_PROFILEThe AWS profile.
AWS_ROLE_ARNThe AWS role ARN to assume using the base access keys.

Azure

Variable NameDescription
AZURE_CLIENT_IDClient ID of the Azure Managed System Identity or the self-managed Service Principal (SP) that should be used for both Azure KeyVault and Storage. The MSI or the SP must have access to Key Management operations (Get, Encrypt and Decrypt) of the KeyVault you plan to use.
AZURE_CLIENT_SECRET(Optional) Client secret of the Azure SP.
AZURE_TENANT_ID(Optional) Tenant ID of the Azure SP.
AZURE_SUBSCRIPTION_ID(Optional) Subscription ID of the Azure AP.
AZURE_STORAGE_DOMAIN(Optional) The custom domain for your storage domain, if any. If this is not provided, the default Azure public domain “blob.core.windows.net” will be used.

Other Environment Variables

Variable NameDescription
PULUMI_DATABASE_USER_NAMEName of the database user the Pulumi Service connects as. Leave default unless you are having trouble connecting to your database.
PULUMI_DATABASE_USER_PASSWORDPassword of the database user the Pulumi Service connects as. Leave default unless you are having trouble connecting to your database.
RECAPTCHA_SECRET_KEYreCAPTCHA secret key for self-service password reset. Create a site key and a secret key from Google.
SAML_CERTIFICATE_PUBLIC_KEYPublic key used by the IdP to sign SAML assertions. Learn how to set SAML_CERTIFICATE_PUBLIC_KEY.
SAML_CERTIFICATE_PRIVATE_KEYPrivate key used by Pulumi to validate the SAML assertions sent by the IdP. Learn how to set SAML_CERTIFICATE_PRIVATE_KEY.
GITHUB_OAUTH_ENDPOINTUsed for GitHub API calls.
GITLAB_OAUTH_ENDPOINTUsed for GitLab API calls.

TLS Environment Variables

API Service

The service is configurable to serve the API using TLS. The following environment variables are all required in order to enable TLS. The values of the environment variables may either be a filepath or the actual value of the entity. If these variables are set, the service will be served over HTTPS (i.e using TLS) using port 8443. If the following variables are not set the service will default to serving over HTTP using port 8080.

Variable NameDescription
API_TLS_CERTIFICATEThe TLS certificate. The certificate must be supplied in X.509 format and must be PEM encoded.
API_TLS_PRIVATE_KEYThe private key associated with the TLS certificate. The private key must be PEM encoded.
API_MIN_TLS_VERSIONThe minimum version of TLS to use when serving the API (must be in <major>.<minor> format, e.g. 1.2).

Note: Self-signed certificates may be used to configure TLS in the event the need for a trusted entity is not necessary. A self-signed cert and private key may be generated using OpenSSL. The following command uses OpenSSL to generate a self-signed certificate. This example will output two files, the certificate (cert.pem) and the private key (key.pem) used to sign it.

openssl \
  req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem \
  -days { days_unitl_expiration } -nodes -subj "/CN={ common_name }"

Database Connections

API Service

The service is configurable to enable connections to the backend SQL database over TLS. The following environment variables are all required to connect to the database using TLS. If these variables are set the service will establish connections to the database using TLS, otherwise the service will default to connecting without TLS. The same ports will be used for communication to the database regardless of whether TLS is configured or not.

Variable NameDescription
DATABASE_CA_CERTIFICATEThe CA certificate used to establish TLS connections with the database. This certificate must be PEM encoded. This must be set to the value of the certificate itself and not a filepath to the location of the certificate file.
DATABASE_MIN_TLS_VERSIONThe minimum TLS version to use for database connections (must be in <major>.<minor> format, e.g. 1.2).

Migrations

The database migrations container is configurable to enable connections to the database over TLS. To use TLS, the following environment variable must be set. The default is to not use TLS.

Variable NameDescription
DATABASE_CA_CERTIFICATEThe CA certificate used to establish TLS connections with the database. This certificate must be PEM encoded. This must be set to the value of the certificate itself and not a filepath to the location of the certificate file.
MYSQL_ROOT_USERNAMEThe root username to log in to the MySQL database. Defaults to root.
MYSQL_ROOT_PASSWORDThe root user password to log in to the MySQL database.
MYSQL_ALLOW_EMPTY_PASSWORDSet to true to allow the container to be started with a blank password for the root user.
PULUMI_DATABASE_ENDPOINTThe database server endpoint in the format host:port. This should be a MySQL 5.6 server.
PULUMI_DATABASE_PING_ENDPOINTThe database server endpoint to ping for availability before login.
RUN_MIGRATIONS_EXTERNALLYRequest for migrations to be run against an external database.