Pulumi Cloud self-hosted API

The Pulumi API is one of the components required for self-hosting Pulumi Cloud in your organization’s environment. It provides the necessary APIs for both the CLI and the Console.

Prerequisites

• Provide a server or virtual machine to install and run the Pulumi components (see Minimum System Requirements below).
• Provide a persistent volume for the service to store checkpoint objects.
• Provider a persistent volume for the MySQL data (optional if you are providing your own DB.)
• If you are providing your own DB instance, ensure that it is accessible within the same Docker network that the service and the UI containers will be running in.
  • The default DB endpoint is pulumi-db:3306. If you wish to change this, set PULUMI_LOCAL_DATABASE_HOST and PULUMI_LOCAL_DATABASE_PORT accordingly (see Script Variables.)
  • If you do not create this network prior to running run-ee.sh, it will create only a bridged network on your local host. Ensure that the DB can be accessed by the API service container.
• Provide an external load balancer with TLS termination.

Minimum System Requirements

Note: The storage recommendations for the Object Storage can be lesser than 200GB depending on your organization size and the expected usage.

What’s In The Container?

The API service is a Go-based application. This is a single binary application that has all of the dependencies it needs in order to run.

Server

This container runs an HTTP server which provides the APIs needed by the Console and the CLI. By default, this container will serve using port 8080 over standard HTTP. If TLS is configured the server will instead serve over port 8443 using HTTPS.

Function Mode

It’s possible to split apart the HTTP Server from the Background Jobs functionality to distribute responsibility across multiple instances. The function mode determines the mode in which the server runs based upon the env variable PULUMI_API_CONTAINER_MODE. If this value is not defined, or doesn’t match a valid option, then it defaults to FULL.

Environment Variables for Core Infrastructure

The core infrastructure components to successfully run the API service are the database, object storage, and encryption services. Depending on your requirements, you can configure additional (optional) identity services as well as enhanced security between the API and the database. The API also supports exporting OpenTelemetry metrics and traces via the OpenTelemetry collector.

Object storage

The service saves the state of every stack, that uses it as the backend, to a persistent object storage. If your org uses policy packs, then anytime a user publishes a policy pack, it gets uploaded to the same object storage service as well. Both the state (checkpoint) and the policy pack must use different buckets on the same object storage service.

Local storage

AWS S3

Azure Storage

Encryption services

The service supports using a master key available in a local-path or in a remote key management service. You only need to configure one of the support services.

Local keys

AWS KMS

Mandatory key actions

Define the following actions in the key’s key policy in AWS KMS. Otherwise, the service will fail to start or will not be able to run crypto operations:

• kms:Encrypt
• kms:Decrypt
• kms:GenerateDataKey
• kms:GenerateDataKeyWithoutPlaintext

Azure KeyVault

Mandatory key operations or permissions

Configure the key in Azure KeyVault to support and allow the following operations. Otherwise, the service will fail to start or will not be able to run crypto operations:

• Encrypt
• Decrypt
• Sign
• Verify

Cloud Provider Authentication

These settings are required if you are running Pulumi Cloud on one of these clouds or using one of their services.

AWS

For more information about authenticating with AWS services, see the AWS SDK docs.

Azure

GitLab OAuth

Only required if using GitLab as the backing identity provider for your organization.

Sending Emails

Other Environment Variables

SSRF protection

To protect against server-side request forgery (SSRF), the API service refuses outbound requests to URLs that users supply when the destination resolves to a loopback, private, link-local, carrier-grade NAT (RFC 6598), or other reserved address. This includes the cloud metadata endpoint 169.254.169.254. The address is checked when the connection is made, so a hostname that resolves to a public address when it’s saved and to a private one later is still refused.

If a feature needs to reach a service on the same host or private network as your installation, such as a secret store that a Pulumi ESC provider reads from, switch the protection off for that feature only by listing it in PULUMI_DISABLE_SSRF_PROTECTION. Names are case-insensitive and unknown names are ignored with a warning in the API service logs.

For example, to let ESC providers and Neo reach services on your private network:

PULUMI_DISABLE_SSRF_PROTECTION=ESC,AGENTS_BYOK

The API service logs a warning once for each scope that is switched off. PULUMI_DISABLE_ESC_SSRF_PROTECTION=true is still honored as ESC, but it’s deprecated and no longer switches off the protection for any other feature.

Outbound proxies

Requests covered by SSRF protection honor the standard HTTPS_PROXY, HTTP_PROXY, and NO_PROXY environment variables. The proxy itself may be on a private address. The API service checks the destination before handing the request to the proxy, so a blocked destination never reaches the proxy, and a request sent directly to the proxy’s own address is refused.

Two limitations apply when a proxy is configured:

• The API service must be able to resolve the destination’s hostname itself. If only the proxy can resolve external names, protected requests fail. Switch off the protection for the affected feature if that applies to your network.
• The proxy resolves the hostname again after the API service has checked it. To guard against DNS rebinding behind a proxy, restrict private destinations with an egress policy on the proxy.

TLS Environment Variables

API Service

The service is configurable to serve the API using TLS. The following environment variables are all required in order to enable TLS. The values of the environment variables may either be a filepath or the actual value of the entity. If these variables are set, the service will be served over HTTPS (i.e using TLS) using port 8443. If the following variables are not set the service will default to serving over HTTP using port 8080.

Note: Self-signed certificates may be used to configure TLS in the event the need for a trusted entity is not necessary. A self-signed cert and private key may be generated using OpenSSL. The following command uses OpenSSL to generate a self-signed certificate. This example will output two files, the certificate (cert.pem) and the private key (key.pem) used to sign it.

openssl \
  req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem \
  -days { days_until_expiration } -nodes -subj "/CN={ common_name }"

Database Connections

API Service

The service is configurable to enable connections to the backend SQL database over TLS. The following environment variables are all required to connect to the database using TLS. If these variables are set the service will establish connections to the database using TLS, otherwise the service will default to connecting without TLS. The same ports will be used for communication to the database regardless of whether TLS is configured or not.

Migrations

The database migrations container is configurable to enable connections to the database over TLS. To use TLS, the following environment variable must be set. The default is to not use TLS.

Audit Logs

Audit Logs are persisted in MySQL by default. Alternative backends can be configured to support a higher volume of writes without scaling out MySQL.

DynamoDB

To use AWS DynamoDB to persist Audit Logs, specify the name of the table in the environment variable PULUMI_AUDIT_LOGS_DYNAMO_TABLE. The table provided must be configured with following attributes:

  hashKey: "org_id",
  rangeKey: "timestamp_id",
  globalSecondaryIndexes: [
      {
          hashKey: "org_user_id",
          projectionType: "ALL",
          name: "org_user",
          rangeKey: "timestamp_id",
      },
  ],
  attributes: [
      {
          name: "org_id",
          type: "S",
      },
      {
          name: "org_user_id",
          type: "S",
      },
      {
          name: "timestamp_id",
          type: "S",
      },
  ],

OpenTelemetry

The API service is configured to export OpenTelemetry metrics and traces to the vendor of your choice via the OpenTelemetry collector. You will need to manage your own OpenTelemetry collector.

The following environment variables are needed to configure OpenTelemetry in the service:

OpenTelemetry is not yet available for the console service.

Metrics endpoint

The API service exposes a metrics endpoint (https://api.pulumi.com/metrics) that is secured by a bearer token. This token is configured using the environment variable METRICS_WEBHOOK_SECRET.

curl -s GET https://api.pulumi.com/metrics -H 'Authorization: webhook-token <METRICS_WEBHOOK_SECRET>

Prometheus

The API service provides two options to get metrics into Prometheus:

1. From the OpenTelemetry collector via a Prometheus remote write exporter. This is a push-based exporter.
2. From a Prometheus exporter by scraping the /metrics endpoint. This is a pull-based exporter.

Prometheus remote write exporter

This option does not use the /metrics endpoint. Instead, it exports metrics from the collector to a Prometheus remote write compatible backend.

Example OpenTelemetry collector configuration for a service using AWS and the AWS Distro for OpenTelemetry Collector:

extensions:
  sigv4auth:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: localhost:4317

processors:
  memory_limiter:

  batch:

exporters:
  logging:

  prometheusremotewrite:
    endpoint: https://aws-managed-prometheus-endpoint/v1/api/remote_write
    auth:
      authenticator: sigv4auth

service:
  telemetry:
    logs:

  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [logging]

    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [prometheusremotewrite]

  extensions: [sigv4auth]

Prometheus exporter

This option requires configuring the environment variable METRICS_WEBHOOK_SECRET to successfully authenticate to the /metrics endpoint.

Example OpenTelemetry collector configuration:

extensions:
  bearertokenauth:
    scheme: webhook-token
    token: ${env:METRICS_WEBHOOK_SECRET}

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: localhost:4317

processors:
  memory_limiter:

  batch:

exporters:
  debug:

  prometheus:
    endpoint: api.pulumi.com:443
    auth:
      authenticator: bearertokenauth
    namespace: pulumi
    resource_to_telemetry_conversion:
      enabled: true

service:
  telemetry:
    logs:

  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [debug]

    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [prometheus]

  extensions: [bearertokenauth]

The bearer token also needs to be included in the Prometheus server configuration:

scrape_configs:
  - job_name: pulumi
    scrape_interval: 15s
    authorization:
      type: webhook-token
      credentials: <METRICS_WEBHOOK_SECRET>
    scheme: https
    static_configs:
      - targets: ["api.pulumi.com"]

Pulumi Deployments

To enable Pulumi Deployments, the following must be configured:

• PULUMI_SERVICE_METADATA_BLOB_STORAGE_ENDPOINT or PULUMI_LOCAL_OBJECTS object storage

• Customer-managed runners - You also need to update the pulumi-workflow-agent.yaml configuration file by setting service_url to <PULUMI_API_DOMAIN>. Example:

  token: pul-d2d2….
  version: v0.0.5
  service_url: https://pulumiapi.acmecorp.com
  

Pulumi ESC

In order to enable Pulumi ESC, the following must be configured:

• PULUMI_SERVICE_METADATA_BLOB_STORAGE_ENDPOINT or PULUMI_LOCAL_OBJECTS object storage

• One of the encryption services

Pulumi IDP Registry

In order to enable the Pulumi IDP Registry, the following must be configured:

• Environment variables for where to store public and private artifacts in a remote object store (see below).
• Cloud provider authentication where the remote object store lives.

AWS S3

Azure Storage

Terraform state backend

To let Terraform and OpenTofu use your self-hosted Pulumi Cloud as a state backend, set the following environment variable on the API service:

Point the DNS record for this domain at the same host that serves PULUMI_API_DOMAIN. A separate domain keeps the Terraform-compatible API distinct from the Pulumi API, but it isn’t required: you can set PULUMI_TFE_DOMAIN to the same value as PULUMI_API_DOMAIN if you don’t want a second DNS record.

In the Terraform or OpenTofu configuration, the terraform login or tofu login command, and the TF_TOKEN_* variable name, use the PULUMI_TFE_DOMAIN value wherever the state backend guide uses tf.pulumi.com.

Operations guide

For production deployment best practices including compute sizing, high-availability configuration, monitoring, backup and recovery, and security hardening, see the Self-hosted operations guide.