Pulumi Cloud self-hosted API

The Pulumi API is one of the components required for self-hosting the Pulumi Cloud in your organization’s environment. It provides the necessary APIs for both the CLI and the Console.

Prerequisites

• Provide a server or virtual machine to install and run the Pulumi components (see Minimum System Requirements below).
• Provide a persistent volume for the service to store checkpoint objects.
• Provider a persistent volume for the MySQL data (optional if you are providing your own DB.)
• If you are providing your own DB instance, ensure that it is accessible within the same Docker network that the service and the UI containers will be running in.
  • The default DB endpoint is pulumi-db:3306. If you wish to change this, set PULUMI_LOCAL_DATABASE_NAME and PULUMI_LOCAL_DATABASE_PORT accordingly (see Script Variables.)
  • If you do not create this network prior to running run-ee.sh, it will create only a bridged network on your local host. Ensure that the DB can be accessed by the API service container.
• Provide an external load balancer with TLS termination.

Minimum System Requirements

Note: The storage recommendations for the Object Storage can be lesser than 200GB depending on your organization size and the expected usage.

What’s In The Container?

The API service is a Go-based application. This is a single binary application that has all of the dependencies it needs in order to run.

Server

This container runs an HTTP server which provides the APIs needed by the Console and the CLI. By default, this container will serve using port 8080 over standard HTTP. If TLS is configured the server will instead serve over port 8443 using HTTPS.

Function Mode

It’s possible to split apart the HTTP Server from the Background Jobs functionality to distribute responsibility across multiple instances. The function mode determines the mode in which the server runs based upon the env variable PULUMI_API_CONTAINER_MODE. If this value is not defined, or doesn’t match a valid option, then it defaults to FULL.

Environment Variables for Core Infrastructure

The core infrastructure components to successfully run the API service are the database, object storage, and encryption services. Depending on your requirements, you can configure additional (optional) identity services as well as enhanced security between the API and the database. The API also supports exporting OpenTelemetry metrics and traces via the OpenTelemetry collector.

Object storage

The service saves the state of every stack, that uses it as the backend, to a persistent object storage. If your org uses policy packs, then anytime a user publishes a policy pack, it gets uploaded to the same object storage service as well. Both the state (checkpoint) and the policy pack must use different buckets on the same object storage service.

Local storage

AWS S3

Azure Storage

Encryption services

The service supports using a master key available in a local-path or in a remote key management service. You only need to configure one of the support services.

Local keys

AWS KMS

Azure KeyVault

Cloud Provider Authentication

These settings are required if you are running the Pulumi Cloud on one of these clouds or using one of their services.

AWS

For more information about authenticating with AWS services, see the AWS SDK docs.

Azure

GitLab OAuth

Only required if using GitLab as the backing identity provider for your organization.

Sending Emails

Other Environment Variables

TLS Environment Variables

API Service

The service is configurable to serve the API using TLS. The following environment variables are all required in order to enable TLS. The values of the environment variables may either be a filepath or the actual value of the entity. If these variables are set, the service will be served over HTTPS (i.e using TLS) using port 8443. If the following variables are not set the service will default to serving over HTTP using port 8080.

Note: Self-signed certificates may be used to configure TLS in the event the need for a trusted entity is not necessary. A self-signed cert and private key may be generated using OpenSSL. The following command uses OpenSSL to generate a self-signed certificate. This example will output two files, the certificate (cert.pem) and the private key (key.pem) used to sign it.

openssl \
  req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem \
  -days { days_unitl_expiration } -nodes -subj "/CN={ common_name }"

Database Connections

API Service

The service is configurable to enable connections to the backend SQL database over TLS. The following environment variables are all required to connect to the database using TLS. If these variables are set the service will establish connections to the database using TLS, otherwise the service will default to connecting without TLS. The same ports will be used for communication to the database regardless of whether TLS is configured or not.

Migrations

The database migrations container is configurable to enable connections to the database over TLS. To use TLS, the following environment variable must be set. The default is to not use TLS.

Audit Logs

Audit Logs are persisted in MySQL by default. Alternative backends can be configured to support a higher volume of writes without scaling out MySQL.

DynamoDB

To use AWS DynamoDB to persist Audit Logs, specify the name of the table in the environment variable PULUMI_AUDIT_LOGS_DYNAMO_TABLE. The table provided must be configured with following attributes:

  hashKey: "org_id",
  rangeKey: "timestamp_id",
  globalSecondaryIndexes: [
      {
          hashKey: "org_user_id",
          projectionType: "ALL",
          name: "org_user",
          rangeKey: "timestamp_id",
      },
  ],
  attributes: [
      {
          name: "org_id",
          type: "S",
      },
      {
          name: "org_user_id",
          type: "S",
      },
      {
          name: "timestamp_id",
          type: "S",
      },
  ],

OpenTelemetry

The API service is configured to export OpenTelemetry metrics and traces to the vendor of your choice via the OpenTelemetry collector. You will need to manage your own OpenTelemetry collector.

The following environment variables are needed to configure OpenTelemetry in the service:

OpenTelemetry is not yet available for the console service.

Metrics endpoint

The API service exposes a metrics endpoint (https://api.pulumi.com/metrics) that is secured by a bearer token. This token is configured using the environment variable METRICS_WEBHOOK_SECRET.

curl -s GET https://api.pulumi.com/metrics -H 'Authorization: webhook-token <METRICS_WEBHOOK_SECRET>

Prometheus

The API service provides two options to get metrics into Prometheus:

1. From the OpenTelemetry collector via a Prometheus remote write exporter. This is a push-based exporter.
2. From a Prometheus exporter by scraping the /metrics endpoint. This is a pull-based exporter.

Prometheus remote write exporter

This option does not use the /metrics endpoint. Instead, it exports metrics from the collector to a Prometheus remote write compatible backend.

Example OpenTelemetry collector configuration for a service using AWS and the AWS Distro for OpenTelemetry Collector:

extensions:
  sigv4auth:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: localhost:4317

processors:
  memory_limiter:

  batch:

exporters:
  logging:

  prometheusremotewrite:
    endpoint: https://aws-managed-prometheus-endpoint/v1/api/remote_write
    auth:
      authenticator: sigv4auth

service:
  telemetry:
    logs:

  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [logging]

    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [prometheusremotewrite]

  extensions: [sigv4auth]

Prometheus exporter

This option requires configuring the environment variable METRICS_WEBHOOK_SECRET to successfully authenticate to the /metrics endpoint.

Example OpenTelemetry collector configuration:

extensions:
  bearertokenauth:
    scheme: webhook-token
    token: ${env:METRICS_WEBHOOK_SECRET}

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: localhost:4317

processors:
  memory_limiter:

  batch:

exporters:
  debug:

  prometheus:
    endpoint: api.pulumi.com:443
    auth:
      authenticator: bearertokenauth
    namespace: pulumi
    resource_to_telemetry_conversion:
      enabled: true

service:
  telemetry:
    logs:

  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [debug]

    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [prometheus]

  extensions: [bearertokenauth]

The bearer token also needs to be included in the Prometheus server configuration:

scrape_configs:
  - job_name: pulumi
    scrape_interval: 15s
    authorization:
      type: webhook-token
      credentials: <METRICS_WEBHOOK_SECRET>
    scheme: https
    static_configs:
      - targets: ["api.pulumi.com"]

Pulumi Deployments

In order to enable Pulumi Deployments, the following must be configured:

• PULUMI_SERVICE_METADATA_BLOB_STORAGE_ENDPOINT or PULUMI_LOCAL_OBJECTS object storage

• Customer-Managed Agents - You also need to update the pulumi-deployment-agent.yaml configuration file by setting service_url to <PULUMI_API_DOMAIN>. Example:

  token: pul-d2d2….
  version: v0.0.5
  service_url: https://pulumiapi.acmecorp.com