Adding PostgreSQL State Backend Support to Pulumi: A Community Contribution Journey

Posted on Thursday, Jun 19, 2025

When managing infrastructure as code at scale, reliable state storage is essential. Pulumi Cloud provides a fully managed, secure, and scalable solution out of the box. For teams that choose to build and maintain their own backend, Pulumi now offers support for PostgreSQL as a DIY state storage option—though this requires additional operational overhead and careful consideration around performance, security, and maintenance.

The Need for Database-Backed State Storage

Traditional DIY backends in Pulumi have relied on object storage systems like AWS S3, Google Cloud Storage, or Azure Blob Storage. While these work well for many use cases, they have limitations when it comes to handling very large state files, complex locking mechanisms, and transactional guarantees that some enterprise environments require.

PostgreSQL stood out as an excellent candidate for state storage due to its:

Large object support: Ability to handle substantial state files without size constraints
ACID compliance: Robust transactional guarantees for state consistency
Mature ecosystem: Well-established tooling and operational practices
Scalability options: From single instances to complex replication setups
Security features: Comprehensive authentication and authorization capabilities

The Community Contribution Process

This feature came to life through PR #19581, which addressed a long-standing community request tracked in issue #5632. The development process showcased the collaborative nature of open-source development, with multiple rounds of feedback, testing, and refinement.

Key Technical Challenges

The implementation required several technical considerations:

1. Integration with Go Cloud Development Kit (CDK)

The PostgreSQL backend needed to integrate seamlessly with Pulumi’s existing blob storage abstraction layer. This was achieved by implementing the blob.BucketURLOpener interface:

// URLHandler is a URL opener for PostgreSQL URLs.
type URLHandler struct{}

// OpenBucketURL implements blob.BucketURLOpener.
func (p URLHandler) OpenBucketURL(ctx context.Context, u *url.URL) (*blob.Bucket, error) {
    pg, err := NewPostgresBucket(ctx, u)
    if err != nil {
        return nil, err
    }
    return pg.Bucket(), nil
}

2. Database Schema Design

The implementation uses a simple but effective schema for storing state data:

CREATE TABLE IF NOT EXISTS pulumi_state (
    key TEXT PRIMARY KEY,
    data JSONB NOT NULL,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);

This design allows for efficient key-based lookups while leveraging PostgreSQL’s native JSON support for flexible state data storage.

3. Cross-Platform Testing

One of the significant challenges was ensuring the implementation worked across different operating systems and CI environments. The solution involved using testcontainers-go to spin up PostgreSQL instances during testing, with appropriate skip conditions for environments where Docker isn’t available:

func skipIfDockerNotAvailable(t *testing.T) {
    if runtime.GOOS == "windows" || runtime.GOOS == "darwin" {
        t.Skip("Skipping test: Docker not available on this platform in CI")
    }
    
    // Additional Docker availability checks...
}

Performance and Limitations

PostgreSQL wins over a plain S3 bucket

ACID transactions with row-level locking – A postgres backend means strong consistency guarantees.
Faster for smaller state files – Community benchmarks show improvements on S3 backend speed for smaller state files.
Point-in-time restore – WAL-based backups allow for easier rollback.
Deep observability – plug pg-metrics into Grafana/APM for real-time insight.

Gaps you’ll still close only with Pulumi Cloud

Server-side policy guardrails (CrossGuard) – automatically enforced policies on every preview or deploy.
Always-on drift detection – automatic scans catch out-of-band changes.
Managed Deployments – fully managed runners, PR previews, and dependent stack updates.
Org-wide RBAC, SSO/SAML, immutable audit logs – turnkey for teams and auditors.
Search & analytics across all cloud resources (Insights) – not just those managed by Pulumi.
Enterprise Secrets & Configuration (ESC) - encrypted secrets and config with RBAC, versioning and audit trail.
SOC 2, high availability, and painless upgrades – Pulumi operates the control plane so you don’t have to.

Getting Started

Using the PostgreSQL backend is straightforward:

Login to your PostgreSQL backend:

pulumi login postgres://pulumi_user:secure_password@localhost:5432/pulumi_state?sslmode=require

Use Pulumi normally:

pulumi up
# Your state is now stored in PostgreSQL!

Community Impact

This contribution demonstrates the power of community-driven development in open source projects. It addresses a real need expressed by Pulumi users while maintaining the high quality standards expected from the project.

The implementation follows Pulumi’s architectural patterns and coding standards, making it a seamless addition to the existing codebase. The comprehensive documentation and testing ensure that future maintainers can easily understand and modify the code.

What’s Next?

The PostgreSQL backend opens up several possibilities for future enhancements:

High Availability: Support for PostgreSQL clustering and replication
Performance Optimizations: Caching strategies and connection pooling
Advanced Features: Custom backup strategies and state analytics
Multi-tenant Support: Isolation patterns for multiple teams or environments

Conclusion

The addition of PostgreSQL state backend support to Pulumi represents more than just a new feature—it’s a testament to the collaborative nature of open source development and the power of community contributions. By providing enterprise-grade state storage options while maintaining the simplicity that makes Pulumi great, this feature enables organizations to adopt infrastructure as code with confidence.

Whether you’re running a small startup or managing infrastructure for a large enterprise, having robust, reliable state storage options is crucial. The PostgreSQL backend provides an improved approach to Do-It-Yourself state management.

If you’re interested in contributing to Pulumi or have ideas for new features, the community welcomes your contributions. This PR serves as an excellent example of how community members can make meaningful improvements to the project while learning from experienced maintainers and following established best practices.