Pulumi Blog: Operators

Pulumi Kubernetes Operator v2.3.0: Preview Mode and Structured Configuration

Eron Wright — Tue, 25 Nov 2025 00:00:00 +0000

We’re excited to announce the release of Pulumi Kubernetes Operator v2.3.0, introducing two powerful capabilities that enhance GitOps workflows: preview mode for validating infrastructure changes before deployment, and structured configuration support for managing complex data types. Building on the success of the v2.0 GA release, this update addresses long-standing community requests while maintaining full backwards compatibility. These features enable safer, more sophisticated infrastructure management patterns for platform engineering teams.

Preview mode: Validate infrastructure changes before deployment

Preview mode enables you to run Pulumi stacks in dry-run fashion, allowing you to visualize what infrastructure changes would occur without actually applying them. This capability is essential for GitOps workflows that require validation gates and incremental rollouts.

By adding preview: true to your Stack specification, the operator runs pulumi preview instead of pulumi up. The Stack’s Ready condition indicates preview success, and you get full status including preview links, standard output, and program outputs—all without making actual infrastructure changes.

This unlocks sophisticated workflow patterns:

Multiple Stacks for what-if analysis: Create several Stack resources pointing to the same Pulumi stack, with all-but-one operating in preview mode to compare different configurations
Branch validation: Preview changes from feature branches before merging to production
Tick-tock rollout patterns: Toggle the preview flag on and off with external verification steps between each deployment phase

Here’s a simple example showing a production stack alongside its preview counterpart:

# Production stack
apiVersion: pulumi.com/v1
kind: Stack
metadata:
 name: my-infrastructure
spec:
 stack: org/project/prod
 projectRepo: https://github.com/example/infra
 branch: main
---
# Preview stack - same Pulumi stack, preview mode
apiVersion: pulumi.com/v1
kind: Stack
metadata:
 name: my-infrastructure-preview
spec:
 stack: org/project/prod
 projectRepo: https://github.com/example/infra
 branch: feature-branch
 preview: true # Only runs pulumi preview

Preview mode closes issue #16, one of our longest-standing feature requests.

Structured configuration: Native support for complex data types

Configuration management takes a significant step forward with native support for complex data types. Previously limited to string values, Stack configuration now supports objects, arrays, numbers, and booleans as first-class citizens.

This enhancement addresses the reality that complex environments require rich configuration. You can now express sophisticated configuration structures inline in your Stack manifests or load them from ConfigMaps with automatic JSON parsing—all using Kubernetes-native patterns.

The implementation leverages Pulumi CLI’s JSON configuration support (available in v3.202.0+) with automatic version detection. If your workspace uses an earlier CLI version, you’ll receive clear guidance to upgrade. Existing string-only configurations continue to work without modification, ensuring full backwards compatibility.

Here’s an example of inline structured configuration:

apiVersion: pulumi.com/v1
kind: Stack
metadata:
 name: my-app
spec:
 stack: org/app/prod
 projectRepo: https://github.com/example/app
 branch: main
 config:
 # String values (existing behavior)
 environment: "production"

 # Objects (NEW)
 database:
 host: "db.example.com"
 port: 5432
 ssl: true

 # Arrays (NEW)
 regions: ["us-west-2", "us-east-1", "eu-west-1"]

 # Numbers and booleans (NEW)
 maxConnections: 100
 enableCaching: true

You can also reference ConfigMaps for more complex scenarios:

apiVersion: v1
kind: ConfigMap
metadata:
 name: app-settings
data:
 database.json: |
 {
 "host": "db.example.com",
 "port": 5432,
 "maxConnections": 100
 }
---
apiVersion: pulumi.com/v1
kind: Stack
metadata:
 name: my-app
spec:
 stack: org/app/prod
 projectRepo: https://github.com/example/app
 branch: main
 configRefs:
 database:
 name: app-settings
 key: database.json
 json: true # Parse as JSON

Note that Secrets are not a supported source of structured configuration values.

Structured configuration closes issue #258 and issue #872, addressing long-standing configuration management needs from the community.

Bug fixes and reliability improvements

This release includes several fixes that improve operational reliability:

Stack name validation: Added validation to limit Stack names to 42 characters, preventing resource naming conflicts (#899)
secretsProvider fix: The secretsProvider parameter now properly applies when creating new stacks (#935)
Helm chart fix: Resolved YAML parsing errors for podLabels containing special characters like colons (#1014)
Stack deletion: Stack deletion is no longer blocked when prerequisite stacks are missing, enabling proper cleanup workflows (#751)
Update TTL: Completed Update objects now respect the ttlAfterCompleted setting for automatic garbage collection (#960)

Upgrading to v2.3.0

This release includes updates to the Custom Resource Definitions (CRDs) to support the new features. If you’re using Helm, you’ll need to manually apply the updated CRDs before upgrading, as Helm v3 does not automatically upgrade CRDs:

# Apply updated CRDs
kubectl apply --server-side -k 'github.com/pulumi/pulumi-kubernetes-operator//operator/config/crd?ref=v2.3.0'

# Upgrade via Helm
helm upgrade pulumi-kubernetes-operator \
 oci://ghcr.io/pulumi/helm-charts/pulumi-kubernetes-operator \
 --version 2.3.0 \
 -n pulumi-kubernetes-operator

If you’re using the quickstart YAML installation method, the CRDs will update automatically when you apply the new manifest.

All changes are backwards compatible—existing Stack resources work without modification. For structured configuration support, ensure your workspace pods use Pulumi CLI v3.202.0 or later; the operator provides automatic version detection with clear upgrade guidance when needed.

Get started today

The v2.3.0 release enhances the Pulumi Kubernetes Operator with safer deployment workflows and more flexible configuration management. Preview mode enables validation-first GitOps patterns, while structured configuration simplifies complex multi-environment setups.

Get started with the Pulumi Kubernetes Operator in our documentation, or view the complete v2.3.0 release notes on GitHub. Join the conversation in the Pulumi Community Slack #kubernetes channel—we’d love to hear how these features impact your workflows.

Why Every Platform Engineer Should Care About Kubernetes Operators

Engin Diri — Tue, 10 Dec 2024 09:31:38 +0100

In one of my recent talks, I mentioned that the foundation of a successful Kubernetes-powered platform is the use of Kubernetes Operators, as they are a great way to automate operational tasks and the lifecycle of complex applications and services on Kubernetes.

In case you missed the talk, here is the link to the recording:

After the talk, I received feedback from the audience that they would like to have more information about Kubernetes operators and some of the advanced operators I would recommend to use in their Kubernetes-powered platforms. And who am I to reject such a great opportunity to share my knowledge with the community?

As you guess, the base for a Kubernetes Operator is, well, Kubernetes. So, if you are not familiar with Kubernetes, let me give you a brief introduction. Kubernetes, or K8s in short, is by far the most powerful open-source container orchestration platform out there - especially for its capability to maintain a desired state. This means you, as the platform admin, can define how you want your cluster and the workload on top of it to look, and Kubernetes (aka the control plane) executes the necessary steps to ensure the desired state is achieved using a feedback loop.

And how does Kubernetes achieve this complex operational task management? Kubernetes leverages the concept of controllers and operators. Both ensure that the cluster resources are in the desired state. But why is there a distinction between controllers and operators inside the Kubernetes ecosystem? There is also great confusion when folks talk about controllers and operators, as the differences are in the details and operators are a subcategory of controllers.

Kubernetes Controllers vs. Operators

K8s controllers are control loops that watch your resources and reconcile the current state with the desired state in a continuous loop. Think about how a thermostat works in your house. The thermostat watches the current temperature and, if it is below the desired temperature, it turns on the heating. If it is above the desired temperature, it turns off the heating. This is how a controller works in Kubernetes - it watches the resources and conditions defined in the desired state and executes the necessary steps to make sure the current state matches the desired state.

Here are some typical Kubernetes controller key functions as part of the maintain, observe, and enforce loop of the control loop:

Function	Description
Rollouts and Rollbacks	Deployment controller takes care of rolling out new versions of your application and rolling back to the previous version in case of failure.
Cluster maintenance	Node controller watches the nodes in the cluster and makes sure that the nodes are in the desired state.
Event handling	Event controller watches for events and takes action based on the events.
Schedule	Controller to run jobs and cron jobs at a specific time.
Resource enforcement	Resource controller watches the resource usage and enforces resource limits in a namespace.
State enforcement	Different controllers watch the state of the resources and enforce the desired state.

As you can see, controllers automate the routine operational tasks, enforce policies, handle failures and take actions to continuously maintain the desired state of the cluster.

Some of the typical use cases for controllers are:

Limit resource usage - Prevent the overload of the cluster by enforcing resource limits with controllers like ResourceQuota.
Auto-scaling - Automatically scales the number of pods based on the different metrics with HorizontalPodAutoscaler.
Task scheduling - Run jobs and cron jobs at a specific time with Job and CronJob controllers.
Run stateful workloads - Deploy and manage stateful workloads like databases with StatefulSet controller, by ensuring that the pods are created in a specific order and have a stable network identity.
Scaling deployments - Deploy and manage stateless workloads like web applications with the Deployment controller, by ensuring that the desired number of pods is running and handling the rolling updates and rollbacks.

Controllers, as you see, reduce the manual effort for the platform engineering teams and gives the cluster self-managing capabilities.

With all the above points, the benefits of controllers are clear - Let me list some of the main ones:

Automation - Controllers automate the routine operational tasks and enforce policies.
Availability - Controllers ensure that the resources are always available, addressing failures and taking actions to maintain the desired state.
Efficiency - Controllers provide an efficient way to manage the resources and workloads in the cluster with quota management and auto-scaling.
Reliability - Controllers help to make workload management more reliable with features like replica management, pod creation and deletion, and scheduling.
Flexibility - Controllers provide a flexible way to handle a variety of workloads and resources in the cluster, such as DaemonSet, StatefulSet, jobs, cron jobs, deployments, and services.
Observability - Controllers provide a view into the cluster resources and the state of the resources via the Kubernetes API.

With Controllers explained, let’s move on to the next level of automation in Kubernetes and talk about Kubernetes Operators.

Kubernetes Operators

As mentioned before, Kubernetes Operators are a subcategory of controllers that use API extensions (Custom Resource) to complete the automation task. Operators are a set of independent controllers, with each controller responsible for its own task. And while controllers can share similar functions with a controller, it is only focused on one domain and only uses Custom Resources to manage the domain.

Keep in mind that controllers work without the need for custom resources or a link to a specific domain.

Operators are custom-built controllers that focus on the deployment, management, and operation of a specific application or service. Some key functions of operators include:

Function	Description
Security	Operators integrate security best practices and policies, such as encryption, access control, and secret management, to protect the application. You can customize these measures and adjust them to the specific needs of the application.
Workload Lifecycle management	Operators manages the full lifecycle of complex stateful applications through deployment, upgrades, and scaling. It even applies logic to automate operations like backup and restore.
Error Handling	Operators detects and handles errors, such as application crashes, network failures, and resource constraints.
Migration	Operators help to migrate the application to a new version, a new cluster, or a new cloud provider.
Disaster Recovery	Operators help to recover the application from a disaster, such as data loss, network failure, or hardware failure.

Operators are a great way to encode the operational knowledge of the platform engineering team into automated controllers. This way, the platform engineering team can focus on the strategic tasks and let the operators handle the boring routine operational tasks. This reduces the manual effort and increases the reliability of the platform.

Some of the typical use cases for operators are:

Database management - Deploy and manage databases such as MySQL, PostgreSQL, and MongoDB.
Storage management - Deploy and manage storage such as Ceph, GlusterFS, or NFS.
Logging and monitoring - Simplify the deployment and management of logging and monitoring solutions like Prometheus, Grafana, or ELK stack.
CI/CD - Automate the deployment and management of CI/CD pipelines like Jenkins, GitLab, or Tekton. - Backup and restore - Automate the backup and restore of applications and data.
Messaging and event streaming - Simplify the deployment and management of messaging and event streaming solutions like Kafka, RabbitMQ, or NATS.
Machine learning - Automate the deployment and management of machine learning workloads such as TensorFlow, PyTorch, or Kubeflow.

With the applications-centric approach provided by operators, the platform engineering team can now enable fully automated operations while enhancing the reliability, efficiency, and observability of the platform and ensuring adherence to best practices and compliance requirements.

And similar to controllers, operators also have distinct benefits:

Simplification - Operators simplify the deployment and management of complex applications and services by providing a declarative way (via Custom Resources).
Higher Productivity - Operators increase the productivity of the platform engineering team by automating the routine operational tasks and allowing users to focus on delivering business value.
Extensibility - Extending the K8s API with Custom Resource Definitions (CRDs) allows administrators to support new applications and services without changing the core Kubernetes code.
Consistency - Operators ensure that they deploy and run the applications and services consistently across different environments, such as on-premises, cloud, or hybrid.
Modularity - Operators are modular and focused on a specific domain.
Legacy Transformation - Operators help to transform legacy applications into cloud-native applications by automating the deployment and management of complex applications.

Here is a visual representation of a Kubernetes Operator in action, which I found very nice and helpful:

Kubernetes Operators: what are they? Some examples Credit: SparkFabrik

Comparison of Kubernetes Controllers and Operators

Here is a quick summary of the differences between Kubernetes controllers and operators:

Feature	Kubernetes Controllers	Kubernetes Operators
Definition	Core control loops within Kubernetes that reconcile the current state with the desired state of resources.	Custom-built controllers with domain-specific logic, typically extending Kubernetes API with Custom Resource Definitions (CRDs).
Scope	General-purpose automation for cluster-wide resource management and workload operations.	Focused on application or service-specific lifecycle management and operations.
Custom Resources (CRDs)	Not mandatory. Most controllers work with built-in Kubernetes resources like Pods, Deployments, and StatefulSets.	Mandatory. Operators depend on CRDs to define and manage application-specific resources.
Domain-Specific Logic	Limited. Primarily focuses on generic Kubernetes operations like scheduling, scaling, and resource enforcement.	Extensive. Includes application-specific tasks such as backups, upgrades, migrations, and disaster recovery.
Automation	Automates operational tasks like resource scheduling, scaling, and maintaining cluster states.	Automates both operational and domain-specific tasks, often encoding deep operational knowledge of specific applications.
Complexity	Relatively low complexity as they focus on Kubernetes-native workflows.	Higher complexity due to domain-specific implementations and integrations with external systems.
Observability	Provides insight into cluster resource states via Kubernetes API and built-in events.	Enhances observability with application-specific metrics, logs, and health checks tailored to the managed application.
Use Cases	- Pod scaling (e.g., HPA) - Resource enforcement (e.g., ResourceQuota) - Rolling updates and rollbacks - Scheduling jobs (e.g., CronJob).	- Database management (e.g., MySQL Operator) - CI/CD pipelines (e.g., Tekton) - Backup and restore - Monitoring solutions (e.g., Prometheus Operator) - Stateful applications (e.g., Kafka Operator).
Deployment	Part of Kubernetes core components or extensions like kube-controller-manager.	External tools developed by vendors, community, or in-house, deployed using Custom Resources.
Key Benefits	- Cluster-wide automation - Ensures resource availability - Efficient and reliable workload management.	- Simplifies application lifecycle management - Increases productivity with domain-specific automation - Consistent deployments across environments.
Flexibility	Flexible for general Kubernetes operations but limited to built-in resources.	Highly flexible and extensible for custom applications and domain-specific requirements.
Target Audience	Platform administrators and Kubernetes users looking to automate Kubernetes-native workflows.	Developers and platform engineers needing advanced automation and lifecycle management for specific applications or services.
Maintenance	Managed and updated by Kubernetes core or community projects.	Requires domain-specific knowledge and regular updates to accommodate application changes and best practices.

With this detailed explanation of Kubernetes controllers and operators, I would like to share some advanced Kubernetes operators that I recommend for use in your Kubernetes-powered platform.

Advanced Kubernetes Operators

As we have seen, Kubernetes Operators are the go-to way to automate operational tasks and lifecycle management. I should mention that not all operators are created equal. Some operators are more advanced and provide more features than others. That is why, for example, OperatorHub.io categorises operators into five capability levels:

Basic Install - Installs the application on the cluster.
Seamless Upgrades - Manages Application upgrades.
Full Lifecycle Management - Manages the full lifecycle of the application.
Deep Insights - Provides monitoring and metrics.
Auto Pilot - Automatically optimizes and tunes the application.

The more advanced the operator, the more features it provides and the more automation it offers. At its most advanced level, the Auto Pilot, the operator automatically optimizes and tunes the application based on the workload and environment.

Here is a list of advanced Kubernetes operators that I recommend using in your Kubernetes-powered platform:

CloudNativePG Operator

CloudNativePG Operator Credit: cloudnative-pg.io

CloudNativePG Operator is a PostgreSQL operator that covers the full lifecycle of PostgreSQL databases on Kubernetes. It simplifies the deployment, management, and scaling of PostgreSQL databases by providing a declarative way to define and manage PostgreSQL clusters using custom resources.

Some of the key features of CloudNativePG Operator are:

Automated Backups - Schedule and manage backups of PostgreSQL databases.
High Availability - Deploy and manage high-availability PostgreSQL clusters.
Scaling - Scale the number of PostgreSQL instances based on the workload.

Installation

Either use the Helm chart or the Kubernetes manifests to deploy the CloudNativePG Operator:

helm repo add cloudnative-pg https://cloudnative-pg.io/charts/
helm repo update
helm upgrade --install cloudnativepg cloudnative-pg/cloudnative-pg --namespace cnpg-system --create-namespace

Or use Pulumi to deploy the CloudNativePG Operator:

import * as k8s from "@pulumi/kubernetes";
import * as k8shelm from "@pulumi/kubernetes/helm/v3";

const ns = new k8s.core.v1.Namespace("cnpg-system", {
 metadata: {
 name: "cnpg-system",
 },
});

new k8shelm.Release("cloudnativepg", {
 chart: "oci://ghcr.io/cloudnative-pg/charts/cloudnative-pg",
 namespace: ns.metadata.name,
 createNamespace: true,
});

import pulumi_kubernetes as k8s
import pulumi_kubernetes.helm.v3 as k8shelm
import pulumi_kubernetes.meta.v1 as meta

ns = k8s.core.v1.Namespace(
 "cnpg-system",
 metadata=meta.ObjectMetaArgs(
 name="cnpg-system",
 ),
)

release = k8shelm.Release(
 "cloudnativepg",
 chart="oci://ghcr.io/cloudnative-pg/charts/cloudnative-pg",
 namespace=ns.metadata["name"],
 create_namespace=True,
)

package main

import (
 k8s "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes"
 apiv1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/apiextensions"
 corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/core/v1"
 helmv3 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/helm/v3"
 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/meta/v1"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {

 ns, err := corev1.NewNamespace(ctx, "cnpg-system", &corev1.NamespaceArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("cnpg-system"),
 },
 })
 if err != nil {
 return err
 }

 _, err = helmv3.NewRelease(ctx, "cloudnativepg", &helmv3.ReleaseArgs{
 Chart: pulumi.String("oci://ghcr.io/cloudnative-pg/charts/cloudnative-pg"),
 Namespace: ns.Metadata.Name().Elem(),
 CreateNamespace: pulumi.Bool(true),
 })
 if err != nil {
 return err
 }
 return nil
 })
}

using System.Threading.Tasks;
using Pulumi;
using Pulumi.Kubernetes.Core.V1;
using Pulumi.Kubernetes.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Core.V1;
using Pulumi.Kubernetes.Types.Inputs.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Meta.V1;
using System.Collections.Generic;
using Pulumi.Kubernetes.ApiExtensions;

class MyStack : Stack
{
 public MyStack()
 {
 var ns = new Namespace("cnpg-system", new NamespaceArgs
 {
 Metadata = new ObjectMetaArgs
 {
 Name = "cnpg-system"
 }
 });

 var release = new Release("cloudnativepg", new ReleaseArgs
 {
 Chart = "oci://ghcr.io/cloudnative-pg/charts/cloudnative-pg",
 Namespace = ns.Metadata.Apply(m => m.Name),
 CreateNamespace = true
 });

}

class Program
{
 static Task<int> Main() => Deployment.RunAsync<MyStack>();
}

Usage

Here is a simple example of a PostgreSQL cluster with backup configured on S3 API-compatible storage:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
 name: cluster-example-with-backup
spec:
 instances: 3
 primaryUpdateStrategy: unsupervised

 # Persistent storage configuration
 storage:
 storageClass: standard
 size: 1Gi

 # Backup properties
 # This assumes a local minio setup
 backup:
 barmanObjectStore:
 destinationPath: s3://backups/
 endpointURL: http://minio:9000
 s3Credentials:
 accessKeyId:
 name: minio
 key: ACCESS_KEY_ID
 secretAccessKey:
 name: minio
 key: ACCESS_SECRET_KEY
 wal:
 compression: gzip

Or use Pulumi:

new k8s.apiextensions.CustomResource("cluster-example-with-backup", {
 apiVersion: "postgresql.cnpg.io/v1",
 kind: "Cluster",
 metadata: {
 name: "cluster-example-with-backup",
 },
 spec: {
 instances: 3,
 primaryUpdateStrategy: "unsupervised",

 // Persistent storage configuration
 storage: {
 storageClass: "standard",
 size: "1Gi",
 },

 // Backup properties
 backup: {
 barmanObjectStore: {
 destinationPath: "s3://backups/",
 endpointURL: "http://minio:9000",
 s3Credentials: {
 accessKeyId: {
 name: "minio",
 key: "ACCESS_KEY_ID",
 },
 secretAccessKey: {
 name: "minio",
 key: "ACCESS_SECRET_KEY",
 },
 },
 wal: {
 compression: "gzip",
 },
 },
 },
 },
});

from pulumi_kubernetes import apiextensions

postgresql_cluster = apiextensions.CustomResource(
 "cluster-example-with-backup",
 api_version="postgresql.cnpg.io/v1",
 kind="Cluster",
 metadata=meta.ObjectMetaArgs(
 name="cluster-example-with-backup",
 ),
 spec={
 "instances": 3,
 "primaryUpdateStrategy": "unsupervised",
 # Persistent storage configuration
 "storage": {
 "storageClass": "standard",
 "size": "1Gi",
 },
 # Backup properties
 "backup": {
 "barmanObjectStore": {
 "destinationPath": "s3://backups/",
 "endpointURL": "http://minio:9000",
 "s3Credentials": {
 "accessKeyId": {
 "name": "minio",
 "key": "ACCESS_KEY_ID",
 },
 "secretAccessKey": {
 "name": "minio",
 "key": "ACCESS_SECRET_KEY",
 },
 },
 "wal": {
 "compression": "gzip",
 },
 },
 },
 },
)

package main

import (
 k8s "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes"
 apiv1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/apiextensions"
 corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/core/v1"
 helmv3 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/helm/v3"
 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/meta/v1"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {

 _, err = apiv1.NewCustomResource(ctx, "cluster-example-with-backup", &apiv1.CustomResourceArgs{
 ApiVersion: pulumi.String("postgresql.cnpg.io/v1"),
 Kind: pulumi.String("Cluster"),
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("cluster-example-with-backup"),
 },
 OtherFields: k8s.UntypedArgs{
 "instances": pulumi.Int(3),
 "primaryUpdateStrategy": pulumi.String("unsupervised"),
 "storage": pulumi.Map{
 "storageClass": pulumi.String("standard"),
 "size": pulumi.String("1Gi"),
 },
 "backup": pulumi.Map{
 "barmanObjectStore": pulumi.Map{
 "destinationPath": pulumi.String("s3://backups/"),
 "endpointURL": pulumi.String("http://minio:9000"),
 "s3Credentials": pulumi.Map{
 "accessKeyId": pulumi.Map{
 "name": pulumi.String("minio"),
 "key": pulumi.String("ACCESS_KEY_ID"),
 },
 "secretAccessKey": pulumi.Map{
 "name": pulumi.String("minio"),
 "key": pulumi.String("ACCESS_SECRET_KEY"),
 },
 },
 "wal": pulumi.Map{
 "compression": pulumi.String("gzip"),
 },
 },
 },
 },
 })
 if err != nil {
 return err
 }

 return nil
 })
}

using System.Threading.Tasks;
using Pulumi;
using Pulumi.Kubernetes.Core.V1;
using Pulumi.Kubernetes.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Core.V1;
using Pulumi.Kubernetes.Types.Inputs.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Meta.V1;
using System.Collections.Generic;
using Pulumi.Kubernetes.ApiExtensions;

class PostGreSQLClusterArgs : CustomResourceArgs
{
 [Input("postgresql.cnpg.io/v1")]
 public Input<string> ApiVersion { get; set; }
 [Input("Cluster")]
 public Input<string> Kind { get; set; }
 public ObjectMetaArgs Metadata { get; set; }
 public Dictionary<string, object> Others { get; set; }
 [Input("spec")]
 public InputMap<object> Spec { get; set; }

 public PostGreSQLClusterArgs(string apiVersion, string kind) : base(apiVersion, kind)
 {
 }
}
class MyStack : Stack
{
 public MyStack()
 {
 var postgresqlCluster = new Pulumi.Kubernetes.ApiExtensions.CustomResource("cluster-example-with-backup", new PostGreSQLClusterArgs("postgresql.cnpg.io/v1", "Cluster")
 {
 ApiVersion = "postgresql.cnpg.io/v1",
 Kind = "Cluster",
 Metadata = new ObjectMetaArgs
 {
 Name = "cluster-example-with-backup",
 },
 Spec = new InputMap<object>
 {
 { "instances", 3 },
 { "primaryUpdateStrategy", "unsupervised" },
 { "storage", new InputMap<object>
 {
 { "storageClass", "standard" },
 { "size", "1Gi" },
 }
 },
 { "backup", new InputMap<object>
 {
 { "barmanObjectStore", new InputMap<object>
 {
 { "destinationPath", "s3://backups/" },
 { "endpointURL", "http://minio:9000" },
 { "s3Credentials", new InputMap<object>
 {
 { "accessKeyId", new InputMap<object>
 {
 { "name", "minio" },
 { "key", "ACCESS_KEY_ID" },
 }
 },
 { "secretAccessKey", new InputMap<object>
 {
 { "name", "minio" },
 { "key", "ACCESS_SECRET_KEY" },
 }
 },
 }
 },
 { "wal", new InputMap<object>
 {
 { "compression", "gzip" },
 }
 },
 }
 },
 }
 },
 }
 });

}

class Program
{
 static Task<int> Main() => Deployment.RunAsync<MyStack>();
}

Summary

CloudNativePG Operator is a great way to deploy and manage PostgreSQL databases on Kubernetes. It simplifies a wide range of tasks, including backups, high availability, and scaling, making it a must-have for PostgreSQL users.

You might also like:

Flux Operator

Flux Operator Credit: fluxcd.control-plane.io/operator

The Flux Operator is a full autopilot solution that provides an alternative to the traditional Flux Bootstrap procedure, removing the operational burden of managing Flux across multiple clusters by automating the installation, configuration, and upgrade of Flux fully.

Some of the key features of Flux Operator are:

GitOps Deployment - Automate the deployment of applications and services using GitOps.
Lifecycle Management - Manage the full lifecycle of Flux, including upgrades and rollbacks.
Deep Insights - Provide deep insights into the state of Flux and the applications deployed with Flux.

Installation

Use the Helm chart to deploy the Flux Operator:

helm upgrade -i flux-operator oci://ghcr.io/controlplaneio-fluxcd/charts/flux-operator --namespace flux-system --create-namespace

Or use Pulumi to deploy the Flux Operator:

import * as k8s from "@pulumi/kubernetes";
import * as k8shelm from "@pulumi/kubernetes/helm/v3";

const namespace = new k8s.core.v1.Namespace("flux-system", {
 metadata: {
 name: "flux-system",
 },
});

new k8s.helm.v3.Release("flux-operator", {
 chart: "oci://ghcr.io/controlplaneio-fluxcd/charts/flux-operator",
 namespace: namespace.metadata.name,
 createNamespace: true,
});

import pulumi_kubernetes as k8s
import pulumi_kubernetes.helm.v3 as k8shelm
import pulumi_kubernetes.meta.v1 as meta

flux_namespace = k8s.core.v1.Namespace(
 "flux-system",
 metadata=meta.ObjectMetaArgs(
 name="flux-system",
 ),
)
flux_operator = k8shelm.Release(
 "flux-operator",
 chart="oci://ghcr.io/controlplaneio-fluxcd/charts/flux-operator",
 namespace=flux_namespace.metadata["name"],
 create_namespace=True,
)

package main

import (
 k8s "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes"
 apiv1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/apiextensions"
 corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/core/v1"
 helmv3 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/helm/v3"
 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/meta/v1"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {

 namespace, err := corev1.NewNamespace(ctx, "flux-system", &corev1.NamespaceArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("flux-system"),
 },
 })
 if err != nil {
 return err
 }

 _, err = helmv3.NewRelease(ctx, "flux-operator", &helmv3.ReleaseArgs{
 Chart: pulumi.String("oci://ghcr.io/controlplaneio-fluxcd/charts/flux-operator"),
 Namespace: namespace.Metadata.Name().Elem(),
 CreateNamespace: pulumi.Bool(true),
 })
 if err != nil {
 return err
 }
 return nil
 })
}

using System.Threading.Tasks;
using Pulumi;
using Pulumi.Kubernetes.Core.V1;
using Pulumi.Kubernetes.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Core.V1;
using Pulumi.Kubernetes.Types.Inputs.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Meta.V1;
using System.Collections.Generic;
using Pulumi.Kubernetes.ApiExtensions;

 var namespaceFluxSystem = new Namespace("flux-system", new NamespaceArgs
 {
 Metadata = new ObjectMetaArgs
 {
 Name = "flux-system",
 },
 });

 var fluxOperator = new Release("flux-operator", new ReleaseArgs
 {
 Chart = "oci://ghcr.io/controlplaneio-fluxcd/charts/flux-operator",
 Namespace = namespaceFluxSystem.Metadata.Apply(metadata => metadata.Name),
 CreateNamespace = true,
 });
}

class Program
{
 static Task<int> Main() => Deployment.RunAsync<MyStack>();
}

Usage

Here is an example of a FluxInstance Custom Resource that defines a Flux instance:

apiVersion: fluxcd.controlplane.io/v1
kind: FluxInstance
metadata:
 name: flux
 namespace: flux-system
 annotations:
 fluxcd.controlplane.io/reconcileEvery: "1h"
 fluxcd.controlplane.io/reconcileTimeout: "5m"
spec:
 distribution:
 version: "2.x"
 registry: "ghcr.io/fluxcd"
 artifact: "oci://ghcr.io/controlplaneio-fluxcd/flux-operator-manifests"
 components:
 - source-controller
 - kustomize-controller
 - helm-controller
 - notification-controller
 - image-reflector-controller
 - image-automation-controller
 cluster:
 type: kubernetes
 multitenant: false
 networkPolicy: true
 domain: "cluster.local"
 kustomize:
 patches:
 - target:
 kind: Deployment
 name: "(kustomize-controller|helm-controller)"
 patch: |
 - op: add
 path: /spec/template/spec/containers/0/args/-
 value: --concurrent=10
 - op: add
 path: /spec/template/spec/containers/0/args/-
 value: --requeue-dependency=5s

Or use Pulumi:

new k8s.apiextensions.CustomResource("flux", {
 apiVersion: "fluxcd.controlplane.io/v1",
 kind: "FluxInstance",
 metadata: {
 name: "flux",
 namespace: "flux-system",
 annotations: {
 "fluxcd.controlplane.io/reconcileEvery": "1h",
 "fluxcd.controlplane.io/reconcileTimeout": "5m",
 },
 },
 spec: {
 distribution: {
 version: "2.x",
 registry: "ghcr.io/fluxcd",
 artifact: "oci://ghcr.io/controlplaneio-fluxcd/flux-operator-manifests",
 },
 components: ["source-controller", "kustomize-controller", "helm-controller", "notification-controller", "image-reflector-controller", "image-automation-controller"],
 cluster: {
 type: "kubernetes",
 multitenant: false,
 networkPolicy: true,
 domain: "cluster.local",
 },
 kustomize: {
 patches: [
 {
 target: {
 kind: "Deployment",
 name: "(kustomize-controller|helm-controller)",
 },
 patch: `
 - op: add
 path: /spec/template/spec/containers/0/args/-
 value: --concurrent=10
 - op: add
 path: /spec/template/spec/containers/0/args/-
 value: --requeue-dependency=5s
 `,
 },
 ],
 },
 },
});

from pulumi_kubernetes import apiextensions

flux_instance = k8s.apiextensions.CustomResource(
 "flux",
 api_version="fluxcd.controlplane.io/v1",
 kind="FluxInstance",
 metadata={
 "name": "flux",
 "namespace": flux_namespace.metadata["name"],
 "annotations": {
 "fluxcd.controlplane.io/reconcileEvery": "1h",
 "fluxcd.controlplane.io/reconcileTimeout": "5m",
 },
 },
 spec={
 "distribution": {
 "version": "2.x",
 "registry": "ghcr.io/fluxcd",
 "artifact": "oci://ghcr.io/controlplaneio-fluxcd/flux-operator-manifests",
 },
 "components": [
 "source-controller",
 "kustomize-controller",
 "helm-controller",
 "notification-controller",
 "image-reflector-controller",
 "image-automation-controller",
 ],
 "cluster": {
 "type": "kubernetes",
 "multitenant": False,
 "networkPolicy": True,
 "domain": "cluster.local",
 },
 "kustomize": {
 "patches": [
 {
 "target": {
 "kind": "Deployment",
 "name": "(kustomize-controller|helm-controller)",
 },
 "patch": [
 {
 "op": "add",
 "path": "/spec/template/spec/containers/0/args/-",
 "value": "--concurrent=10",
 },
 {
 "op": "add",
 "path": "/spec/template/spec/containers/0/args/-",
 "value": "--requeue-dependency=5s",
 },
 ],
 }
 ]
 },
 },
)

package main

import (
 k8s "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes"
 apiv1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/apiextensions"
 corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/core/v1"
 helmv3 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/helm/v3"
 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/meta/v1"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {

 _, err = apiv1.NewCustomResource(ctx, "flux", &apiv1.CustomResourceArgs{
 ApiVersion: pulumi.String("fluxcd.controlplane.io/v1"),
 Kind: pulumi.String("FluxInstance"),
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("flux"),
 Namespace: pulumi.String("flux-system"),
 Annotations: pulumi.StringMap{
 "fluxcd.controlplane.io/reconcileEvery": pulumi.String("1h"),
 "fluxcd.controlplane.io/reconcileTimeout": pulumi.String("5m"),
 },
 },
 OtherFields: k8s.UntypedArgs{
 "distribution": pulumi.Map{
 "version": pulumi.String("2.x"),
 "registry": pulumi.String("ghcr.io/fluxcd"),
 "artifact": pulumi.String("oci://ghcr.io/controlplaneio-fluxcd/flux-operator-manifests"),
 },
 "components": pulumi.StringArray{
 pulumi.String("source-controller"),
 pulumi.String("kustomize-controller"),
 pulumi.String("helm-controller"),
 pulumi.String("notification-controller"),
 pulumi.String("image-reflector-controller"),
 pulumi.String("image-automation-controller"),
 },
 "cluster": pulumi.Map{
 "type": pulumi.String("kubernetes"),
 "multitenant": pulumi.Bool(false),
 "networkPolicy": pulumi.Bool(true),
 "domain": pulumi.String("cluster.local"),
 },
 "kustomize": pulumi.Map{
 "patches": pulumi.Array{
 pulumi.Map{
 "target": pulumi.Map{
 "kind": pulumi.String("Deployment"),
 "name": pulumi.String("(kustomize-controller|helm-controller)"),
 },
 "patch": pulumi.Array{
 pulumi.Map{
 "op": pulumi.String("add"),
 "path": pulumi.String("/spec/template/spec/containers/0/args/-"),
 "value": pulumi.String("--concurrent=10"),
 },
 pulumi.Map{
 "op": pulumi.String("add"),
 "path": pulumi.String("/spec/template/spec/containers/0/args/-"),
 "value": pulumi.String("--requeue-dependency=5s"),
 },
 },
 },
 },
 },
 },
 })
 if err != nil {
 return err
 }

 return nil
 })
}

using System.Threading.Tasks;
using Pulumi;
using Pulumi.Kubernetes.Core.V1;
using Pulumi.Kubernetes.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Core.V1;
using Pulumi.Kubernetes.Types.Inputs.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Meta.V1;
using System.Collections.Generic;
using Pulumi.Kubernetes.ApiExtensions;

class FluxInstanceArgs : CustomResourceArgs
{
 public string ApiVersion { get; set; }
 public string Kind { get; set; }
 public ObjectMetaArgs Metadata { get; set; }
 public Dictionary<string, object> Spec { get; set; }

 public FluxInstanceArgs(string apiVersion, string kind) : base(apiVersion, kind)
 {
 }
}
 var flux = new Pulumi.Kubernetes.ApiExtensions.CustomResource("flux", new FluxInstanceArgs("fluxcd.controlplane.io/v1", "FluxInstance")
 {
 ApiVersion = "fluxcd.controlplane.io/v1",
 Kind = "FluxInstance",
 Metadata = new ObjectMetaArgs
 {
 Name = "flux",
 Annotations =
 {
 { "fluxcd.controlplane.io/reconcileEvery", "1h" },
 { "fluxcd.controlplane.io/reconcileTimeout", "5m" },
 }
 },
 Spec = new Dictionary<string, object>
 {
 { "distribution", new Dictionary<string, object>
 {
 { "version", "2.x" },
 { "registry", "ghcr.io/fluxcd" },
 { "artifact", "oci://ghcr.io/controlplaneio-fluxcd/flux-operator-manifests" }
 }
 },
 { "components", new[]
 {
 "source-controller",
 "kustomize-controller",
 "helm-controller",
 "notification-controller",
 "image-reflector-controller",
 "image-automation-controller"
 }
 },
 { "cluster", new Dictionary<string, object>
 {
 { "type", "kubernetes" },
 { "multitenant", false },
 { "networkPolicy", true },
 { "domain", "cluster.local" }
 }
 },
 { "kustomize", new Dictionary<string, object>
 {
 { "patches", new[]
 {
 new Dictionary<string, object>
 {
 { "target", new Dictionary<string, object>
 {
 { "kind", "Deployment" },
 { "name", "(kustomize-controller|helm-controller)" }
 }
 },
 { "patch", new[]
 {
 new Dictionary<string, object>
 {
 { "op", "add" },
 { "path", "/spec/template/spec/containers/0/args/-" },
 { "value", "--concurrent=10" }
 },
 new Dictionary<string, object>
 {
 { "op", "add" },
 { "path", "/spec/template/spec/containers/0/args/-" },
 { "value", "--requeue-dependency=5s" }
 }
 }
 }
 }
 }
 }
 }
 }
 },
 });

}

class Program
{
 static Task<int> Main() => Deployment.RunAsync<MyStack>();
}

Summary

The Flux Operator is another excellent example of an advanced Kubernetes operator, providing full lifecycle management with numerous automation features.

Strimzi Operator

Strimzi Operator Credit: strimzi.io

Last but not least, the Strimzi Operator is a full lifecycle management operator for Apache Kafka on Kubernetes. As event-driven architectures are becoming increasingly popular, Apache Kafka has become the de-facto standard for building scalable and reliable event streaming platforms. The Strimzi Operator makes it easier to deploy, manage, and scale Apache Kafka clusters on Kubernetes.

Some of the key features of Strimzi Operator are:

Cluster Management - Deploy and manage Apache Kafka clusters.
Mirror Clusters - Deploy and manage mirror clusters for disaster recovery.
Streaming Capabilities - With Kafka Connect you can stream data between Kafka and other systems.

Installation

Use the Helm chart to deploy the Strimzi Operator:

helm repo add strimzi https://strimzi.io/charts/
helm repo update
helm install strimzi strimzi/strimzi-kafka-operator -n kafka --create-namespace

Or use Pulumi to deploy the Strimzi Operator:

import * as k8s from "@pulumi/kubernetes";
import * as k8shelm from "@pulumi/kubernetes/helm/v3";

const namespaceKafka = new k8s.core.v1.Namespace("kafka", {
 metadata: {
 name: "kafka",
 },
});

new k8s.helm.v3.Release("strimzi-kafka-operator", {
 chart: "strimzi-kafka-operator",
 namespace: namespaceKafka.metadata.name,
 repositoryOpts: {
 repo: "https://strimzi.io/charts/",
 },
 createNamespace: true,
});

import pulumi_kubernetes as k8s
import pulumi_kubernetes.helm.v3 as k8shelm
import pulumi_kubernetes.meta.v1 as meta

kafka_namespace = k8s.core.v1.Namespace(
 "kafka",
 metadata=meta.ObjectMetaArgs(
 name="kafka",
 ),
)

strimzi_kafka_operator = k8shelm.Release(
 "strimzi-kafka-operator",
 chart="strimzi-kafka-operator",
 namespace=kafka_namespace.metadata["name"],
 repository_opts={
 "repo": "https://strimzi.io/charts/",
 },
 create_namespace=True,
)

package main

import (
 k8s "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes"
 apiv1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/apiextensions"
 corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/core/v1"
 helmv3 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/helm/v3"
 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/meta/v1"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 namespaceKafka, err := corev1.NewNamespace(ctx, "kafka", &corev1.NamespaceArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("kafka"),
 },
 })
 if err != nil {
 return err
 }

 _, err = helmv3.NewRelease(ctx, "strimzi-kafka-operator", &helmv3.ReleaseArgs{
 Chart: pulumi.String("strimzi-kafka-operator"),
 Namespace: namespaceKafka.Metadata.Name(),
 RepositoryOpts: helmv3.RepositoryOptsArgs{
 Repo: pulumi.String("https://strimzi.io/charts/"),
 },
 CreateNamespace: pulumi.Bool(true),
 })
 if err != nil {
 return err
 }

 return nil
 })
}

using System.Threading.Tasks;
using Pulumi;
using Pulumi.Kubernetes.Core.V1;
using Pulumi.Kubernetes.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Core.V1;
using Pulumi.Kubernetes.Types.Inputs.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Meta.V1;
using System.Collections.Generic;
using Pulumi.Kubernetes.ApiExtensions;
class MyStack : Stack
{
 public MyStack()
 {
 var namespaceKafka = new Namespace("kafka", new NamespaceArgs
 {
 Metadata = new ObjectMetaArgs
 {
 Name = "kafka",
 }
 });

 var strimziKafkaOperator = new Release("strimzi-kafka-operator", new ReleaseArgs
 {
 Chart = "strimzi-kafka-operator",
 Namespace = namespaceKafka.Metadata.Apply(ns => ns.Name),
 RepositoryOpts = new RepositoryOptsArgs
 {
 Repo = "https://strimzi.io/charts/",
 },
 CreateNamespace = true,
 });

}

class Program
{
 static Task<int> Main() => Deployment.RunAsync<MyStack>();
}

Usage

Here is an example of a Kafka Custom Resource that defines a Kafka cluster:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
 name: example-kafka-node-pool
 labels:
 strimzi.io/cluster: example-kafka-cluster
spec:
 replicas: 1
 roles:
 - controller
 - broker
 storage:
 type: ephemeral
---
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
 name: example-kafka-cluster
 annotations:
 strimzi.io/kraft: enabled
 strimzi.io/node-pools: enabled
spec:
 kafka:
 replicas: 1
 version: 3.8.0
 storage:
 type: ephemeral
 metadataVersion: 3.8-IV0
 listeners:
 - name: plain
 port: 9092
 type: internal
 tls: false
 - name: tls
 port: 9093
 type: internal
 tls: true
 config:
 offsets.topic.replication.factor: 1
 transaction.state.log.replication.factor: 1
 transaction.state.log.min.isr: 1
 default.replication.factor: 1
 min.insync.replicas: 1
 entityOperator:
 topicOperator: {}
 userOperator: {}

This example defines a Kafka cluster with one replica and two listeners, one for plain and one for TLS and uses ephemeral storage.

To create a topic, you can use the following KafkaTopic Custom Resource:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
 name: example-kafka-cluster
 labels:
 strimzi.io/cluster:
spec:
 partitions: 5
 replicas: 1
 config:
 retention.ms: 7200000
 segment.bytes: 1073741824

Or use Pulumi:

new k8s.apiextensions.CustomResource("example-kafka-node-pool", {
 apiVersion: "kafka.strimzi.io/v1beta2",
 kind: "KafkaNodePool",
 metadata: {
 name: "example-kafka-node-pool",
 labels: {
 "strimzi.io/cluster": "example-kafka-cluster",
 },
 namespace: "kafka",
 },
 spec: {
 replicas: 1,
 roles: ["controller", "broker"],
 storage: {
 type: "ephemeral",
 },
 },
});

new k8s.apiextensions.CustomResource("example-kafka-cluster", {
 apiVersion: "kafka.strimzi.io/v1beta2",
 kind: "Kafka",
 metadata: {
 name: "example-kafka-cluster",
 annotations: {
 "strimzi.io/kraft": "enabled",
 "strimzi.io/node-pools": "enabled",
 },
 namespace: "kafka",
 },
 spec: {
 kafka: {
 replicas: 1,
 version: "3.8.0",
 storage: {
 type: "ephemeral",
 },
 metadataVersion: "3.8-IV0",
 listeners: [
 {
 name: "plain",
 port: 9092,
 type: "internal",
 tls: false,
 },
 {
 name: "tls",
 port: 9093,
 type: "internal",
 tls: true,
 },
 ],
 config: {
 "offsets.topic.replication.factor": 1,
 "transaction.state.log.replication.factor": 1,
 "transaction.state.log.min.isr": 1,
 "default.replication.factor": 1,
 "min.insync.replicas": 1,
 },
 },
 entityOperator: {
 topicOperator: {},
 userOperator: {},
 },
 },
});

new k8s.apiextensions.CustomResource("example-kafka-topic", {
 apiVersion: "kafka.strimzi.io/v1beta2",
 kind: "KafkaTopic",
 metadata: {
 name: "example-kafka-topic",
 labels: {
 "strimzi.io/cluster": "example-kafka-cluster",
 },
 namespace: "kafka",
 },
 spec: {
 partitions: 5,
 replicas: 1,
 config: {
 "retention.ms": 7200000,
 "segment.bytes": 1073741824,
 },
 },
});

from pulumi_kubernetes import apiextensions

kafka_node_pool = k8s.apiextensions.CustomResource(
 "example-kafka-node-pool",
 api_version="kafka.strimzi.io/v1beta2",
 kind="KafkaNodePool",
 metadata={
 "name": "example-kafka-node-pool",
 "labels": {
 "strimzi.io/cluster": "example-kafka-cluster",
 },
 "namespace": kafka_namespace.metadata["name"],
 },
 spec={
 "replicas": 1,
 "roles": ["controller", "broker"],
 "storage": {
 "type": "ephemeral",
 },
 },
)

kafka_cluster = k8s.apiextensions.CustomResource(
 "example-kafka-cluster",
 api_version="kafka.strimzi.io/v1beta2",
 kind="Kafka",
 metadata={
 "name": "example-kafka-cluster",
 "annotations": {
 "strimzi.io/kraft": "enabled",
 "strimzi.io/node-pools": "enabled",
 },
 "namespace": kafka_namespace.metadata["name"],
 },
 spec={
 "kafka": {
 "replicas": 1,
 "version": "3.8.0",
 "storage": {
 "type": "ephemeral",
 },
 "metadataVersion": "3.8-IV0",
 "listeners": [
 {
 "name": "plain",
 "port": 9092,
 "type": "internal",
 "tls": False,
 },
 {
 "name": "tls",
 "port": 9093,
 "type": "internal",
 "tls": True,
 },
 ],
 "config": {
 "offsets.topic.replication.factor": 1,
 "transaction.state.log.replication.factor": 1,
 "transaction.state.log.min.isr": 1,
 "default.replication.factor": 1,
 "min.insync.replicas": 1,
 },
 },
 "entityOperator": {
 "topicOperator": {},
 "userOperator": {},
 },
 },
)

kafka_topic = k8s.apiextensions.CustomResource(
 "example-kafka-topic",
 api_version="kafka.strimzi.io/v1beta2",
 kind="KafkaTopic",
 metadata={
 "name": "example-kafka-topic",
 "labels": {
 "strimzi.io/cluster": "example-kafka-cluster",
 },
 "namespace": kafka_namespace.metadata["name"],
 },
 spec={
 "partitions": 5,
 "replicas": 1,
 "config": {
 "retention.ms": 7200000,
 "segment.bytes": 1073741824,
 },
 },
)

package main

import (
 k8s "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes"
 apiv1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/apiextensions"
 corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/core/v1"
 helmv3 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/helm/v3"
 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v4/go/kubernetes/meta/v1"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {

 _, err = apiv1.NewCustomResource(ctx, "example-kafka-node-pool", &apiv1.CustomResourceArgs{
 ApiVersion: pulumi.String("kafka.strimzi.io/v1beta2"),
 Kind: pulumi.String("KafkaNodePool"),
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("example-kafka-node-pool"),
 Namespace: pulumi.String("kafka"),
 Labels: pulumi.StringMap{
 "strimzi.io/cluster": pulumi.String("example-kafka-cluster"),
 },
 },
 OtherFields: k8s.UntypedArgs{
 "replicas": pulumi.Int(1),
 "roles": pulumi.StringArray{
 pulumi.String("controller"),
 pulumi.String("broker"),
 },
 "storage": pulumi.Map{
 "type": pulumi.String("ephemeral"),
 },
 },
 })
 if err != nil {
 return err
 }

 _, err = apiv1.NewCustomResource(ctx, "example-kafka-cluster", &apiv1.CustomResourceArgs{
 ApiVersion: pulumi.String("kafka.strimzi.io/v1beta2"),
 Kind: pulumi.String("Kafka"),
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("example-kafka-cluster"),
 Namespace: pulumi.String("kafka"),
 Annotations: pulumi.StringMap{
 "strimzi.io/kraft": pulumi.String("enabled"),
 "strimzi.io/node-pools": pulumi.String("enabled"),
 },
 },
 OtherFields: k8s.UntypedArgs{
 "kafka": pulumi.Map{
 "replicas": pulumi.Int(1),
 "version": pulumi.String("3.8.0"),
 "storage": pulumi.Map{
 "type": pulumi.String("ephemeral"),
 },
 "metadataVersion": pulumi.String("3.8-IV0"),
 "listeners": pulumi.Array{
 pulumi.Map{
 "name": pulumi.String("plain"),
 "port": pulumi.Int(9092),
 "type": pulumi.String("internal"),
 "tls": pulumi.Bool(false),
 },
 pulumi.Map{
 "name": pulumi.String("tls"),
 "port": pulumi.Int(9093),
 "type": pulumi.String("internal"),
 "tls": pulumi.Bool(true),
 },
 },
 "config": pulumi.Map{
 "offsets.topic.replication.factor": pulumi.Int(1),
 "transaction.state.log.replication.factor": pulumi.Int(1),
 "transaction.state.log.min.isr": pulumi.Int(1),
 "default.replication.factor": pulumi.Int(1),
 "min.insync.replicas": pulumi.Int(1),
 },
 },
 "entityOperator": pulumi.Map{
 "topicOperator": pulumi.Map{},
 "userOperator": pulumi.Map{},
 },
 },
 })
 if err != nil {
 return err
 }

 _, err = apiv1.NewCustomResource(ctx, "example-kafka-topic", &apiv1.CustomResourceArgs{
 ApiVersion: pulumi.String("kafka.strimzi.io/v1beta2"),
 Kind: pulumi.String("KafkaTopic"),
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("example-kafka-topic"),
 Namespace: pulumi.String("kafka"),
 Labels: pulumi.StringMap{
 "strimzi.io/cluster": pulumi.String("example-kafka-cluster"),
 },
 },
 OtherFields: k8s.UntypedArgs{
 "partitions": pulumi.Int(5),
 "replicas": pulumi.Int(1),
 "config": pulumi.Map{
 "retention.ms": pulumi.Int(7200000),
 "segment.bytes": pulumi.Int(1073741824),
 },
 },
 })
 if err != nil {
 return err
 }

 return nil
 })
}

using System.Threading.Tasks;
using Pulumi;
using Pulumi.Kubernetes.Core.V1;
using Pulumi.Kubernetes.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Core.V1;
using Pulumi.Kubernetes.Types.Inputs.Helm.V3;
using Pulumi.Kubernetes.Types.Inputs.Meta.V1;
using System.Collections.Generic;
using Pulumi.Kubernetes.ApiExtensions;

class KafkaNodePoolArgs : CustomResourceArgs
{
 public string ApiVersion { get; set; }
 public string Kind { get; set; }
 public ObjectMetaArgs Metadata { get; set; }
 public Dictionary<string, object> Spec { get; set; }

 public KafkaNodePoolArgs(string apiVersion, string kind) : base(apiVersion, kind)
 {
 }
}

class KafkaClusterArgs : CustomResourceArgs
{
 public string ApiVersion { get; set; }
 public string Kind { get; set; }
 public ObjectMetaArgs Metadata { get; set; }
 public Dictionary<string, object> Spec { get; set; }

 public KafkaClusterArgs(string apiVersion, string kind) : base(apiVersion, kind)
 {
 }
}

class KafkaTopicArgs : CustomResourceArgs
{
 public string ApiVersion { get; set; }
 public string Kind { get; set; }
 public ObjectMetaArgs Metadata { get; set; }
 public Dictionary<string, object> Spec { get; set; }

 public KafkaTopicArgs(string apiVersion, string kind) : base(apiVersion, kind)
 {
 }
}

class MyStack : Stack
{
 public MyStack()
 {
 var kafkaNodePool = new Pulumi.Kubernetes.ApiExtensions.CustomResource("exampleKafkaNodePool", new KafkaNodePoolArgs("kafka.strimzi.io/v1beta2", "KafkaNodePool")
 {
 ApiVersion = "kafka.strimzi.io/v1beta2",
 Kind = "KafkaNodePool",
 Metadata = new ObjectMetaArgs
 {
 Name = "example-kafka-node-pool",
 Labels =
 {
 { "strimzi.io/cluster", "example-kafka-cluster" }
 }
 },
 Spec = new Dictionary<string, object>
 {
 { "replicas", 1 },
 { "roles", new List<string> { "controller", "broker" } },
 { "storage", new Dictionary<string, object> { { "type", "ephemeral" } } }
 }
 });

 var kafkaCluster = new Pulumi.Kubernetes.ApiExtensions.CustomResource("exampleKafkaCluster", new KafkaClusterArgs("kafka.strimzi.io/v1beta2", "Kafka")
 {
 ApiVersion = "kafka.strimzi.io/v1beta2",
 Kind = "Kafka",
 Metadata = new ObjectMetaArgs
 {
 Name = "example-kafka-cluster",
 Annotations =
 {
 { "strimzi.io/kraft", "enabled" },
 { "strimzi.io/node-pools", "enabled" }
 }
 },
 Spec = new Dictionary<string, object>
 {
 { "kafka", new Dictionary<string, object> {
 { "replicas", 1 },
 { "version", "3.8.0" },
 { "storage", new Dictionary<string, object> { { "type", "ephemeral" } } },
 { "metadataVersion", "3.8-IV0" },
 { "listeners", new List<Dictionary<string, object>> {
 new Dictionary<string, object> {
 { "name", "plain" },
 { "port", 9092 },
 { "type", "internal" },
 { "tls", false }
 },
 new Dictionary<string, object> {
 { "name", "tls" },
 { "port", 9093 },
 { "type", "internal" },
 { "tls", true }
 }
 }},
 { "config", new Dictionary<string, object> {
 { "offsets.topic.replication.factor", 1 },
 { "transaction.state.log.replication.factor", 1 },
 { "transaction.state.log.min.isr", 1 },
 { "default.replication.factor", 1 },
 { "min.insync.replicas", 1 }
 }}
 }},
 { "entityOperator", new Dictionary<string, object> {
 { "topicOperator", new Dictionary<string, object>() },
 { "userOperator", new Dictionary<string, object>() }
 }}
 }
 });

 var kafkaTopic = new Pulumi.Kubernetes.ApiExtensions.CustomResource("exampleKafkaTopic", new KafkaTopicArgs("kafka.strimzi.io/v1beta2", "KafkaTopic")
 {
 ApiVersion = "kafka.strimzi.io/v1beta2",
 Kind = "KafkaTopic",
 Metadata = new ObjectMetaArgs
 {
 Name = "example-kafka-topic",
 Labels =
 {
 { "strimzi.io/cluster", "example-kafka-cluster" }
 }
 },
 Spec = new Dictionary<string, object>
 {
 { "partitions", 5 },
 { "replicas", 1 },
 { "config", new Dictionary<string, object> {
 { "retention.ms", 7200000 },
 { "segment.bytes", 1073741824 }
 }}
 }
 });
}

class Program
{
 static Task<int> Main() => Deployment.RunAsync<MyStack>();
}

Summary

The Strimzi Operator demonstrates the power of Kubernetes Operators by providing full lifecycle management for Apache Kafka on Kubernetes. It takes care of the heavy lifting of deploying and managing Kafka clusters, making the Strimzi Operator a valuable tool that should be part of every Kubernetes-powered platform, as it covers a wide range of use cases.

Conclusion

In this blog post, I highlighted the importance of Kubernetes Operators for platform engineers and how they can harness the advanced automation capabilities of operators to simplify deploying, managing, and scaling applications and services on Kubernetes.

I provided some examples of advanced Kubernetes operators that I recommend, but if you want to try using Kubernetes Operators in your Kubernetes-powered platform, here are some great guides to get you started with Pulumi and Kubernetes:

Get Started with Pulumi

Use Pulumi's open-source SDK to create, deploy, and manage infrastructure on any cloud.

Pulumi Kubernetes Operator 2.0

Eron Wright — Fri, 18 Oct 2024 00:00:00 +0000

Update: “Pulumi Kubernetes Operator 2.0 is Now Generally Available!”

A few years ago we released the Pulumi Kubernetes Operator, a cloud-native way to manage and deploy cloud infrastructure using Pulumi from within your Kubernetes environment. We’ve heard your feedback about limitations related to scalability and isolation. Today, we’re excited to announce a preview release of version v2.0 of the Pulumi Kubernetes Operator. We’ve put a new, horizontally scalable architecture in place along with a variety of new security features and customization options. Let’s dig in!

What is the Pulumi Kubernetes Operator?

The Pulumi Kubernetes Operator defines a Kubernetes Custom Resource called pulumi.com/v1/Stack, which represents a Pulumi stack. The Pulumi stack can be authored in any supported Pulumi language (TypeScript, Python, Go, .NET, Java, YAML) and can deploy and manage cloud infrastructure in any supported cloud (AWS, Azure, GCP, Kubernetes and 60+ additional cloud and SaaS providers). The Pulumi Kubernetes Operator triggers cloud deployments based on changes to the Stack Custom Resource or the resources it uses.

As a result, the Pulumi Kubernetes Operator enables users to specify the desired state of their cloud infrastructure by using resources managed directly in their Kubernetes cluster. Modifying those Kubernetes resources will trigger creation, updates, and deletion of the underlying cloud infrastructure that they manage.

For example, the Operator can use the Stack resource below to run a Pulumi program, as defined in a GitHub repository, to deploy NGINX. Subsequent commits to the repository, or changes to the stack’s configuration, will cause the Operator to re-deploy the infrastructure.

apiVersion: pulumi.com/v1
kind: Stack
metadata:
 name: nginx-k8s-stack-production
spec:
 stack: acmecorp/nginx/production
 projectRepo: https://github.com/acmecorp/pulumi-nginx
 commit: 2b0889718d3e63feeb6079ccd5e4488d8601e353

The Pulumi Kubernetes Operator can also be used along with Pulumi’s Kubernetes Provider, which allows Pulumi programs to deploy resources to Kubernetes – either in the same cluster as the Operator, or in another cluster. This can be used to package up complex Kubernetes infrastructure into more abstracted units, to mix Kubernetes and cloud infrastructure into a single unit of deployment, to define how application workloads are deployed into a cluster, or for a wide variety of other use cases. The Pulumi Kubernetes Provider has full support for the Kubernetes API available in every Pulumi language, including support for Helm.

What’s New in Pulumi Kubernetes Operator 2.0?

The 2.0 release is based on a whole new architecture for running Pulumi programs in your Kubernetes cluster. The Operator now allocates a dedicated pod for each Stack to serve as the execution environment for Pulumi stack operations. Previously, all stack operations took place within the Operator’s own pod. This new approach effectively isolates each stack’s compute and memory resources, improves the isolation of secrets, and opens up new customization options.

These pods are referred to as workspace pods. Under the hood, a new Workspace Custom Resource handles the lifecycle of the workspace pod, and is created automatically by the system. A related Update Custom Resource applies a stack operation to a given workspace.

The Stack Custom Resource is still the primary API for using the Pulumi Kubernetes Operator and is largely unchanged. See the Migration section below to understand how to prepare your Stack objects for use with the 2.0 release.

Note: a key limitation of the new system is that cross-namespace references are strictly forbidden. For example, a Stack cannot refer to a Secret in another namespace.

Installation

With the 2.0 release, the supported installation modes are limited to a cluster-wide installation. You’ll need to be a cluster admin to install the Operator. The Operator is able to handle stacks across all the namespaces of your cluster.

We provide three ways to install the Operator:

a Pulumi program (see: deploy/pulumi-operator-yaml/)
a Helm chart (see: deploy/helm/pulumi-operator/)
a simple manifest file for dev environments (see: deploy/quickstart/install.yaml)

Please uninstall any earlier version of the Pulumi Kubernetes Operator, then install the new version using one of the above options. Note that this will remove any existing Stack objects in your cluster, so be sure to export their manifests or have a way to restore them.

Security

To protect your Kubernetes cluster, workspace pods are configured to have minimal permissions as defined by the ‘restricted’ profile of the Pod Security Standards. If necessary, you may configure a stack to use the ‘baseline’ profile.

A Kubernetes ServiceAccount provides an identity for your workspace pod, and we recommend that you create a service account for your stack rather than using the default account. If your Pulumi program uses the Kubernetes provider to manage resources within the cluster, the stack’s service account will need permissions, e.g. a RoleBinding to the ClusterRole named cluster-admin. See “Service Accounts” for more information.

The workspace pod has a RPC endpoint that is used by the Operator to run stack operations. The pod protects the RPC endpoint using Kubernetes RBAC, and for that reason you must bind the ClusterRole named system:auth-delegator to the stack’s service account.

Scalability

Each stack is run in its own “workspace” pod, and so the system scales horizontally to support many stacks. The workspace pod is, by default, configured to use minimal resources when idle, and to ‘burst’ into the unused memory and compute resources of the node during a stack operation. You may customize the resource constraints to fit the needs of your stack by using the workspaceTemplate field. If your Kubernetes cluster has high utilization, we recommend increasing the resource requests and limits. See “Pod Quality of Service Classes” for more information.

Once a stack reaches its desired state, the workspace pod may be retained or deleted based on the workspaceReclaimPolicy field. The default behavior is to retain the workspace for subsequent updates. This feature enables you to make a trade-off decision between performance and efficiency.

Stability

A Kubernetes cluster is a dynamic environment where a given pod may be terminated at any time, e.g. due to maintenance operations, unplanned node events, or resource pressure. This can cause problems for ongoing stack operations, and historically may cause a stack to become “locked”, necessitating manual intervention with the use of pulumi cancel.

The new system has been made more resilient by being responsive to pod termination requests. It responds by proactively sending CTRL-C to the ongoing stack operation, to gracefully wind down the operation.

If your stack has a lot of resources, consider increasing the pod’s termination grace period (default: 10 minutes).

Customization and Extensibility

Let’s discuss how to customize the stack’s environment. The Stack resource continues to provide ways to set environment variables and to set stack configuration values. What’s new is the ability to:

use a custom Docker image for your stack environment
set the stack environment’s compute and storage resources
attach volumes, init containers, and sidecars

This is accomplished using the workspaceTemplate field of the Stack resource, to customize the Workspace object that is automatically created by the system for a given stack.

By default, the system uses the official pulumi/pulumi Docker image. You may wish to create your own Docker image that is based on it, e.g. to pre-install your program’s dependencies (using pulumi install) for performance reasons. Be prepared to update the image over time to incorporate the latest version of the Pulumi CLI.

One use case for an extra init container is to customize how your program’s sources are obtained. The system supports Git repositories and Flux sources out-of-the-box, but you have the option of handling it yourself, e.g. to generate a Pulumi project or to download from another source. You simply inject an init container, mount the share volume to /share, and then place the project files into /share/workspace. Contact us to learn more.

Here’s an example of a stack with some customizations applied:

apiVersion: pulumi.com/v1
kind: Stack
metadata:
 name: my-stack
spec:
 ...
 workspaceTemplate:
 spec:
 image: example/pulumi-with-dependencies:v3.334
 podTemplate:
 metadata:
 labels:
 example.com/mylabel: bar
 spec:
 terminationGracePeriodSeconds: 3600
 tolerations:
 - key: "example.com/foo"
 operator: "Exists"
 effect: "NoSchedule"
 initContainers:
 - name: extra # add an extra init container
 image: busybox
 command: ["sh", "-c", "echo 'Hello, extra init container!'"]
 securityContext:
 allowPrivilegeEscalation: false
 capabilities:
 add:
 - NET_BIND_SERVICE
 drop:
 - ALL
 volumeMounts:
 - name: share
 mountPath: /share
 containers:
 - name: pulumi # customize the main container where Pulumi stack operations are executed
 volumeMounts:
 - name: oidc-token
 mountPath: /var/run/secrets/pulumi
 volumes:
 - name: oidc-token # mount an OIDC token
 projected:
 sources:
 - serviceAccountToken:
 audience: urn:pulumi:org:ORG_NAME
 path: token
 expirationSeconds: 3600

Operability

Since each stack has its own long-running pod, you may inspect the console output for a given stack by accessing the pod logs. The pod’s name is based on the stack name; for example, given a stack named my-stack, look for a pod named my-stack-workspace-0. We recommend the stern tool to access pod logs. See “Pod and container logs” for more information.

If you need to run an interactive Pulumi command for your stack, e.g. pulumi import, exec into the workspace pod. Navigate to the /share/workspace directory and you should find your program there.

Use the spec.pulumiLogLevel field to increase the log verbosity level of the Pulumi CLI.

Walkthrough

Here’s a quick demonstration of the new system. Let’s deploy the random-yaml program from the pulumi/examples repository.

We’ll be using Flux for this demonstration; to follow along, install Flux (see “Installation”).

Setup the GitRepository Source

For this demonstration, we’ll use Flux to clone and cache the pulumi/examples repository, since it is a large repository and the Flux cache provides a big performance improvement. The system works well without Flux too.

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
 name: pulumi-examples
 namespace: default
spec:
 interval: 24h
 ref:
 branch: master
 url: https://github.com/pulumi/examples

Install a Pulumi Access Token

Create a Secret containing a Pulumi access token to be used by the stack to authenticate to Pulumi Cloud. Follow these instructions to create a personal, organization, or team access token.

Store the access token into a Kubernetes Secret. Here’s an easy way to create a Secret named pulumi-api-secret. Replace TOKEN with the token that you created.

$ kubectl create secret generic pulumi-api-secret -n default --from-literal=accessToken=TOKEN

Create a Service Account

The stack requires a Kubernetes service account with a binding to the system:auth-delegator cluster role.

If you don’t have admin permissions in your Kubernetes cluster, ask your administrator to create the service account for you. The Pulumi Kubernetes Operator installer also pre-creates a usable ServiceAccount named pulumi in the default namespace.

apiVersion: v1
kind: ServiceAccount
metadata:
 name: random-yaml
 namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
 name: random-yaml:system:auth-delegator
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: ClusterRole
 name: system:auth-delegator
subjects:
- kind: ServiceAccount
 name: random-yaml
 namespace: default

Create the Stack

Let’s apply a Stack object to deploy the program. Note that the specification makes reference to the GitRepository (pulumi-examples), the Secret (pulumi-api-secret), and the ServiceAccount (random-yaml) created earlier.

apiVersion: pulumi.com/v1
kind: Stack
metadata:
 name: random-yaml
 namespace: default
spec:
 serviceAccountName: random-yaml
 fluxSource:
 sourceRef:
 apiVersion: source.toolkit.fluxcd.io/v1
 kind: GitRepository
 name: pulumi-examples
 dir: random-yaml/
 stack: dev
 refresh: true
 continueResyncOnCommitMatch: true
 resyncFrequencySeconds: 300
 destroyOnFinalize: true
 envRefs:
 PULUMI_ACCESS_TOKEN:
 type: Secret
 secret:
 name: pulumi-api-secret
 key: accessToken

This stack is configured to deploy immediately and to resync every five minutes. The term “resync” means to execute pulumi up, which is useful for stacks that produce resources or outputs that require a periodic update. Without resync, pulumi up runs whenever a new commit is pushed to the repository.

Observe the Workspace

The system automatically creates a Workspace object named random-yaml to provision an execution environment. Let’s watch as the workspace’s status progresses towards readiness. For this demonstration, we’ll use the --watch flag to observe the status updates:

$ kubectl get workspace --watch
NAME IMAGE READY ADDRESS
random-yaml pulumi/pulumi:3.134.1-nonroot
random-yaml pulumi/pulumi:3.134.1-nonroot False
random-yaml pulumi/pulumi:3.134.1-nonroot False random-yaml-workspace.default.svc.cluster.local:50051
random-yaml pulumi/pulumi:3.134.1-nonroot False random-yaml-workspace.default.svc.cluster.local:50051
random-yaml pulumi/pulumi:3.134.1-nonroot True random-yaml-workspace.default.svc.cluster.local:50051

Observe the Updates

Once the workspace is ready, the system proceeds to run a periodic update. Each update is represented by an Update object. Let’s watch the progression across three updates:

$ kubectl get update --watch
NAME WORKSPACE PROGRESSING FAILED COMPLETE URL
random-yaml-5kmn7dcm random-yaml
random-yaml-5kmn7dcm random-yaml True False False
random-yaml-5kmn7dcm random-yaml False False True https://app.pulumi.com/eron-pulumi-corp/random/dev/updates/1
random-yaml-rftp8gj8 random-yaml
random-yaml-rftp8gj8 random-yaml True False False
random-yaml-rftp8gj8 random-yaml False False True https://app.pulumi.com/eron-pulumi-corp/random/dev/updates/2
random-yaml-sjpw5hm8 random-yaml
random-yaml-sjpw5hm8 random-yaml True False False
random-yaml-sjpw5hm8 random-yaml False False True https://app.pulumi.com/eron-pulumi-corp/random/dev/updates/3

Observe the Stack

The Stack object maintains a status block showing the result of the latest attempt and whether an update is currently running. Taking a look, we see a successful outcome.

$ kubectl get stack random-yaml -oyaml

apiVersion: pulumi.com/v1
kind: Stack
...
status:
 conditions:
 - lastTransitionTime: "2024-10-17T20:41:49Z"
 message: the stack has been processed and is up to date
 reason: ProcessingCompleted
 status: "True"
 type: Ready
 lastUpdate:
 failures: 0
 generation: 2
 lastAttemptedCommit: sha256:44d554df090dcdeb9bae908cd38155c8c93db05f64dc72982027e8e1294af0d3
 lastResyncTime: "2024-10-17T20:41:49Z"
 lastSuccessfulCommit: sha256:44d554df090dcdeb9bae908cd38155c8c93db05f64dc72982027e8e1294af0d3
 name: random-yaml-sjpw5hm8
 permalink: https://app.pulumi.com/eron-pulumi-corp/random/dev/updates/3
 state: succeeded
 type: up
 observedGeneration: 2
 outputs:
 password: '[secret]'

Delete the Stack

To complete the demonstration, let’s delete the Stack object. Because spec.destroyOnFinalize is enabled, deleting the object causes the Pulumi stack to be destroyed. That is, the stack resources are deleted.

$ kubectl delete stack random-yaml
stack.pulumi.com "random-yaml" deleted

$ kubectl get workspace
No resources found in default namespace.

$ kubectl get update
No resources found in default namespace.

Run Pulumi inside your cluster

Use the Pulumi Kubernetes Operator to deploy and manage cloud infrastructure with the Stack custom resource, driven by the Git changes your cluster already watches.

Get started

Migration

Ready to leverage these new features? Let’s quickly cover how to migrate your existing setup to Pulumi Kubernetes Operator 2.0.

The Stack custom resource continues to be the primary way to use the Pulumi Kubernetes Operator, and your existing stack definitions should continue to work after you follow a simple migration procedure:

Create a ServiceAccount and associated ClusterRoleBinding as shown in the Security section.
If your stack uses the Pulumi Kubernetes Provider, create a RoleBinding or a ClusterRoleBinding to grant the necessary permissions, e.g. to the well-known ClusterRole named cluster-admin. See “Using RBAC Authorization” for more information.
Edit your Stack manifest(s) to use the service account, by setting the spec.serviceAccountName field.
Double-check that your Stack objects do not make use of cross-namespace references.

If you’re using a private Git repository as your program source, use the Stack’s secretRefs field to reference Git credentials. SSH-based access to the repository may require a custom image with your host’s public key(s) baked in. Contact us to learn more.

If you’re using Flux integration, there’s an additional step to be performed by your cluster administrator. Please create a NetworkPolicy to allow the workspace pods to access the Flux source controller. Here’s the manifest to be applied:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
 name: allow-pulumi-fetch-flux-artifacts
 namespace: flux-system
spec:
 podSelector:
 matchLabels:
 app: source-controller
 ingress:
 - ports:
 - protocol: TCP
 port: http
 from:
 - namespaceSelector:
 matchLabels:
 kubernetes.io/metadata.name: default
 - podSelector:
 matchLabels:
 app.kubernetes.io/managed-by: pulumi-kubernetes-operator
 app.kubernetes.io/name: pulumi
 app.kubernetes.io/component: workspace
 policyTypes:
 - Ingress

Conclusion

The Pulumi Kubernetes Operator is a great way to deploy and manage cloud infrastructure from within your Kubernetes environment. With the 2.0 release, we’ve solved the scalability and isolation limitations that affected early adopters. You can get started with the Pulumi Kubernetes Operator today. Please join us on the #kubernetes channel of the Pulumi Community Slack workspace to give feedback, and enjoy!

Pulumi+Kubernetes: New Flux Integration and Inline Programs

Luke Hoban — Sun, 23 Oct 2022 00:00:00 +0000

Pulumi’s Universal Infrastructure as Code platform works with all major clouds and over 100 cloud and SaaS providers, but among all its uses one of the most important is the ability to bring rich Infrastructure as Code tools and practices to Kubernetes projects and teams.

Kubernetes is one of the most used platforms in Pulumi, second only to AWS, with thousands of organizations using Pulumi to manage clusters at scale. Pulumi supports a wide variety of use cases around Kubernetes - from cluster creation and management, to rich and expressive workload definition, to continuous delivery and infrastructure GitOps.

Check out version 2.0 of the Pulumi Kubernetes Operator.

Today we are announcing a set of major updates which deepen and extend Pulumi’s support for Kubernetes and the Kubernetes ecosystem. In this post, we’ll highlight a few of these exciting enhancements:

Pulumi Kubernetes Operator v1.10: New integration with Flux for richer GitOps support, and ability to deploy Pulumi stacks from directly within the Kubernetes resource model
New Pulumi Provider for Flux: Manage Flux with Infrastructure as Code
Pulumi Kubernetes Provider v3.22: Server Side Apply and Resource Patch

You can learn more about Pulumi and Kubernetes or jump in and get started right away, or read on to learn more about what’s new for Pulumi and Kubernetes!

Pulumi Kubernetes Operator v1.10: Flux Integration and Inline Infrastructure support

The Pulumi Kubernetes Operator provides a cloud-native way to manage and deploy cloud infrastructure using Pulumi from within your Kubernetes environment. Infrastructure as Code defined in Pulumi can be automatically deployed using a GitOps model driven from within a Kubernetes environment - leveraging existing Kubernetes-based orchestration, RBAC, policy and resource model investments. We released Pulumi Kubernetes Operator 1.0 last year, and since then have seen users scale up usage of the Operator to drive hundreds of thousands of Pulumi infrastructure updates.

Pulumi Kubernetes Operator v1.10 expands the ways that Pulumi infrastructure can be defined. In addition to the existing Git-based source that has been supported to date, this release adds two new ways to define your infrastructure:

Integration with Flux Sources - enabling any Flux Source to provide the Pulumi program that defines your cloud infrastructure.
Support for Inline Infrastructure sources - a new pulumi.com/v1/Program Custom Resource that can be used to describe cloud infrastructure directly within the Kubernetes resource model.

Together, these open up a wide variety of new ways to apply the Kubernetes Operator as an interface for defining and delivering cloud infrastructure.

Integration with Flux Sources

We’re excited to work with Flux, a CNCF project, to bring even richer GitOps support to the Pulumi Kubernetes Operator. While the Pulumi Kubernetes Operator offers basic GitOps and branch tracking support, it supports only a fraction of the features that Flux’s SourceController provides. Instead of rebuilding all of these features, we chose to provide rich integration with Flux Sources, enabling Pulumi programs to be provided via Flux, and then handed off to drive infrastructure deployments using the Pulumi Kubernetes Operator Stack resource.

Using Flux offers a variety of important new features to Pulumi Kubernetes Operator users:

Support for Flux’s Notification Controller and webhooks to drive highly responsive updates when new commits are pushed
Access to sources other than Git repositories: S3 buckets and OCI repositories
Support for platforms like Azure DevOps that require specific Git implementations
Sources and Stacks can be defined by different teams, since they are separate resources which can have independent RBAC rules applied.
Many advanced Git features including: following semver ranges, excluding files from source, recursively cloning submodules, verifying revision signatures, and more.

To use the Pulumi Kubernetes Operator with Flux, install each of the necessary controllers for Flux and the Pulumi Kubernetes Operator, and then the necessary Role and RoleBindings to allow the Pulumi Kubernetes Operator to read from Flux sources.

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
 name: flux-source-access
rules:
- apiGroups:
 - source.toolkit.fluxcd.io
 resources:
 - "*"
 verbs:
 - get
 - list
 - watch
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
 name: pulumi-flux-source-access
subjects:
- kind: ServiceAccount
 name: pulumi-kubernetes-operator
roleRef:
 kind: Role
 name: flux-source-access
 apiGroup: rbac.authorization.k8s.io

Once those pre-requisites have been met, simply deploy the Flux Source (in this case a GitRepository) and Pulumi Stack resource that define your infrastructure delivery:

---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: GitRepository
metadata:
 name: pkodev
spec:
 interval: 30s
 ref:
 branch: main
 url: https://github.com/lukehoban/pko-dev
---
apiVersion: pulumi.com/v1
kind: Stack
metadata:
 name: pkodev
spec:
 stack: lukehoban/pkodev/test
 fluxSource:
 sourceRef:
 apiVersion: source.toolkit.fluxcd.io/v1beta2
 kind: GitRepository
 name: pkodev
 dir: basic/
 destroyOnFinalize: true

Note that by using the new fluxSource property on a pulumi.com/v1/Stack resource, you can configure it to use the corresponding source instead of an inline definition of the target Git repo and branch. The GitRepository could be replaced with any other Flux Source. For example, to pull our source code from an S3 bucket, use this instead:

---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: Bucket
metadata:
 name: pkodev
spec:
 interval: 5m0s
 provider: aws
 bucketName: podinfo
 endpoint: s3.amazonaws.com
 region: us-east-1
 timeout: 30s
---
apiVersion: pulumi.com/v1
kind: Stack
metadata:
 name: pkodev
spec:
 stack: lukehoban/pkodev/test
 fluxSource:
 sourceRef:
 apiVersion: source.toolkit.fluxcd.io/v1beta2
 kind: Bucket
 name: pkodev
 dir: basic/
 destroyOnFinalize: true

If we are using the GitRepository source along with GitHub, we can use Flux features like WebHook Receivers to expose a webhook that GitHub can trigger on pushes so that updates will be pushed into Flux and the Pulumi Kubernetes Operator directly (instead of waiting to pull them):

---
apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Receiver
metadata:
 name: webapp
 namespace: flux-system
spec:
 type: github
 events:
 - "ping"
 - "push"
 secretRef:
 name: webhook-token
 resources:
 - kind: GitRepository
 name: pkodev

For users who need any of these additional GitOps capabilities, you can now use the best of Pulumi and the best of Flux together with ease.

This integration is made possible by the excellent “open and extensible” GitOps toolkit that Flux v2 provides. We’re excited to work with the Flux community and the team at Weaveworks to help end-users accelerate their GitOps projects:

“When we donated Flux to the CNCF we wanted to empower teams to easily build GitOps automations with the best tools. We’re excited to welcome Pulumi to the Flux and Weave GitOps ecosystem.”

– Alexis Richardson, CEO – Weaveworks

Define Pulumi Cloud Infrastructure Directly in the Kubernetes Resource Model

One of the important benefits of the Pulumi Kubernetes Operator is that it provides the Kubernetes resource model as an interface for defining and managing cloud infrastructure expressed via Pulumi. However, to date, the cloud infrastructure definition itself still has to be stored in Git (or, with the Flux integration above, an S3 bucket or OCI repository). Many users have asked for a model where the Pulumi cloud infrastructure definition itself is provided directly within the Kubernetes resource model as well. With the new pulumi.com/v1/Program custom resource, this is now possible!

This support is enabled by Pulumi’s introduction of Pulumi YAML - the ability to describe cloud infrastructure with access to the entire Pulumi ecosystem of cloud providers and components via simple YAML programs. Importantly, those YAML programs can now be defined directly inside the Kubernetes cluster itself.

Because Pulumi offers access to over 100 cloud providers and components, with thousands of infrastructure resources available across them, the pulumi.com/v1/Program resource offers access to defining rich and complex cloud infrastructure directly from within Kubernetes. And because Pulumi YAML infrastructure definitions support all the same reliable dependency tracking and rich IaC features as any other Pulumi programs, users get all of these benefits from within Kubernetes as well.

To deploy cloud infrastructure directly from within Kubernetes, install the Pulumi Kubernetes Operator and then deploy the Program and Stack resources below with kubectl apply (or via a Pulumi program!):

apiVersion: pulumi.com/v1
kind: Program
metadata:
 name: staticwebsite
program:
 resources:
 bucket:
 type: aws:s3:Bucket
 properties:
 website:
 indexDocument: index.html
 index.html:
 type: aws:s3:BucketObject
 properties:
 bucket: ${bucket.id}
 content: <h1>Hello, world!</h1>
 contentType: text/html
 acl: public-read
 outputs:
 url: http://${bucket.websiteEndpoint}
---
apiVersion: pulumi.com/v1
kind: Stack
metadata:
 name: staticwebsite
spec:
 stack: lukehoban/staticwebsite/test
 programRef:
 name: staticwebsite
 destroyOnFinalize: true
 config:
 aws:region: us-east-1

The result will be to deploy an S3 Bucket and BucketObject exposing a static website into AWS. The Program resource defines a Pulumi program written in Pulumi YAML using any cloud provider resources Pulumi supports, ready to be deployed into as many Pulumi stacks as needed. The program declares an S3 Bucket with website configured, and an index.html file in a BucketObject which will be served from that website. The Stack resource deploys an instance of this program, using the configuration (us-east-1 region) specified.

You can access the URL of the deployed website defined as a program output url via:

$ kubectl get stack staticwebsite -o jsonpath={.status.outputs}

Of course, this could be used to define any cloud infrastructure - an AWS RDS Database, an Azure KeyVault, a CloudFlare DNS Record, a Google Cloud Function, or even another Kubernetes Cluster via EKS, AKS, GKE, Oracle Cloud, Civo, Digital Ocean and more.

Updating the Program resource will immediately drive an update to the associated Pulumi Stack, incrementally updating the managed infrastructure and maintaining the desired cloud infrastructure state.

New Pulumi Provider for Flux

In addition to Flux integration with the Pulumi Kubernetes Operator, we’ve also added a new Flux Provider to the Pulumi Registry. This provider, built by Pulumi community member @oun, enables managing Flux directly via Pulumi Infrastructure as Code.

The Flux provider can be used along with the Kubernetes provider and GitHub provider to stand up and manage a complete E2E GitOps solution from within a single deployment - no need for manual bash, Helm, or runbooks - just simple reliable Infrastructure as Code.

import * as kubernetes from "@pulumi/kubernetes";
import * as flux from "@worawat/flux";

// Get the manifests needed to install Flux with the given components and policies
const fluxManifests = flux.getFluxInstallOutput({
 targetPath: "local",
 components: ['source-controller'],
 networkPolicy: true,
});

// Deploy the manifests into our cluster
const fluxGroup = fluxManifests.content.apply(fluxYaml => {
 new kubernetes.yaml.ConfigGroup("flux-install", {
 yaml: fluxYaml,
 });
});

Pulumi Kubernetes Provider v3.22: Server Side Apply and Resource Patch

In July, we released preview support for Server-Side Apply in the Pulumi Kubernetes provider. We highlighted the benefits and needs that led to adding Server-Side Apply support:

We first launched the Pulumi Kubernetes provider in 2018, and it has grown to be one of the most used providers across the Pulumi ecosystem. Many of our customers rely on this provider to manage critical production workloads, and a common request was for an easier way to manage shared Kubernetes resources with Pulumi. We’ve been listening, and are grateful to everyone who took the time to share their feedback with us!

A lot has changed since 2018, and our provider continues to evolve to support new use cases and the ever-growing richness of the Kubernetes ecosystem. One such development is Server-Side Apply (SSA), which is a resource management strategy that was introduced in Kubernetes v1.18. Kubernetes clients that use SSA can safely share the management of Kubernetes resources by making the API Server responsible for computing diffs and resolving conflicts.

We have kept a close eye on the SSA feature since it was announced and have been waiting patiently for its broad availability in production clusters. The wait is over, and we are excited to bring the power of SSA to Pulumi users!

With the most recent v3.22 release of the Kubernetes provider, Server-Side Apply support is now out of preview and enabled by default for all Kubernetes resources.

This makes several important new features available across the breadth of the Kubernetes provider:

New Patch resource types corresponding to every Kubernetes resource kind. (e.g., NamespacePatch, DeploymentPatch)
“Upsert” support; create a resource if it does not exist, or update the existing resource with specified changes.
Diffs no longer depend on the last-applied-configuration annotation, which fixes a number of subtle issues that users of the provider previously could run into.

These features bring Pulumi and Kubernetes even closer together, enabling Pulumi’s rich Infrastructure as Code to fit even more naturally into the Kubernetes desired state and shared ownership model.

Check out the Managing Resources with Server Side Apply article for more details on these new features.

Conclusion

At Pulumi, we are passionate about pushing the state of the art of Infrastructure as Code forward, and applying modern Infrastructure as Code to support Kubernetes projects and teams is a critical component of that vision. All of these new features around Pulumi and Kubernetes help to further empower teams to build scalable and robust solutions around Kubernetes and Pulumi IaC.

We’re so happy to be able to work with the Flux project to bring all of the benefits of its composable GitOps toolkit to Pulumi users. We’re excited about the new use cases that Inline Infrastructure support in the Pulumi Kubernetes Operator enables. And we’re thrilled to bring Server Side Apply and Patch support to the Pulumi Kubernetes Provider.

You can learn more about Pulumi and Kubernetes or jump in right away by following the getting started guide.

Pulumi Kubernetes Operator 1.0

Luke Hoban — Tue, 12 Oct 2021 00:00:00 +0000

Check out version 2.0 of the Pulumi Kubernetes Operator.

Last year we released the Pulumi Kubernetes Operator, a new cloud-native way to manage and deploy cloud infrastructure using Pulumi from within your Kubernetes environment. Since then, we’ve worked with many Pulumi users who have adopted the Pulumi Kubernetes Operator at increasingly larger scales and for a wide variety of use cases. Today, we’re excited to make the 1.0 release of the Pulumi Kubernetes Operator available.

The Pulumi Kubernetes Operator defines a Kubernetes Custom Resource called pulumi.com/v1/Stack, which represents a Pulumi stack. The Pulumi stack can be authored in any supported Pulumi language (TypeScript, Python, Go, .NET) and can deploy and manage cloud infrastructure in any supported cloud (AWS, Azure, GCP, Kubernetes and 60+ additional cloud and SaaS providers). The Pulumi Kubernetes Operator triggers cloud deployments based on changes to the Stack Custom Resource or the resources it tracks.

As a result, the Pulumi Kubernetes Operator enables users to specify the desired state of their cloud infrastructure using resources managed directly in their Kubernetes cluster, which trigger creation, update and deletion of the detailed cloud infrastructure they need.

For example, the following Kubernetes resource can be created to deploy a Pulumi program which provisions cloud infrastructure to manage NGINX provided in the target Git repo and commit, and future updates to the commit or config settings will deploy updates to the infrastructure.

apiVersion: pulumi.com/v1
kind: Stack
metadata:
 name: nginx-k8s-stack-production
spec:
 stack: acmecorp/nginx/production
 projectRepo: https://github.com/acmecorp/pulumi-nginx
 commit: 2b0889718d3e63feeb6079ccd5e4488d8601e353

The Pulumi Kubernetes Operator can also be used along with Pulumi’s Kubernetes provider, which allows Pulumi programs to deploy resources to Kubernetes, either in the same cluster as the operator, or in another cluster. This can be used to package up complex Kubernetes infrastructure into more abstracted units, to mix Kubernetes and cloud infrastructure into a single unit of deployment, to define how application workloads are deployed into a cluster, or for a wide variety of additional use cases. The Pulumi Kubernetes provider has full support for the Kubernetes API available in every Pulumi language, along with newly improved support for Helm.

We’re at KubeCon North America 2021 this week - come meet us at booth SU8 if you are there! We also have two Kuberentes-focused webinars coming up where you can learn more about working with Pulumi and Kubernetes: Provision, Manage and Scale Kubernetes on Any Cloud (Oct 14th) and From Zero to Production in Kubernetes (Oct 21st).

What’s New in Pulumi Kubernetes Operator 1.0?

The 1.0 release includes many significant enhancements since the initial launch, as well as scalability, performance and quality improvements.

GitOps workflows with branch tracking and private Git repos

The operator now supports specifying a branch instead of a commit in a target Git repo. When a branch is specified, the operator will track any changes to what that branch is tracking, and trigger deployments automatically when new code is pushed. This is a “pull-based” model, where the operator watches the Git repo for changes. This ensures that the operator can be run in a private network environment that is not reachable from the internet (or via GitHub webhooks), and that the operator is fully in control of how these updates are discovered and processed.

This opens up a ton of new use cases for the Pulumi Kubernetes Operator, allowing the declarative model to be pushed from being purely within the Kubernetes resource, to allowing the exact commit to deploy to be specified in a Git repo branch. This allows infrastructure configuration to be defined once for a given environment, and then actively updated via git push. In this mode, the Pulumi Kubernetes Operator can be used in place of traditional CI/CD integrations or other cloud infrastructure deployment workflows.

The same stack as above can be changed to track a branch by using the branch option:

apiVersion: pulumi.com/v1
kind: Stack
metadata:
 name: nginx-k8s-stack-production
spec:
 stack: acmecorp/nginx/production
 projectRepo: https://github.com/acmecorp/pulumi-nginx
 branch: refs/heads/production

This will watch the production branch of the repo for Git pushes, and then trigger updates to the production stack of the NGINX deployment based on the new code pushed to that branch.

To help identify what commit was last deployed when tracking a branch, the Operator publishes information about the last update under .status.lastUpdate, including the lastAttemptedCommit property and lastSuccessfulCommit property.

Along with support for branch tracking, the Operator also now supports access to private Git repositories, critical for deploying your production infrastructure via the Pulumi Kubernetes Operator.

Building on the Pulumi Automation API

A couple months after we released the Pulumi Kubernetes Operator, we also released the initial version of the Pulumi Automation API. The Automation API provides an SDK for embedding Pulumi deployments inside your own software and systems, allowing you to build your own interfaces for managing cloud infrastructure. The design of the Automation API was heavily influenced by our experience building the Pulumi Kubernetes Operator, which was a perfect example of one of these custom interfaces - the Pulumi Kubernetes Operator embeds Pulumi deployments inside Kubernetes, offering a new interface (Kubernetes Stack resources) for managing cloud infrastructure.

Since then, we’ve moved the Pulumi Kubernetes Operator over to use the Pulumi Automation API, significantly reducing the size of the operator implementation, building on a robust foundation shared with dozens of other projects in the Pulumi ecosystem, and benefiting from many improvements and features added to the Pulumi Automation API. Earlier this year, the Pulumi Automation API also reached GA as part of the Pulumi 3.0 release.

Support for any state and secrets management backend

Pulumi supports a wide variety of state management and secrets management backends. Since the initial release of the Pulumi kubernetes Operator, we have added support for using not just the Pulumi Service backends, but also all of the other open source backend options, such as S3, Azure Blob and GCS for state storage, and KMS, KeyVault and Vault for secrets management.

apiVersion: pulumi.com/v1
kind: Stack
metadata:
 name: s3-bucket-stack
spec:
 backend: s3://my-s3-backend
 stack: acmecorp/operator-dev
 projectRepo: https://github.com/acmecorp/pytest1
 initOnCreate: true
 branch: refs/head/master
 config:
 aws:region: us-west-2

Worry-free Upgrades

The Pulumi Kubernetes Operator manages cloud infrastructure deployments which can take seconds to tens of minutes to complete. Although Pulumi ensures these deployments can be continued if interrupted, we want to ensure as little disruption to critical cloud infrastructure deployments as possible via the Operator. As a result, we have added a few key features to ensure that the Pulumi Kubernetes Operator can be safely upgraded to new versions without disrupting any deployments that are in-flight.

This includes capabilities like Leader Election, Graceful Shutdown and more.

Prometheus Metrics

The Pulumi Kubernetes Operator automatically manages a lot of details of Pulumi deployments, and as a result, it’s critical to be able to get insight into the operation of the Operator. Previously, this was available either via operator logs, or via the Pulumi Service. As part of the 1.0 release, the Operator is also now exporting Prometheus-compatible metrics for stacks_active and stacks_failing. These can be used to drive reporting and other workflows inside the Kubernetes cluster.

$ kubectl port-forward service/pulumi-kubernetes-operator-metrics 8383:8383 &
$ curl -s localhost:8383/metrics | grep "stacks_active\|stacks_failing"
# HELP stacks_active Number of stacks currently tracked by the Pulumi Kubernetes Operator
# TYPE stacks_active gauge
stacks_active 2
# HELP stacks_failing Number of stacks currently registered where the last reconcile failed
# TYPE stacks_failing gauge
stacks_failing{name="stack-test-aws-s3-commit-change-5wuczx",namespace="default"} 1

Documentation

We’ve added new documentation and enhanced existing documentation to ensure it’s easy to get started with the Pulumi Kubernetes Operator.

Full Stack Custom Resource API documentation is now available. And guides for deploying the Pulumi Kubernetes Operator and creating Stack resources are available for use either from kubectl/YAML or using the Pulumi Kubernetes Provider.

What’s Next

The 1.0 release of the Pulumi Kubernetes Operator is an important milestone, but there is still lots more planned to further expand the capabilities and scalability of the Operator. Some upcoming areas we’ll be investing:

Conclusion

The Pulumi Kubernetes Operator is a great way to deploy and manage cloud infrastructure from within your Kubernetes environment. With the 1.0 release, we’ve added many significant enhancements, as well as scalability, performance and quality improvements. You can get started with the Pulumi Kubernetes Operator today. Enjoy!

Introducing the Pulumi Kubernetes Operator

Mike Metral — Wed, 12 Aug 2020 00:00:00 +0000

Check out version 2.0 of the Pulumi Kubernetes Operator.

Kubernetes developers and operators work together to manage workloads and to continuously ship software through CI/CD. These users have an affinity for automation and pipelines, and richer integration with Kubernetes is a growing theme across the cloud native ecosystem.

We’re excited to introduce the Pulumi Kubernetes Operator: a Kubernetes controller that deploys cloud infrastructure in Pulumi Stacks for you and your team.

These program stacks include virtual machines, block storage, managed Kubernetes clusters, API resources, serverless functions and more!

Overview

The Pulumi Kubernetes Operator is an extension pattern that enables Kubernetes users to create a Pulumi Stack as a first-class API resource, and use the StackController to drive the updates of the Stack until success.

Check out how to deploy the operator to a Kubernetes cluster using any of Pulumi’s supported languages, or through YAML manifests and kubectl.

You can also get started with Pulumi and create a new managed Kubernetes cluster on Amazon EKS, Google GKE, or Azure AKS if you don’t have an existing cluster.

How it Works

When the operator is deployed, the StackController waits for Stack CustomResources to be created, patched, or deleted in the cluster.

On these events, the Kubernetes controller invokes a reconciliation loop to process the request until it has reached success. For Stack creation or updates, this is a successful pulumi up. For deletions, it is a successful pulumi destroy and pulumi stack rm.

Stacks can be written in Typescript, Python, .NET, and Go, and span cloud providers such as Amazon AWS, Google GCP, and Microsoft Azure, as well as many cloud native services!

The following example showcases a typical Pulumi program in Typescript, that can be instantiated as a Stack.

import * as aws from "@pulumi/aws";

// Create an AWS resource (S3 Bucket)
const names = [];
for (let i = 0; i < 2; i++) {
 const bucket = new aws.s3.Bucket(`my-bucket-${i}`, {
 acl: "public-read",
 });
 names.push(bucket.id);
}

// Export the name of the buckets
export const bucketNames = names;

You can find more examples on GitHub of the types of infrastructure you can manage with Pulumi.

Check out Kubernetes Controllers work if you’d like to learn more.

Creating a Pulumi Stack in Kubernetes

The Stack CustomResourceDefinition (CRD) encapsulates an existing Pulumi infrastructure project in a Git repo, a specific commit SHA to deploy, and any additional settings to control the Pulumi update run.

These settings include your Pulumi API access token, environment variables, config and secrets, and lifecycle controls for the update.

Deploying an NGINX Stack on Kubernetes

Check out an example of creating a Kubernetes NGINX Deployment as a Stack by the Operator in its cluster.

Choose your preferred language below, or check out how to create Pulumi Stacks using kubectl.

import * as pulumi from "@pulumi/pulumi";
import * as k8s from "@pulumi/kubernetes";
import * as kx from "@pulumi/kubernetesx";

// Get the Pulumi API token.
const pulumiConfig = new pulumi.Config();
const pulumiAccessToken = pulumiConfig.requireSecret("pulumiAccessToken")

// Create the API token as a Kubernetes Secret.
const accessToken = new kx.Secret("accesstoken", {
 stringData: { accessToken: pulumiAccessToken },
});

// Create an NGINX deployment in-cluster.
const mystack = new k8s.apiextensions.CustomResource("my-stack", {
 apiVersion: 'pulumi.com/v1alpha1',
 kind: 'Stack',
 spec: {
 accessTokenSecret: accessToken.metadata.name,
 stack: "<YOUR_ORG>/k8s-nginx/dev",
 initOnCreate: true,
 projectRepo: "https://github.com/pulumi/examples",
 repoDir: "kubernetes-ts-nginx",
 commit: "e2e5eb426dbf5b57c50bba0f8eb54fe982ceddb1",
 destroyOnFinalize: true,
 }
});

import pulumi
from pulumi_kubernetes import core, apiextensions

# Get the Pulumi API token.
pulumi_config = pulumi.Config()
pulumi_access_token = pulumi_config.require_secret("pulumiAccessToken")

# Create the API token as a Kubernetes Secret.
access_token = core.v1.Secret("accesstoken", string_data={ "access_token": pulumi_access_token })

# Create an NGINX deployment in-cluster.
my_stack = apiextensions.CustomResource("my-stack",
 api_version="pulumi.com/v1alpha1",
 kind="Stack",
 spec={
 "access_token_secret": access_token.metadata["name"],
 "stack": "<YOUR_ORG>/k8s-nginx/dev",
 "init_on_create": True,
 "project_repo": "https://github.com/pulumi/examples",
 "repo_dir: "kubernetes-ts-nginx",
 "commit": "e2e5eb426dbf5b57c50bba0f8eb54fe982ceddb1",
 "destroy_on_finalize": True,
 }
)

using Pulumi;
using Pulumi.Kubernetes.ApiExtensions;
using Pulumi.Kubernetes.Core.V1;
using Pulumi.Kubernetes.Types.Inputs.Core.V1;

class StackArgs : CustomResourceArgs
{
 [Input("spec")]
 public Input<StackSpecArgs>? Spec { get; set; }

 public StackArgs() : base("pulumi.com/v1alpha1", "Stack")
 {
 }
}

class StackSpecArgs : ResourceArgs
{
 [Input("accessTokenSecret")]
 public Input<string>? AccessTokenSecret { get; set; }

 [Input("stack")]
 public Input<string>? Stack { get; set; }

 [Input("initOnCreate")]
 public Input<bool>? InitOnCreate { get; set; }

 [Input("projectRepo")]
 public Input<string>? ProjectRepo { get; set; }

 [Input("commit")]
 public Input<string>? Commit { get; set; }

 [Input("destroyOnFinalize")]
 public Input<bool>? DestroyOnFinalize { get; set; }
}

class MyStack : Stack
{
 public MyStack()
 {
 // Get the Pulumi API token.
 var config = new Config();
 var pulumiAccessToken = config.RequireSecret("pulumiAccessToken");

 // Create the API token as a Kubernetes Secret.
 var accessToken = new Secret("accesstoken", new SecretArgs
 {
 StringData =
 {
 {"accessToken", pulumiAccessToken}
 }
 });

 // Create an NGINX deployment in-cluster.
 var myStack = new Pulumi.Kubernetes.ApiExtensions.CustomResource("nginx", new StackArgs
 {
 Spec = new StackSpecArgs
 {
 AccessTokenSecret = accessToken.Metadata.Apply(m => m.Name),
 Stack = "<YOUR_ORG>/k8s-nginx/dev",
 InitOnCreate = true,
 ProjectRepo = "https://github.com/pulumi/examples",
 RepoDir = "kubernetes-ts-nginx",
 Commit = "e2e5eb426dbf5b57c50bba0f8eb54fe982ceddb1",
 DestroyOnFinalize = true,
 }
 });
 }
}

package main

import (
 "github.com/pulumi/pulumi-kubernetes/sdk/v2/go/kubernetes"
 apiextensions "github.com/pulumi/pulumi-kubernetes/sdk/v2/go/kubernetes/apiextensions"
 corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v2/go/kubernetes/core/v1"
 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v2/go/kubernetes/meta/v1"
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi"
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi/config"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // Get the Pulumi API token.
 c := config.New(ctx, "")
 pulumiAccessToken := c.Require("pulumiAccessToken")

 // Create the API token as a Kubernetes Secret.
 accessToken, err := corev1.NewSecret(ctx, "accesstoken", &corev1.SecretArgs{
 StringData: pulumi.StringMap{"accessToken": pulumi.String(pulumiAccessToken)},
 })
 if err != nil {
 return err
 }

 // Create an NGINX deployment in-cluster.
 _, err = apiextensions.NewCustomResource(ctx, "my-stack", &apiextensions.CustomResourceArgs{
 ApiVersion: pulumi.String("pulumi.com/v1alpha1"),
 Kind: pulumi.String("Stack"),
 OtherFields: kubernetes.UntypedArgs{
 "spec": map[string]interface{}{
 "accessTokenSecret": accessToken.Metadata.Name(),
 "stack": "<YOUR_ORG>/k8s-nginx/dev",
 "initOnCreate": true,
 "projectRepo": "https://github.com/pulumi/examples",
 "repoDir": "kubernetes-ts-nginx",
 "commit": "e2e5eb426dbf5b57c50bba0f8eb54fe982ceddb1",
 "destroyOnFinalize": true,
 },
 },
 }, pulumi.DependsOn([]pulumi.Resource{accessToken}))
 return err
 })
}

Deploying an Amazon EKS Stack

Check out an example of creating a new AWS VPC and Kubernetes cluster on AWS EKS as a Stack by the Operator.

import * as pulumi from "@pulumi/pulumi";
import * as k8s from "@pulumi/kubernetes";
import * as kx from "@pulumi/kubernetesx";

// Get the Pulumi API token and AWS creds.
const pulumiConfig = new pulumi.Config();
const pulumiAccessToken = pulumiConfig.requireSecret("pulumiAccessToken")
const awsAccessKeyId = pulumiConfig.require("awsAccessKeyId")
const awsSecretAccessKey = pulumiConfig.requireSecret("awsSecretAccessKey")
const awsSessionToken = pulumiConfig.requireSecret("awsSessionToken")

// Create the creds as Kubernetes Secrets.
const accessToken = new kx.Secret("accesstoken", {
 stringData: { accessToken: pulumiAccessToken},
});
const awsCreds = new kx.Secret("aws-creds", {
 stringData: {
 "AWS_ACCESS_KEY_ID": awsAccessKeyId,
 "AWS_SECRET_ACCESS_KEY": awsSecretAccessKey,
 "AWS_SESSION_TOKEN": awsSessionToken,
 },
});

// Create a Kubernetes cluster on AWS EKS.
const mystack = new k8s.apiextensions.CustomResource("my-stack", {
 apiVersion: 'pulumi.com/v1alpha1',
 kind: 'Stack',
 spec: {
 stack: "<YOUR_ORG>/pulumi-aws-eks/dev",
 projectRepo: "https://github.com/metral/pulumi-aws-eks",
 commit: "fc4ab1a3e49d48cf5c764cd8cd626879a90bcc45",
 accessTokenSecret: accessToken.metadata.name,
 config: {
 "aws:region": "us-west-2",
 },
 envSecrets: [awsCreds.metadata.name],
 initOnCreate: true,
 destroyOnFinalize: true,
 }
});

import pulumi
from pulumi_kubernetes import core, apiextensions

# Get the Pulumi API token.
pulumi_config = pulumi.Config()
pulumi_access_token = pulumi_config.require_secret("pulumiAccessToken")
aws_access_key_id = pulumi_config.require("awsAccessKeyId")
aws_secret_access_key = pulumi_config.require_secret("awsSecretAccessKey")
aws_session_token = pulumi_config.require_secret("awsSessionToken")

# Create the creds as Kubernetes Secrets.
access_token = core.v1.Secret("accesstoken", string_data={ "access_token": pulumi_access_token })
aws_creds = core.v1.Secret("aws-creds", string_data={
 "AWS_ACCESS_KEY_ID": aws_access_key_id,
 "AWS_SECRET_ACCESS_KEY": aws_secret_access_key,
 "AWS_SESSION_TOKEN": aws_session_token,
})

# Create a Kubernetes cluster on AWS EKS.
my_stack = apiextensions.CustomResource("my-stack",
 api_version="pulumi.com/v1alpha1",
 kind="Stack",
 spec={
 "stack": "<YOUR_ORG>/pulumi-aws-eks/dev",
 "project_repo": "https://github.com/metral/pulumi-aws-eks",
 "commit": "fc4ab1a3e49d48cf5c764cd8cd626879a90bcc45",
 "access_token_secret": access_token.metadata["name"],
 "config": {
 "aws:region": "us-west-2",
 },
 "env_secrets": [aws_creds.metadata["name"]],
 "init_on_create": True,
 "destroy_on_finalize": True,
 }
)

using System;
using Pulumi;
using Pulumi.Kubernetes.ApiExtensions;
using Pulumi.Kubernetes.Core.V1;
using Pulumi.Kubernetes.Types.Inputs.Core.V1;

class StackArgs : CustomResourceArgs
{
 [Input("spec")]
 public Input<StackSpecArgs>? Spec { get; set; }

 public StackArgs() : base("pulumi.com/v1alpha1", "Stack")
 {
 }
}

class StackSpecArgs : ResourceArgs
{
 [Input("accessTokenSecret")]
 public Input<string>? AccessTokenSecret { get; set; }

 [Input("stack")]
 public Input<string>? Stack { get; set; }

 [Input("initOnCreate")]
 public Input<bool>? InitOnCreate { get; set; }

 [Input("projectRepo")]
 public Input<string>? ProjectRepo { get; set; }

 [Input("commit")]
 public Input<string>? Commit { get; set; }

 [Input("destroyOnFinalize")]
 public Input<bool>? DestroyOnFinalize { get; set; }

 [Input("envSecrets")]
 public InputList<String>? EnvSecrets { get; set; }

 [Input("config")]
 public InputMap<String>? Config { get; set; }
}

class MyStack : Stack
{
 public MyStack()
 {
 // Get the Pulumi API token.
 var config = new Config();
 var pulumiAccessToken = config.RequireSecret("pulumiAccessToken");
 var awsAccessKeyId = config.Require("awsAccessKeyId");
 var awsSecretAccessKey = config.RequireSecret("awsSecretAccessKey");
 var awsSessionToken = config.RequireSecret("awsSessionToken");

 // Create the creds as Kubernetes Secrets.
 var accessToken = new Secret("accesstoken", new SecretArgs
 {
 StringData =
 {
 {"accessToken", pulumiAccessToken}
 }
 });
 var awsCreds = new Secret("aws-creds", new SecretArgs
 {
 StringData =
 {
 {"AWS_ACCESS_KEY_ID", awsAccessKeyId},
 {"AWS_SECRET_ACCESS_KEY", awsSecretAccessKey},
 {"AWS_SESSION_TOKEN", awsSessionToken}
 }
 });

 // Create an Kubernetes cluster on AWS EKS.
 var myStack = new Pulumi.Kubernetes.ApiExtensions.CustomResource("my-stack", new StackArgs
 {
 Spec = new StackSpecArgs
 {
 Stack = "<YOUR_ORG>/pulumi-aws-eks/dev",
 ProjectRepo = "https://github.com/metral/pulumi-aws-eks",
 Commit = "fc4ab1a3e49d48cf5c764cd8cd626879a90bcc45",
 AccessTokenSecret = accessToken.Metadata.Apply(m => m.Name),
 Config =
 {
 {"aws:region", "us-west-2"}
 },
 EnvSecrets = {awsCreds.Metadata.Apply(m => m.Name)},
 InitOnCreate = true,
 DestroyOnFinalize = true,
 }
 });
 }
}

package main

import (
 "github.com/pulumi/pulumi-kubernetes/sdk/v2/go/kubernetes"
 apiextensions "github.com/pulumi/pulumi-kubernetes/sdk/v2/go/kubernetes/apiextensions"
 corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v2/go/kubernetes/core/v1"
 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v2/go/kubernetes/meta/v1"
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi"
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi/config"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // Get the Pulumi API token and AWS creds.
 config := config.New(ctx, "")
 pulumiAccessToken := config.Require("pulumiAccessToken")
 awsAccessKeyID := config.Require("awsAccessKeyId")
 awsSecretAccessKey := config.Require("awsSecretAccessKey")
 awsSessionToken := config.Require("awsSessionToken")

 // Create the creds as Kubernetes Secrets.
 accessToken, err := corev1.NewSecret(ctx, "accesstoken", &corev1.SecretArgs{
 StringData: pulumi.StringMap{"accessToken": pulumi.String(pulumiAccessToken)},
 })
 if err != nil {
 return err
 }
 awsCreds, err := corev1.NewSecret(ctx, "aws-creds", &corev1.SecretArgs{
 StringData: pulumi.StringMap{
 "AWS_ACCESS_KEY_ID": pulumi.String(awsAccessKeyID),
 "AWS_SECRET_ACCESS_KEY": pulumi.String(awsSecretAccessKey),
 "AWS_SESSION_TOKEN": pulumi.String(awsSessionToken),
 },
 })
 if err != nil {
 return err
 }

 // Create an Kubernetes cluster on AWS EKS.
 _, err = apiextensions.NewCustomResource(ctx, "my-stack",
 &apiextensions.CustomResourceArgs{
 ApiVersion: pulumi.String("pulumi.com/v1alpha1"),
 Kind: pulumi.String("Stack"),
 OtherFields: kubernetes.UntypedArgs{
 "spec": map[string]interface{}{
 "stack": "<YOUR_ORG>/s3-op-project/dev",
 "projectRepo": "https://github.com/metral/pulumi-aws-eks",
 "commit": "fc4ab1a3e49d48cf5c764cd8cd626879a90bcc45",
 "accessTokenSecret": accessToken.Metadata.Name(),
 "config": map[string]string{
 "aws:region": "us-west-2",
 },
 "envSecrets": []interface{}{awsCreds.Metadata.Name()},
 "initOnCreate": true,
 "destroyOnFinalize": true,
 },
 },
 }, pulumi.DependsOn([]pulumi.Resource{accessToken, awsCreds}))

 return nil
 })
}

CI/CD Scenarios

As you build out your Kubernetes workloads, you’ll need to implement update strategies to ship new software easily and reliably. Kubernetes API resources like Deployments are used to model stateless apps and operate their lifecycles e.g. recreate app replicas, or perform a rolling update.

A common approach that most teams rely on is the Blue / Green deployment. This starts with an initial version of the app, and then deploys an updated version alongside the initial version. Once ready, traffic is switched over from the initial version to the new version with no downtime.

In Kubernetes, Blue/Green is modeled by using a Service that load balances and selects a set of Deployment pods labeled “blue,” and then switching to a different selection of updated, “green” pods.

Similar to the Stack CustomResources in the previous sections, we can deploy a Stack of a Blue/Green Kubernetes app in Pulumi, and step through a sequence of its Git commits similar to how a CI/CD pipeline does. Additionally, we can use the Pulumi config system to parameterize settings and secrets on how to manage the Stack CustomResource project.

Deploying the active blue and passive green versions of the app can be done on the example Stack with the following steps.

pulumi config set --secret pulumiAccessToken <YOUR_PULUMI_API_TOKEN>
pulumi config set stackProjectRepo https://github.com/metral/pulumi-blue-green
pulumi config set stackCommit b19759220f25476605620fdfffeface39a630246
pulumi up -y

After deployment, we’ll test that the v1 / “blue” endpoint is live and running. The v2 / “green” app will be on standby.

while true; do curl http://34.83.25.150:80/version ; echo; sleep 1; done

{"id":1,"content":"current"}
{"id":1,"content":"current"}
{"id":1,"content":"current"}
{"id":1,"content":"current"}
...

When we’re ready to switch traffic, the blue/green transition is done by updating the config to the Git new commit of the selector swap, and running an update.

pulumi config set stackCommit f4bf2f7b54ec441d5af374933cca4ef09f2bce24
pulumi up -y

After a few seconds you should start seeing the generated traffic returning the new version in a clean cut-over with no dropped packets.

...
{"id":1,"content":"current"}
{"id":1,"content":"current"}
{"id":1,"content":"current"}
{"id":2,"content":"new"}
{"id":2,"content":"new"}
{"id":2,"content":"new"}
{"id":2,"content":"new"}
...

Check out the full walkthrough for a tutorial, and the video clip below for a demo.

Wrap-Up

The Pulumi Kubernetes Operator helps deploy Pulumi Stacks in your Kubernetes cluster, and naturally fits in with your CI/CD process.

We covered examples like deploying an NGINX Stack in the same cluster as the Operator, creating a new Kubernetes cluster in an AWS EKS Stack, and how Stacks can be abstracted for deployment strategies like Blue/Green.

There are many more examples of deploying infrastructure, containers, serverless, and Kubernetes across cloud providers and cloud native services.

Next Steps

Check out the GitHub repo to experiment deploying the operator and some test Stacks.

Learn more about how Pulumi works with Kubernetes, and Get Started if you’re new.

You can help to shape this experience directly by providing feedback on GitHub. We love to hear from our users!

You can explore more content by checking out PulumiTV on YouTube, work through Kubernetes tutorials to dive deeper, and join the Community Slack to engage with users and the Pulumi team.