Pulumi Blog: Continuous delivery

How We Eliminated Long-Lived CI Secrets Across 70+ Repos

Boris Schlosser — Tue, 31 Mar 2026 00:00:00 +0000

Supply chain attacks on CI/CD pipelines are accelerating. A growing pattern involves attackers compromising popular GitHub Actions through tag poisoning — rewriting trusted version tags to point to malicious code that harvests environment variables, cloud credentials, and API tokens from runner environments. The stolen credentials are then exfiltrated to attacker-controlled infrastructure, often before anyone notices.

For every engineering organization, the question is no longer if your CI pipeline will encounter a compromised dependency, but what is exposed when it does.

At Pulumi, we asked ourselves that question and decided the answer should be “nothing useful.” Here’s how we got there.

The problem with static CI secrets

Most organizations store long-lived cloud credentials, API tokens, and service account keys as GitHub repository or organization secrets. But this approach has several well-known problems:

Broad availability. Every workflow run on a repository can access every secret stored in that repo. A compromised action in any workflow can read them all.
No expiration. Secrets persist until someone manually rotates them. If exfiltrated, they give attackers persistent access for weeks or months.
No granular audit trail. GitHub tells you a secret was used, but not which workflow, which step, or what it was used for.
Secret sprawl. Across dozens or hundreds of repos, the same credentials are often duplicated, making rotation a coordinated, error-prone effort.

In a supply chain attack scenario, this is exactly what attackers count on: a single compromised action that can dump a trove of long-lived credentials.

Our approach: zero static secrets

We replaced every static GitHub Secret across our CI pipelines with short-lived, dynamically fetched credentials using Pulumi ESC and OpenID Connect (OIDC). The credential flow works in layers, each scoped and ephemeral:

GitHub generates a short-lived OIDC token scoped to the specific workflow run, repository, and branch. This token is cryptographically signed by GitHub’s OIDC provider.
The token is exchanged with Pulumi Cloud for a short-lived Pulumi access token. Pulumi Cloud validates the OIDC claims (organization, repository, branch) against a configured trust policy before issuing the token.
The Pulumi access token opens an ESC environment to retrieve the credentials the workflow needs — cloud provider keys, API tokens, or other secrets.
Cloud credentials themselves are dynamic. ESC environments use OIDC login providers to fetch short-lived credentials directly from AWS, Azure, or GCP. No static keys or cloud credentials are stored anywhere.

The pulumi/esc-action GitHub Action handles this entire flow in a single workflow step.

sequenceDiagram
participant Runner as GitHub Actions Runner
participant GH as GitHub OIDC Provider
participant PC as Pulumi Cloud
participant ESC as Pulumi ESC
participant Cloud as Cloud Provider (AWS/Azure/GCP)
Runner->>GH: Request OIDC token (scoped to workflow run)
GH-->>Runner: Short-lived JWT
Runner->>PC: Exchange JWT for Pulumi access token
PC-->>Runner: Short-lived access token
Runner->>ESC: Open environment with access token
ESC->>Cloud: OIDC login (assume role / federated identity)
Cloud-->>ESC: Short-lived cloud credentials
ESC-->>Runner: Cloud credentials + secrets
Note over Runner,Cloud: Nothing is stored. Everything expires.

What the change looks like

Before this migration, our workflows referenced static secrets stored in GitHub:

env:
 AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
 AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

After the migration, an ESC environment handles credential fetching via OIDC. Here is what the environment definition looks like:

values:
 aws:
 login:
 fn::open::aws-login:
 oidc:
 duration: 1h
 roleArn: arn:aws:iam::123456789012:role/pulumi-esc-role
 sessionName: esc-${context.pulumi.user.login}
 # Optional: scope down the session beyond what the role allows
 policyArns:
 - arn:aws:iam::123456789012:policy/ci-build-minimal
 environmentVariables:
 AWS_ACCESS_KEY_ID: ${aws.login.accessKeyId}
 AWS_SECRET_ACCESS_KEY: ${aws.login.secretAccessKey}

The roleArn and optional policyArns make least-privilege straightforward: each login provider assumes a specific role, and policyArns can scope the session down further. You can use multiple login providers in one environment or separate environments per workflow to match permissions to each job’s needs.

The workflow itself becomes minimal — a single step that authenticates via OIDC and injects the credentials:

permissions:
 contents: read
 id-token: write  # Required for OIDC

steps:
 - name: Fetch secrets from ESC
 uses: pulumi/esc-action@v3
 with:
 environment: '<your-organization>/<your-esc-env>'

The static secrets.* references are gone entirely. Every credential is fetched at runtime through ESC.

Scale: 70+ repos, zero static secrets

We didn’t do this for one or two flagship repos; we rolled it out across every Pulumi provider repository: AWS, Azure, GCP, Kubernetes, and over 60 more. The migration was managed centrally through our ci-mgmt tooling, which generates consistent workflow configurations across all provider repos.

The pattern is the same everywhere:

Each repo has a corresponding ESC environment under a github-secrets/ project.
All workflow-level ${{ secrets.* }} references have been removed.
The pulumi/esc-action step with OIDC auth is the single entry point for all credentials.

When every repo follows the same pattern like this, security posture is much more easily verifiable and auditable.

Auditability and centralized control

Beyond eliminating static secrets, this migration gave us centralized visibility and control that GitHub Secrets cannot provide:

Audit logging. ESC records which credentials were accessed, when, and by which workflow. This is a meaningful improvement over GitHub’s binary “secret was used” signal.
Centralized access policies. Access rules are defined once in ESC rather than scattered across individual repository settings pages.
Single-point rotation. Because ESC environments can import other environments, shared credentials live in a common base that all 70+ repo environments are composed of. Update it once, and every repo picks up the change on its next run.
Dynamic credentials by default. For cloud providers like AWS, Azure, and GCP, ESC fetches credentials via OIDC at open time. There is nothing to rotate because nothing is stored.

What happens if a GitHub Action is compromised

With this architecture in place, here is what an attacker gets if a compromised GitHub Action runs in our CI:

No GitHub Secrets to dump. The repository settings page has no stored secrets for a malicious action to exfiltrate.
OIDC tokens are scoped and short-lived. The GitHub-issued JWT is valid only for the specific workflow run and expires within minutes.
Cloud credentials are ephemeral. Any AWS, Azure, or GCP credentials fetched through ESC are short-lived and scoped to the role assumed during that run.
No persistent access. There are no long-lived tokens to reuse hours or days later.

Compare this to the traditional model, where a single compromised action could exfiltrate AWS access keys that remain valid until someone manually rotates them — which could be weeks or months.

The goal is not to prevent every possible attack. It is to make the blast radius as small as possible when something goes wrong.

Get started

If you want to adopt the same pattern in your own CI pipelines:

Tutorial: Using ESC with GitHub Actions — Step-by-step setup guide.
Announcing the Pulumi ESC GitHub Action — Full feature overview and capabilities.
Configuring OIDC for ESC — Set up OIDC trust between ESC and your cloud providers.
Pulumi ESC documentation — Full documentation for environments, secrets, and configuration.

Your CI secrets do not have to be a liability. With OIDC and Pulumi ESC, they do not have to exist at all.

Pulumi Kubernetes Operator v2.3.0: Preview Mode and Structured Configuration

Eron Wright — Tue, 25 Nov 2025 00:00:00 +0000

We’re excited to announce the release of Pulumi Kubernetes Operator v2.3.0, introducing two powerful capabilities that enhance GitOps workflows: preview mode for validating infrastructure changes before deployment, and structured configuration support for managing complex data types. Building on the success of the v2.0 GA release, this update addresses long-standing community requests while maintaining full backwards compatibility. These features enable safer, more sophisticated infrastructure management patterns for platform engineering teams.

Preview mode: Validate infrastructure changes before deployment

Preview mode enables you to run Pulumi stacks in dry-run fashion, allowing you to visualize what infrastructure changes would occur without actually applying them. This capability is essential for GitOps workflows that require validation gates and incremental rollouts.

By adding preview: true to your Stack specification, the operator runs pulumi preview instead of pulumi up. The Stack’s Ready condition indicates preview success, and you get full status including preview links, standard output, and program outputs—all without making actual infrastructure changes.

This unlocks sophisticated workflow patterns:

Multiple Stacks for what-if analysis: Create several Stack resources pointing to the same Pulumi stack, with all-but-one operating in preview mode to compare different configurations
Branch validation: Preview changes from feature branches before merging to production
Tick-tock rollout patterns: Toggle the preview flag on and off with external verification steps between each deployment phase

Here’s a simple example showing a production stack alongside its preview counterpart:

# Production stack
apiVersion: pulumi.com/v1
kind: Stack
metadata:
 name: my-infrastructure
spec:
 stack: org/project/prod
 projectRepo: https://github.com/example/infra
 branch: main
---
# Preview stack - same Pulumi stack, preview mode
apiVersion: pulumi.com/v1
kind: Stack
metadata:
 name: my-infrastructure-preview
spec:
 stack: org/project/prod
 projectRepo: https://github.com/example/infra
 branch: feature-branch
 preview: true # Only runs pulumi preview

Preview mode closes issue #16, one of our longest-standing feature requests.

Structured configuration: Native support for complex data types

Configuration management takes a significant step forward with native support for complex data types. Previously limited to string values, Stack configuration now supports objects, arrays, numbers, and booleans as first-class citizens.

This enhancement addresses the reality that complex environments require rich configuration. You can now express sophisticated configuration structures inline in your Stack manifests or load them from ConfigMaps with automatic JSON parsing—all using Kubernetes-native patterns.

The implementation leverages Pulumi CLI’s JSON configuration support (available in v3.202.0+) with automatic version detection. If your workspace uses an earlier CLI version, you’ll receive clear guidance to upgrade. Existing string-only configurations continue to work without modification, ensuring full backwards compatibility.

Here’s an example of inline structured configuration:

apiVersion: pulumi.com/v1
kind: Stack
metadata:
 name: my-app
spec:
 stack: org/app/prod
 projectRepo: https://github.com/example/app
 branch: main
 config:
 # String values (existing behavior)
 environment: "production"

 # Objects (NEW)
 database:
 host: "db.example.com"
 port: 5432
 ssl: true

 # Arrays (NEW)
 regions: ["us-west-2", "us-east-1", "eu-west-1"]

 # Numbers and booleans (NEW)
 maxConnections: 100
 enableCaching: true

You can also reference ConfigMaps for more complex scenarios:

apiVersion: v1
kind: ConfigMap
metadata:
 name: app-settings
data:
 database.json: |
 {
 "host": "db.example.com",
 "port": 5432,
 "maxConnections": 100
 }
---
apiVersion: pulumi.com/v1
kind: Stack
metadata:
 name: my-app
spec:
 stack: org/app/prod
 projectRepo: https://github.com/example/app
 branch: main
 configRefs:
 database:
 name: app-settings
 key: database.json
 json: true # Parse as JSON

Note that Secrets are not a supported source of structured configuration values.

Structured configuration closes issue #258 and issue #872, addressing long-standing configuration management needs from the community.

Bug fixes and reliability improvements

This release includes several fixes that improve operational reliability:

Stack name validation: Added validation to limit Stack names to 42 characters, preventing resource naming conflicts (#899)
secretsProvider fix: The secretsProvider parameter now properly applies when creating new stacks (#935)
Helm chart fix: Resolved YAML parsing errors for podLabels containing special characters like colons (#1014)
Stack deletion: Stack deletion is no longer blocked when prerequisite stacks are missing, enabling proper cleanup workflows (#751)
Update TTL: Completed Update objects now respect the ttlAfterCompleted setting for automatic garbage collection (#960)

Upgrading to v2.3.0

This release includes updates to the Custom Resource Definitions (CRDs) to support the new features. If you’re using Helm, you’ll need to manually apply the updated CRDs before upgrading, as Helm v3 does not automatically upgrade CRDs:

# Apply updated CRDs
kubectl apply --server-side -k 'github.com/pulumi/pulumi-kubernetes-operator//operator/config/crd?ref=v2.3.0'

# Upgrade via Helm
helm upgrade pulumi-kubernetes-operator \
 oci://ghcr.io/pulumi/helm-charts/pulumi-kubernetes-operator \
 --version 2.3.0 \
 -n pulumi-kubernetes-operator

If you’re using the quickstart YAML installation method, the CRDs will update automatically when you apply the new manifest.

All changes are backwards compatible—existing Stack resources work without modification. For structured configuration support, ensure your workspace pods use Pulumi CLI v3.202.0 or later; the operator provides automatic version detection with clear upgrade guidance when needed.

Get started today

The v2.3.0 release enhances the Pulumi Kubernetes Operator with safer deployment workflows and more flexible configuration management. Preview mode enables validation-first GitOps patterns, while structured configuration simplifies complex multi-environment setups.

Get started with the Pulumi Kubernetes Operator in our documentation, or view the complete v2.3.0 release notes on GitHub. Join the conversation in the Pulumi Community Slack #kubernetes channel—we’d love to hear how these features impact your workflows.

Pulumi Kubernetes Operator 2.0

Eron Wright — Fri, 18 Oct 2024 00:00:00 +0000

Update: “Pulumi Kubernetes Operator 2.0 is Now Generally Available!”

A few years ago we released the Pulumi Kubernetes Operator, a cloud-native way to manage and deploy cloud infrastructure using Pulumi from within your Kubernetes environment. We’ve heard your feedback about limitations related to scalability and isolation. Today, we’re excited to announce a preview release of version v2.0 of the Pulumi Kubernetes Operator. We’ve put a new, horizontally scalable architecture in place along with a variety of new security features and customization options. Let’s dig in!

What is the Pulumi Kubernetes Operator?

The Pulumi Kubernetes Operator defines a Kubernetes Custom Resource called pulumi.com/v1/Stack, which represents a Pulumi stack. The Pulumi stack can be authored in any supported Pulumi language (TypeScript, Python, Go, .NET, Java, YAML) and can deploy and manage cloud infrastructure in any supported cloud (AWS, Azure, GCP, Kubernetes and 60+ additional cloud and SaaS providers). The Pulumi Kubernetes Operator triggers cloud deployments based on changes to the Stack Custom Resource or the resources it uses.

As a result, the Pulumi Kubernetes Operator enables users to specify the desired state of their cloud infrastructure by using resources managed directly in their Kubernetes cluster. Modifying those Kubernetes resources will trigger creation, updates, and deletion of the underlying cloud infrastructure that they manage.

For example, the Operator can use the Stack resource below to run a Pulumi program, as defined in a GitHub repository, to deploy NGINX. Subsequent commits to the repository, or changes to the stack’s configuration, will cause the Operator to re-deploy the infrastructure.

apiVersion: pulumi.com/v1
kind: Stack
metadata:
 name: nginx-k8s-stack-production
spec:
 stack: acmecorp/nginx/production
 projectRepo: https://github.com/acmecorp/pulumi-nginx
 commit: 2b0889718d3e63feeb6079ccd5e4488d8601e353

The Pulumi Kubernetes Operator can also be used along with Pulumi’s Kubernetes Provider, which allows Pulumi programs to deploy resources to Kubernetes – either in the same cluster as the Operator, or in another cluster. This can be used to package up complex Kubernetes infrastructure into more abstracted units, to mix Kubernetes and cloud infrastructure into a single unit of deployment, to define how application workloads are deployed into a cluster, or for a wide variety of other use cases. The Pulumi Kubernetes Provider has full support for the Kubernetes API available in every Pulumi language, including support for Helm.

What’s New in Pulumi Kubernetes Operator 2.0?

The 2.0 release is based on a whole new architecture for running Pulumi programs in your Kubernetes cluster. The Operator now allocates a dedicated pod for each Stack to serve as the execution environment for Pulumi stack operations. Previously, all stack operations took place within the Operator’s own pod. This new approach effectively isolates each stack’s compute and memory resources, improves the isolation of secrets, and opens up new customization options.

These pods are referred to as workspace pods. Under the hood, a new Workspace Custom Resource handles the lifecycle of the workspace pod, and is created automatically by the system. A related Update Custom Resource applies a stack operation to a given workspace.

The Stack Custom Resource is still the primary API for using the Pulumi Kubernetes Operator and is largely unchanged. See the Migration section below to understand how to prepare your Stack objects for use with the 2.0 release.

Note: a key limitation of the new system is that cross-namespace references are strictly forbidden. For example, a Stack cannot refer to a Secret in another namespace.

Installation

With the 2.0 release, the supported installation modes are limited to a cluster-wide installation. You’ll need to be a cluster admin to install the Operator. The Operator is able to handle stacks across all the namespaces of your cluster.

We provide three ways to install the Operator:

a Pulumi program (see: deploy/pulumi-operator-yaml/)
a Helm chart (see: deploy/helm/pulumi-operator/)
a simple manifest file for dev environments (see: deploy/quickstart/install.yaml)

Please uninstall any earlier version of the Pulumi Kubernetes Operator, then install the new version using one of the above options. Note that this will remove any existing Stack objects in your cluster, so be sure to export their manifests or have a way to restore them.

Security

To protect your Kubernetes cluster, workspace pods are configured to have minimal permissions as defined by the ‘restricted’ profile of the Pod Security Standards. If necessary, you may configure a stack to use the ‘baseline’ profile.

A Kubernetes ServiceAccount provides an identity for your workspace pod, and we recommend that you create a service account for your stack rather than using the default account. If your Pulumi program uses the Kubernetes provider to manage resources within the cluster, the stack’s service account will need permissions, e.g. a RoleBinding to the ClusterRole named cluster-admin. See “Service Accounts” for more information.

The workspace pod has a RPC endpoint that is used by the Operator to run stack operations. The pod protects the RPC endpoint using Kubernetes RBAC, and for that reason you must bind the ClusterRole named system:auth-delegator to the stack’s service account.

Scalability

Each stack is run in its own “workspace” pod, and so the system scales horizontally to support many stacks. The workspace pod is, by default, configured to use minimal resources when idle, and to ‘burst’ into the unused memory and compute resources of the node during a stack operation. You may customize the resource constraints to fit the needs of your stack by using the workspaceTemplate field. If your Kubernetes cluster has high utilization, we recommend increasing the resource requests and limits. See “Pod Quality of Service Classes” for more information.

Once a stack reaches its desired state, the workspace pod may be retained or deleted based on the workspaceReclaimPolicy field. The default behavior is to retain the workspace for subsequent updates. This feature enables you to make a trade-off decision between performance and efficiency.

Stability

A Kubernetes cluster is a dynamic environment where a given pod may be terminated at any time, e.g. due to maintenance operations, unplanned node events, or resource pressure. This can cause problems for ongoing stack operations, and historically may cause a stack to become “locked”, necessitating manual intervention with the use of pulumi cancel.

The new system has been made more resilient by being responsive to pod termination requests. It responds by proactively sending CTRL-C to the ongoing stack operation, to gracefully wind down the operation.

If your stack has a lot of resources, consider increasing the pod’s termination grace period (default: 10 minutes).

Customization and Extensibility

Let’s discuss how to customize the stack’s environment. The Stack resource continues to provide ways to set environment variables and to set stack configuration values. What’s new is the ability to:

use a custom Docker image for your stack environment
set the stack environment’s compute and storage resources
attach volumes, init containers, and sidecars

This is accomplished using the workspaceTemplate field of the Stack resource, to customize the Workspace object that is automatically created by the system for a given stack.

By default, the system uses the official pulumi/pulumi Docker image. You may wish to create your own Docker image that is based on it, e.g. to pre-install your program’s dependencies (using pulumi install) for performance reasons. Be prepared to update the image over time to incorporate the latest version of the Pulumi CLI.

One use case for an extra init container is to customize how your program’s sources are obtained. The system supports Git repositories and Flux sources out-of-the-box, but you have the option of handling it yourself, e.g. to generate a Pulumi project or to download from another source. You simply inject an init container, mount the share volume to /share, and then place the project files into /share/workspace. Contact us to learn more.

Here’s an example of a stack with some customizations applied:

apiVersion: pulumi.com/v1
kind: Stack
metadata:
 name: my-stack
spec:
 ...
 workspaceTemplate:
 spec:
 image: example/pulumi-with-dependencies:v3.334
 podTemplate:
 metadata:
 labels:
 example.com/mylabel: bar
 spec:
 terminationGracePeriodSeconds: 3600
 tolerations:
 - key: "example.com/foo"
 operator: "Exists"
 effect: "NoSchedule"
 initContainers:
 - name: extra # add an extra init container
 image: busybox
 command: ["sh", "-c", "echo 'Hello, extra init container!'"]
 securityContext:
 allowPrivilegeEscalation: false
 capabilities:
 add:
 - NET_BIND_SERVICE
 drop:
 - ALL
 volumeMounts:
 - name: share
 mountPath: /share
 containers:
 - name: pulumi # customize the main container where Pulumi stack operations are executed
 volumeMounts:
 - name: oidc-token
 mountPath: /var/run/secrets/pulumi
 volumes:
 - name: oidc-token # mount an OIDC token
 projected:
 sources:
 - serviceAccountToken:
 audience: urn:pulumi:org:ORG_NAME
 path: token
 expirationSeconds: 3600

Operability

Since each stack has its own long-running pod, you may inspect the console output for a given stack by accessing the pod logs. The pod’s name is based on the stack name; for example, given a stack named my-stack, look for a pod named my-stack-workspace-0. We recommend the stern tool to access pod logs. See “Pod and container logs” for more information.

If you need to run an interactive Pulumi command for your stack, e.g. pulumi import, exec into the workspace pod. Navigate to the /share/workspace directory and you should find your program there.

Use the spec.pulumiLogLevel field to increase the log verbosity level of the Pulumi CLI.

Walkthrough

Here’s a quick demonstration of the new system. Let’s deploy the random-yaml program from the pulumi/examples repository.

We’ll be using Flux for this demonstration; to follow along, install Flux (see “Installation”).

Setup the GitRepository Source

For this demonstration, we’ll use Flux to clone and cache the pulumi/examples repository, since it is a large repository and the Flux cache provides a big performance improvement. The system works well without Flux too.

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
 name: pulumi-examples
 namespace: default
spec:
 interval: 24h
 ref:
 branch: master
 url: https://github.com/pulumi/examples

Install a Pulumi Access Token

Create a Secret containing a Pulumi access token to be used by the stack to authenticate to Pulumi Cloud. Follow these instructions to create a personal, organization, or team access token.

Store the access token into a Kubernetes Secret. Here’s an easy way to create a Secret named pulumi-api-secret. Replace TOKEN with the token that you created.

$ kubectl create secret generic pulumi-api-secret -n default --from-literal=accessToken=TOKEN

Create a Service Account

The stack requires a Kubernetes service account with a binding to the system:auth-delegator cluster role.

If you don’t have admin permissions in your Kubernetes cluster, ask your administrator to create the service account for you. The Pulumi Kubernetes Operator installer also pre-creates a usable ServiceAccount named pulumi in the default namespace.

apiVersion: v1
kind: ServiceAccount
metadata:
 name: random-yaml
 namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
 name: random-yaml:system:auth-delegator
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: ClusterRole
 name: system:auth-delegator
subjects:
- kind: ServiceAccount
 name: random-yaml
 namespace: default

Create the Stack

Let’s apply a Stack object to deploy the program. Note that the specification makes reference to the GitRepository (pulumi-examples), the Secret (pulumi-api-secret), and the ServiceAccount (random-yaml) created earlier.

apiVersion: pulumi.com/v1
kind: Stack
metadata:
 name: random-yaml
 namespace: default
spec:
 serviceAccountName: random-yaml
 fluxSource:
 sourceRef:
 apiVersion: source.toolkit.fluxcd.io/v1
 kind: GitRepository
 name: pulumi-examples
 dir: random-yaml/
 stack: dev
 refresh: true
 continueResyncOnCommitMatch: true
 resyncFrequencySeconds: 300
 destroyOnFinalize: true
 envRefs:
 PULUMI_ACCESS_TOKEN:
 type: Secret
 secret:
 name: pulumi-api-secret
 key: accessToken

This stack is configured to deploy immediately and to resync every five minutes. The term “resync” means to execute pulumi up, which is useful for stacks that produce resources or outputs that require a periodic update. Without resync, pulumi up runs whenever a new commit is pushed to the repository.

Observe the Workspace

The system automatically creates a Workspace object named random-yaml to provision an execution environment. Let’s watch as the workspace’s status progresses towards readiness. For this demonstration, we’ll use the --watch flag to observe the status updates:

$ kubectl get workspace --watch
NAME IMAGE READY ADDRESS
random-yaml pulumi/pulumi:3.134.1-nonroot
random-yaml pulumi/pulumi:3.134.1-nonroot False
random-yaml pulumi/pulumi:3.134.1-nonroot False random-yaml-workspace.default.svc.cluster.local:50051
random-yaml pulumi/pulumi:3.134.1-nonroot False random-yaml-workspace.default.svc.cluster.local:50051
random-yaml pulumi/pulumi:3.134.1-nonroot True random-yaml-workspace.default.svc.cluster.local:50051

Observe the Updates

Once the workspace is ready, the system proceeds to run a periodic update. Each update is represented by an Update object. Let’s watch the progression across three updates:

$ kubectl get update --watch
NAME WORKSPACE PROGRESSING FAILED COMPLETE URL
random-yaml-5kmn7dcm random-yaml
random-yaml-5kmn7dcm random-yaml True False False
random-yaml-5kmn7dcm random-yaml False False True https://app.pulumi.com/eron-pulumi-corp/random/dev/updates/1
random-yaml-rftp8gj8 random-yaml
random-yaml-rftp8gj8 random-yaml True False False
random-yaml-rftp8gj8 random-yaml False False True https://app.pulumi.com/eron-pulumi-corp/random/dev/updates/2
random-yaml-sjpw5hm8 random-yaml
random-yaml-sjpw5hm8 random-yaml True False False
random-yaml-sjpw5hm8 random-yaml False False True https://app.pulumi.com/eron-pulumi-corp/random/dev/updates/3

Observe the Stack

The Stack object maintains a status block showing the result of the latest attempt and whether an update is currently running. Taking a look, we see a successful outcome.

$ kubectl get stack random-yaml -oyaml

apiVersion: pulumi.com/v1
kind: Stack
...
status:
 conditions:
 - lastTransitionTime: "2024-10-17T20:41:49Z"
 message: the stack has been processed and is up to date
 reason: ProcessingCompleted
 status: "True"
 type: Ready
 lastUpdate:
 failures: 0
 generation: 2
 lastAttemptedCommit: sha256:44d554df090dcdeb9bae908cd38155c8c93db05f64dc72982027e8e1294af0d3
 lastResyncTime: "2024-10-17T20:41:49Z"
 lastSuccessfulCommit: sha256:44d554df090dcdeb9bae908cd38155c8c93db05f64dc72982027e8e1294af0d3
 name: random-yaml-sjpw5hm8
 permalink: https://app.pulumi.com/eron-pulumi-corp/random/dev/updates/3
 state: succeeded
 type: up
 observedGeneration: 2
 outputs:
 password: '[secret]'

Delete the Stack

To complete the demonstration, let’s delete the Stack object. Because spec.destroyOnFinalize is enabled, deleting the object causes the Pulumi stack to be destroyed. That is, the stack resources are deleted.

$ kubectl delete stack random-yaml
stack.pulumi.com "random-yaml" deleted

$ kubectl get workspace
No resources found in default namespace.

$ kubectl get update
No resources found in default namespace.

Run Pulumi inside your cluster

Use the Pulumi Kubernetes Operator to deploy and manage cloud infrastructure with the Stack custom resource, driven by the Git changes your cluster already watches.

Get started

Migration

Ready to leverage these new features? Let’s quickly cover how to migrate your existing setup to Pulumi Kubernetes Operator 2.0.

The Stack custom resource continues to be the primary way to use the Pulumi Kubernetes Operator, and your existing stack definitions should continue to work after you follow a simple migration procedure:

Create a ServiceAccount and associated ClusterRoleBinding as shown in the Security section.
If your stack uses the Pulumi Kubernetes Provider, create a RoleBinding or a ClusterRoleBinding to grant the necessary permissions, e.g. to the well-known ClusterRole named cluster-admin. See “Using RBAC Authorization” for more information.
Edit your Stack manifest(s) to use the service account, by setting the spec.serviceAccountName field.
Double-check that your Stack objects do not make use of cross-namespace references.

If you’re using a private Git repository as your program source, use the Stack’s secretRefs field to reference Git credentials. SSH-based access to the repository may require a custom image with your host’s public key(s) baked in. Contact us to learn more.

If you’re using Flux integration, there’s an additional step to be performed by your cluster administrator. Please create a NetworkPolicy to allow the workspace pods to access the Flux source controller. Here’s the manifest to be applied:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
 name: allow-pulumi-fetch-flux-artifacts
 namespace: flux-system
spec:
 podSelector:
 matchLabels:
 app: source-controller
 ingress:
 - ports:
 - protocol: TCP
 port: http
 from:
 - namespaceSelector:
 matchLabels:
 kubernetes.io/metadata.name: default
 - podSelector:
 matchLabels:
 app.kubernetes.io/managed-by: pulumi-kubernetes-operator
 app.kubernetes.io/name: pulumi
 app.kubernetes.io/component: workspace
 policyTypes:
 - Ingress

Conclusion

The Pulumi Kubernetes Operator is a great way to deploy and manage cloud infrastructure from within your Kubernetes environment. With the 2.0 release, we’ve solved the scalability and isolation limitations that affected early adopters. You can get started with the Pulumi Kubernetes Operator today. Please join us on the #kubernetes channel of the Pulumi Community Slack workspace to give feedback, and enjoy!

Pulumi+Kubernetes: New Flux Integration and Inline Programs

Luke Hoban — Sun, 23 Oct 2022 00:00:00 +0000

Pulumi’s Universal Infrastructure as Code platform works with all major clouds and over 100 cloud and SaaS providers, but among all its uses one of the most important is the ability to bring rich Infrastructure as Code tools and practices to Kubernetes projects and teams.

Kubernetes is one of the most used platforms in Pulumi, second only to AWS, with thousands of organizations using Pulumi to manage clusters at scale. Pulumi supports a wide variety of use cases around Kubernetes - from cluster creation and management, to rich and expressive workload definition, to continuous delivery and infrastructure GitOps.

Check out version 2.0 of the Pulumi Kubernetes Operator.

Today we are announcing a set of major updates which deepen and extend Pulumi’s support for Kubernetes and the Kubernetes ecosystem. In this post, we’ll highlight a few of these exciting enhancements:

Pulumi Kubernetes Operator v1.10: New integration with Flux for richer GitOps support, and ability to deploy Pulumi stacks from directly within the Kubernetes resource model
New Pulumi Provider for Flux: Manage Flux with Infrastructure as Code
Pulumi Kubernetes Provider v3.22: Server Side Apply and Resource Patch

You can learn more about Pulumi and Kubernetes or jump in and get started right away, or read on to learn more about what’s new for Pulumi and Kubernetes!

Pulumi Kubernetes Operator v1.10: Flux Integration and Inline Infrastructure support

The Pulumi Kubernetes Operator provides a cloud-native way to manage and deploy cloud infrastructure using Pulumi from within your Kubernetes environment. Infrastructure as Code defined in Pulumi can be automatically deployed using a GitOps model driven from within a Kubernetes environment - leveraging existing Kubernetes-based orchestration, RBAC, policy and resource model investments. We released Pulumi Kubernetes Operator 1.0 last year, and since then have seen users scale up usage of the Operator to drive hundreds of thousands of Pulumi infrastructure updates.

Pulumi Kubernetes Operator v1.10 expands the ways that Pulumi infrastructure can be defined. In addition to the existing Git-based source that has been supported to date, this release adds two new ways to define your infrastructure:

Integration with Flux Sources - enabling any Flux Source to provide the Pulumi program that defines your cloud infrastructure.
Support for Inline Infrastructure sources - a new pulumi.com/v1/Program Custom Resource that can be used to describe cloud infrastructure directly within the Kubernetes resource model.

Together, these open up a wide variety of new ways to apply the Kubernetes Operator as an interface for defining and delivering cloud infrastructure.

Integration with Flux Sources

We’re excited to work with Flux, a CNCF project, to bring even richer GitOps support to the Pulumi Kubernetes Operator. While the Pulumi Kubernetes Operator offers basic GitOps and branch tracking support, it supports only a fraction of the features that Flux’s SourceController provides. Instead of rebuilding all of these features, we chose to provide rich integration with Flux Sources, enabling Pulumi programs to be provided via Flux, and then handed off to drive infrastructure deployments using the Pulumi Kubernetes Operator Stack resource.

Using Flux offers a variety of important new features to Pulumi Kubernetes Operator users:

Support for Flux’s Notification Controller and webhooks to drive highly responsive updates when new commits are pushed
Access to sources other than Git repositories: S3 buckets and OCI repositories
Support for platforms like Azure DevOps that require specific Git implementations
Sources and Stacks can be defined by different teams, since they are separate resources which can have independent RBAC rules applied.
Many advanced Git features including: following semver ranges, excluding files from source, recursively cloning submodules, verifying revision signatures, and more.

To use the Pulumi Kubernetes Operator with Flux, install each of the necessary controllers for Flux and the Pulumi Kubernetes Operator, and then the necessary Role and RoleBindings to allow the Pulumi Kubernetes Operator to read from Flux sources.

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
 name: flux-source-access
rules:
- apiGroups:
 - source.toolkit.fluxcd.io
 resources:
 - "*"
 verbs:
 - get
 - list
 - watch
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
 name: pulumi-flux-source-access
subjects:
- kind: ServiceAccount
 name: pulumi-kubernetes-operator
roleRef:
 kind: Role
 name: flux-source-access
 apiGroup: rbac.authorization.k8s.io

Once those pre-requisites have been met, simply deploy the Flux Source (in this case a GitRepository) and Pulumi Stack resource that define your infrastructure delivery:

---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: GitRepository
metadata:
 name: pkodev
spec:
 interval: 30s
 ref:
 branch: main
 url: https://github.com/lukehoban/pko-dev
---
apiVersion: pulumi.com/v1
kind: Stack
metadata:
 name: pkodev
spec:
 stack: lukehoban/pkodev/test
 fluxSource:
 sourceRef:
 apiVersion: source.toolkit.fluxcd.io/v1beta2
 kind: GitRepository
 name: pkodev
 dir: basic/
 destroyOnFinalize: true

Note that by using the new fluxSource property on a pulumi.com/v1/Stack resource, you can configure it to use the corresponding source instead of an inline definition of the target Git repo and branch. The GitRepository could be replaced with any other Flux Source. For example, to pull our source code from an S3 bucket, use this instead:

---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: Bucket
metadata:
 name: pkodev
spec:
 interval: 5m0s
 provider: aws
 bucketName: podinfo
 endpoint: s3.amazonaws.com
 region: us-east-1
 timeout: 30s
---
apiVersion: pulumi.com/v1
kind: Stack
metadata:
 name: pkodev
spec:
 stack: lukehoban/pkodev/test
 fluxSource:
 sourceRef:
 apiVersion: source.toolkit.fluxcd.io/v1beta2
 kind: Bucket
 name: pkodev
 dir: basic/
 destroyOnFinalize: true

If we are using the GitRepository source along with GitHub, we can use Flux features like WebHook Receivers to expose a webhook that GitHub can trigger on pushes so that updates will be pushed into Flux and the Pulumi Kubernetes Operator directly (instead of waiting to pull them):

---
apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Receiver
metadata:
 name: webapp
 namespace: flux-system
spec:
 type: github
 events:
 - "ping"
 - "push"
 secretRef:
 name: webhook-token
 resources:
 - kind: GitRepository
 name: pkodev

For users who need any of these additional GitOps capabilities, you can now use the best of Pulumi and the best of Flux together with ease.

This integration is made possible by the excellent “open and extensible” GitOps toolkit that Flux v2 provides. We’re excited to work with the Flux community and the team at Weaveworks to help end-users accelerate their GitOps projects:

“When we donated Flux to the CNCF we wanted to empower teams to easily build GitOps automations with the best tools. We’re excited to welcome Pulumi to the Flux and Weave GitOps ecosystem.”

– Alexis Richardson, CEO – Weaveworks

Define Pulumi Cloud Infrastructure Directly in the Kubernetes Resource Model

One of the important benefits of the Pulumi Kubernetes Operator is that it provides the Kubernetes resource model as an interface for defining and managing cloud infrastructure expressed via Pulumi. However, to date, the cloud infrastructure definition itself still has to be stored in Git (or, with the Flux integration above, an S3 bucket or OCI repository). Many users have asked for a model where the Pulumi cloud infrastructure definition itself is provided directly within the Kubernetes resource model as well. With the new pulumi.com/v1/Program custom resource, this is now possible!

This support is enabled by Pulumi’s introduction of Pulumi YAML - the ability to describe cloud infrastructure with access to the entire Pulumi ecosystem of cloud providers and components via simple YAML programs. Importantly, those YAML programs can now be defined directly inside the Kubernetes cluster itself.

Because Pulumi offers access to over 100 cloud providers and components, with thousands of infrastructure resources available across them, the pulumi.com/v1/Program resource offers access to defining rich and complex cloud infrastructure directly from within Kubernetes. And because Pulumi YAML infrastructure definitions support all the same reliable dependency tracking and rich IaC features as any other Pulumi programs, users get all of these benefits from within Kubernetes as well.

To deploy cloud infrastructure directly from within Kubernetes, install the Pulumi Kubernetes Operator and then deploy the Program and Stack resources below with kubectl apply (or via a Pulumi program!):

apiVersion: pulumi.com/v1
kind: Program
metadata:
 name: staticwebsite
program:
 resources:
 bucket:
 type: aws:s3:Bucket
 properties:
 website:
 indexDocument: index.html
 index.html:
 type: aws:s3:BucketObject
 properties:
 bucket: ${bucket.id}
 content: <h1>Hello, world!</h1>
 contentType: text/html
 acl: public-read
 outputs:
 url: http://${bucket.websiteEndpoint}
---
apiVersion: pulumi.com/v1
kind: Stack
metadata:
 name: staticwebsite
spec:
 stack: lukehoban/staticwebsite/test
 programRef:
 name: staticwebsite
 destroyOnFinalize: true
 config:
 aws:region: us-east-1

The result will be to deploy an S3 Bucket and BucketObject exposing a static website into AWS. The Program resource defines a Pulumi program written in Pulumi YAML using any cloud provider resources Pulumi supports, ready to be deployed into as many Pulumi stacks as needed. The program declares an S3 Bucket with website configured, and an index.html file in a BucketObject which will be served from that website. The Stack resource deploys an instance of this program, using the configuration (us-east-1 region) specified.

You can access the URL of the deployed website defined as a program output url via:

$ kubectl get stack staticwebsite -o jsonpath={.status.outputs}

Of course, this could be used to define any cloud infrastructure - an AWS RDS Database, an Azure KeyVault, a CloudFlare DNS Record, a Google Cloud Function, or even another Kubernetes Cluster via EKS, AKS, GKE, Oracle Cloud, Civo, Digital Ocean and more.

Updating the Program resource will immediately drive an update to the associated Pulumi Stack, incrementally updating the managed infrastructure and maintaining the desired cloud infrastructure state.

New Pulumi Provider for Flux

In addition to Flux integration with the Pulumi Kubernetes Operator, we’ve also added a new Flux Provider to the Pulumi Registry. This provider, built by Pulumi community member @oun, enables managing Flux directly via Pulumi Infrastructure as Code.

The Flux provider can be used along with the Kubernetes provider and GitHub provider to stand up and manage a complete E2E GitOps solution from within a single deployment - no need for manual bash, Helm, or runbooks - just simple reliable Infrastructure as Code.

import * as kubernetes from "@pulumi/kubernetes";
import * as flux from "@worawat/flux";

// Get the manifests needed to install Flux with the given components and policies
const fluxManifests = flux.getFluxInstallOutput({
 targetPath: "local",
 components: ['source-controller'],
 networkPolicy: true,
});

// Deploy the manifests into our cluster
const fluxGroup = fluxManifests.content.apply(fluxYaml => {
 new kubernetes.yaml.ConfigGroup("flux-install", {
 yaml: fluxYaml,
 });
});

Pulumi Kubernetes Provider v3.22: Server Side Apply and Resource Patch

In July, we released preview support for Server-Side Apply in the Pulumi Kubernetes provider. We highlighted the benefits and needs that led to adding Server-Side Apply support:

We first launched the Pulumi Kubernetes provider in 2018, and it has grown to be one of the most used providers across the Pulumi ecosystem. Many of our customers rely on this provider to manage critical production workloads, and a common request was for an easier way to manage shared Kubernetes resources with Pulumi. We’ve been listening, and are grateful to everyone who took the time to share their feedback with us!

A lot has changed since 2018, and our provider continues to evolve to support new use cases and the ever-growing richness of the Kubernetes ecosystem. One such development is Server-Side Apply (SSA), which is a resource management strategy that was introduced in Kubernetes v1.18. Kubernetes clients that use SSA can safely share the management of Kubernetes resources by making the API Server responsible for computing diffs and resolving conflicts.

We have kept a close eye on the SSA feature since it was announced and have been waiting patiently for its broad availability in production clusters. The wait is over, and we are excited to bring the power of SSA to Pulumi users!

With the most recent v3.22 release of the Kubernetes provider, Server-Side Apply support is now out of preview and enabled by default for all Kubernetes resources.

This makes several important new features available across the breadth of the Kubernetes provider:

New Patch resource types corresponding to every Kubernetes resource kind. (e.g., NamespacePatch, DeploymentPatch)
“Upsert” support; create a resource if it does not exist, or update the existing resource with specified changes.
Diffs no longer depend on the last-applied-configuration annotation, which fixes a number of subtle issues that users of the provider previously could run into.

These features bring Pulumi and Kubernetes even closer together, enabling Pulumi’s rich Infrastructure as Code to fit even more naturally into the Kubernetes desired state and shared ownership model.

Check out the Managing Resources with Server Side Apply article for more details on these new features.

Conclusion

At Pulumi, we are passionate about pushing the state of the art of Infrastructure as Code forward, and applying modern Infrastructure as Code to support Kubernetes projects and teams is a critical component of that vision. All of these new features around Pulumi and Kubernetes help to further empower teams to build scalable and robust solutions around Kubernetes and Pulumi IaC.

We’re so happy to be able to work with the Flux project to bring all of the benefits of its composable GitOps toolkit to Pulumi users. We’re excited about the new use cases that Inline Infrastructure support in the Pulumi Kubernetes Operator enables. And we’re thrilled to bring Server Side Apply and Patch support to the Pulumi Kubernetes Provider.

You can learn more about Pulumi and Kubernetes or jump in right away by following the getting started guide.

Patterns for Drift Detection with Pulumi

David Flanagan — Thu, 16 Jun 2022 09:26:46 -0500

Portions of this blog post are out of date. See the Pulumi Deployments drift detection blog post for an updated overview and examples.

Let’s face it, at some point someone is going to modify your carefully-crafted and automated infrastructure without updating your Pulumi program. These changes cause the desired state of our Pulumi program’s to be inconsistent with the state of the world. These inconsistencies are often referred to as “drift”. In this article, I want to cover a couple of patterns for detecting and reconciling this drift with your Pulumi programs.

Why is Drift Detection Important?

Reconciling drift as quickly as possible is very important in the infrastructure as code (IaC) space because the longer drift is left unattended, the harder it becomes for automation to reconcile in a manner that can be considered safe for your organization, SLAs, and customers happiness. Imagine this as if it were your car. If a single tyre needs replaced, you might be OK for a while and not notice any problems driving. If you leave that unattended and another tyre runs flat, then you’ve got a slightly more serious problem. As these smaller problems accumulate you’re applying a multiplier of risk to your overall infrastructure, especially during reconciliation. The more resources Pulumi needs to handle during this step, the more likely for some collateral damage.

It’s important that you build a system that can tolerate a nominal amount of drift but actively works to repair that as quickly as possible.

Let’s see how.

Refreshing the World

When you run a pulumi up, Pulumi will create a directed-acyclic graph (DAG) from your Pulumi program. This graph is then compared to your state file, and any changes that need to happen to have the DAG match the state file are applied. Pulumi does not reach out to “the world” to detect if what is in your state file is actually still up to date, at-least not without adding the --refresh option.

That’s the secret sauce to ensuring that we can begin to detect and reconcile drift. Of course, this comes with a couple of caveats that you need to be aware of before adopting --refresh as your default behavior.

Firstly, it’s slower. The --refresh option will take longer because the Pulumi program needs to reach out to all of the providers and fetch information on the resources in your state file.

Secondly, the refresh will sync potentially “untracked” fields within your resources and could trigger more updates to your resources than you’ve had previously. This isn’t a bad thing and Pulumi’s defaulting will hopefully ensure that resources aren’t needlessly updated, but it’s definitely worth keeping in mind.

With that being said, let’s take a look at two common strategies for updating Pulumi programs.

Workflow Approach

The most common approach is to use your CI/CD / build system to run Pulumi whenever your source code changes. This works quite well, but it means that any drift isn’t detected and reconciled until someone pushes some changes. If nobody commits to the repository for 2 days, a week, or even longer - then a few problems can arise. The longer the gap, the more chance of drift causing more problems during the next run. Drift can cause resources to be recreated, which may be unexpected, and if left long enough, the “drift” might become expected behavior within your system and the next reconcile becomes a “break”.

When using the workflow approach for your Pulumi program’s, I highly encourage you to schedule builds as frequently as you can. Whether that’s hourly or daily, the more you run your Pulumi and handle drift, the easier and more comfortable you’ll be with the automation.

If you want to do this with GitHub Actions, you can use the cron settings on your workflow. This is an example of my own production infrastructure:

name: infrastructure

on:
 workflow_dispatch: {}
 push:
 branches:
 - main
 schedule:
 - cron: "6 6 * * *"

CircleCI also makes this rather easy too:

 daily-run-workflow:
 triggers:
 - schedule:
 cron: "6 6 * * *"
 filters:
 branches:
 only:
 - main

Operator Approach

Now, scheduled executions are great, but if you want to take that to the next level - using the Pulumi Kubernetes Operator is the next step. The operator can be configured to run every few minutes, ensuring that your world is always consistent with your Pulumi program. Here’s an example YAML representation of a Pulumi stack:

apiVersion: pulumi.com/v1
kind: Stack
metadata:
 name: some-repo-production
spec:
 stack: production
 projectRepo: https://github.com/some/repo
 branch: "refs/heads/main"
 destroyOnFinalize: true
 continueResyncOnCommitMatch: true
 refresh: true
 resyncFrequencySeconds: 60

The last three settings are the most important part of this configuration. Unfortunately, these are not the defaults - but there’s an issue tracking this change.

continueResyncOnCommitMatch is a setting that tells Pulumi to run even though the code itself hasn’t changed, which when used with refresh: true means we’ll always update the state of the world and reconcile any drift. resyncFreqeuncySeconds is the number of seconds between each reconcile.

Making Refresh Your Default

Before you go updating all your workflow files, or adopting the operator, for all the benefits of continuous reconciliation and drift detection, let’s cover a few safety tips.

Always run a pulumi up --refresh locally / manually first and inspect the changes. Pulumi may wish to replace some resources and if you don’t want that to happen - you need to prepare those resources for your new continuous reconciliation.

Protect

If you have a resource that should NEVER be deleted, make sure you protect it. You can do this by adding a ResourceOption to the resource in your Pulumi program.

{
 protect: true
}

opts=ResourceOptions(protect=True)

pulumi.Protect(true)

new CustomResourceOptions { Protect = true }

CustomResourceOptions.builder()
 .protect(true)

options:
 protect: true

Ignore Changes

If Pulumi wants to replace a resource that you don’t believe needs to be updated, you can ask Pulumi to ignoreChanges on certain properties. Of course, it might be best to have your code match those properties, but in the event that you can’t - you can use the code below to ignore:

{
 ignoreChanges: ["prop"]
}

opts=ResourceOptions(ignore_changes=["prop"])

pulumi.IgnoreChanges([]string{"prop"})

new CustomResourceOptions { IgnoreChanges = { "prop" } }

CustomResourceOptions.builder()
 .ignoreChanges("prop")

options:
 ignoreChanges:
 - prop

Delete Before Replace

If you have any resources that don’t use auto-naming, you’ll also need to ensure they have the deleteBeforeReplace option enabled.

{
 deleteBeforeReplace: true
}

opts=ResourceOptions(delete_before_replace=["prop"])

pulumi.DeleteBeforeReplace(true)

new CustomResourceOptions { DeleteBeforeReplace = true }

CustomResourceOptions.builder()
 .deleteBeforeReplace(true)

options:
 deleteBeforeReplace: true

Once you’ve run a couple of pulumi up --refresh and you’re comfortable with the proposed executions, you can adopt the automation and sleep easier, knowing that you’re handling drift and keeping your infrastructure up to date.

Want to See More?

Watch me demo some of these features live, on YouTube, on my PulumiTV series: Modern Infrastructure.

Upgrade Strategies: An Introduction for IaC

Laura Santamaria — Mon, 07 Mar 2022 09:59:56 -0600

When you’re working with infrastructure, you’re inevitably going to need to upgrade or update that infrastructure. Whether it’s an operating system update or a desire to get CPU or memory upgrades, you will need the ability to pick resources and change them as necessary. In the past, this kind of upgrade would be done on the basis of individual resources, with each one being updated and checked either by hand or programmatically before moving onto the next resource. If you’ve ever done a database migration or if you ever did the recommended way of upgrading your computer’s operating system including all of the backup steps, you’re familiar with this process. Stand up the new resource. Check everything works. Move over the data. Check again. Tear down the old infrastructure. In a cloud computing environment, though, you’re often dealing with hundreds or thousands of resources, and doing one-by-one replacement is a nightmare that takes ages. However, there are other options, many borrowed from the application deployment world, that we have available to us because we write infrastructure as code.

Generally, there’s a few strategies for replacing something in a cloud computing environment. You’ll often hear about them as deployment strategies. These strategies generally differ based on the order of operations: Do you create a new resource before you delete the old one? Do you replace some and pause to gather data? Do you YOLO and just toss everything out and build new? All of these strategies are worth considering depending on the needs of your situation. A production system likely needs to maintain uptime, or the amount of time a system is available to the end user, to meet service-level agreements (SLAs), which might include an availability promise such as a promise of 5 nines (99.999%) of uptime. In that case, a YOLO strategy won’t be very acceptable, will it?

As many of these deployment strategies initially came from the application world, we can’t just use the same deployment strategies exactly as they appear for an application because there’s a bit more going on under the hood. We need to consider what we’re building and where we are on our stack. If we have multiple instances of our application running on identical containers, for example, we can treat the containers a bit differently than, say, the single load balancer in front of all of the containers or the node underneath a single-node Kubernetes cluster.

To illustrate all of the following deployment scenarios, let’s imagine we have a system where an application is deployed on some infrastructure. What the application is doesn’t really matter; we’re going to focus on the infrastructure here. We’ll pick a simple situation: There’s a security update for one of your pieces of infrastructure.

Big Bang

Let’s say the infrastructure you’re working with is a large number of virtual machines (VMs) that are running an outdated, insecure version of an operating system. If you weren’t concerned about uptime or the stored data on those systems, you could certainly just tear them down and stand up new ones—this situation could reflect that perhaps you’re working with a development environment that is ephemeral and isn’t in use right now, or you don’t have any load just yet. Some folks refer to this kind of upgrade strategy as a “big bang” strategy, though folks who run data centers would likely say that the big bang strategy still at least required testing and careful consideration of data transitions and you would rarely completely wipe the original system before replacing it with another. If you’re wondering where this upgrade strategy came from, it was originally used in non-cloud-native systems where you likely didn’t have the hardware available to run other deployment strategies. In these cases, you would have maintenance windows and change processes, and the whole process was mapped out well ahead of time. This kind of fast switchover, especially with little to no testing, really is not a good idea for a cloud-native or cloud-based production environment, and it’s frowned upon if you’re working with a cloud-based environment that others are also using. There’s so many better options than scheduling maintenance windows and ripping apart systems when you have the capability to stand up parallel virtual hardware.

Blue-Green Deployments

Now, let’s say that instead of tearing down and then standing up new VMs, you instead were to create an almost identical environment with the new operating system, test it under load, and then transition your traffic over and wait for success before tearing down the old version. This is known as a blue-green deployment, and it’s a fairly popular strategy. A blue-green deployment is, in short, a create-check-delete process. You may have heard this strategy called “red/black deployment” or “a/b deployment.” The rationale behind using “blue” and “green” as names originally comes from needing easy names that didn’t have any kind of connotation of one group of systems being “better” than the other (e.g., “red” is the same color as alert lights, so don’t you want to keep the “black” deployment?). So you certainly can name it whatever you’d like in your company, but the concept is the same. The “blue” environment is currently running. To upgrade, you stand up an almost identical environment with the upgrade you want to perform. That environment is your “green” environment. Then, you run any test that you can against that green environment to ensure it’s ready. Finally, you switch your traffic over to the green system and monitor for any issues. Once you’re sure the environment is stable and can handle the load of your normal traffic (typically measured in hours to days depending on when you made the switch and what your traffic patterns demonstrate), you tear down the blue environment.

Note that, while a blue-green deployment traditionally is standing up and tearing down entire systems, you don’t have to do an entire system if you can do a subsystem instead so long as that subsystem is self-contained. The idea behind the rollout strategy is the same: You use a load balancer to transition traffic from one complete system to another. Since we’re talking infrastructure, the more traditional version needs to be modified by thinking of subsystems. For example, if you’re replacing the load balancer as well, you would stand up the green load balancer, point it at the rest of the blue deployment, make the switch of the traffic to the green load balancer, and then complete the move from the rest of the blue deployment to the green deployment. Then, finally, the blue load balancer and the rest of the blue deployment can be decommissioned.

Canary Deployments

Now, a blue-green deployment isn’t the only upgrade strategy, and it’s not always the best one for a cloud-native system. There’s also a strategy called a canary deployment. In a canary deployment, you stand up new infrastructure and move traffic over in small increments, such as 5% of overall traffic. That small slice of traffic is considered a “canary,” an indicator of failure or success of such a move. While it’s often used in application deployment as a way to gather user feedback on an application change, this strategy is also useful for infrastructure as it highlights issues that only occur under true, random load before they become a problem for a full deployment. This strategy is fairly popular in microservices architectures and with platforms like Kubernetes as it’s much easier to implement than with more traditional, VM-based systems.

Rolling Deployments

The final upgrade strategy we’ll consider is called a rolling deployment. In a rolling deployment, each element of a system is replaced one at a time, with each new instance being checked before decommissioning and removing the old instance. That check is called a health check on some platforms, and the basic idea is that the deployment tool sends a request to the new element and waits for a response that indicates the system is functional and responsive as expected. Once the health check on the new element clears, the old one is removed, and the deployment tool “rolls” to the next element in the system being updated. We find this one in many Kubernetes-based applications, and it can work well for cloud-native infrastructure, as well.

In the next article in this series, we’ll explore how these three strategies are similar and different for infrastructure as code, and why you would use one over the other. Stay tuned!

In the next parts of the series, we’ll try these deployment strategies with Pulumi, exploring how to use code to define each kind with a test system. Watch this space!

Meanwhile, while you’re waiting, we did a few videos on this topic over at PulumiTV, like this one on blue-green deployments with Pulumi and Python on GCP.

Six Things You Might Not Know About the Pulumi Service

Chris Smith — Mon, 24 Jan 2022 00:00:00 +0000

As a reader of this blog, you’ve probably heard of the Pulumi Service, the default state-management backend of the Pulumi CLI. If that’s the case, there’s also a good chance you’ve heard of a number of the Service’s key features, like helping you organize your projects and stacks, collaborate with others with the help of organizations, or handle sensitive data securely with built-in support for encrypted secrets.

What you might not know, though, is that we’re adding new features to the Pulumi Service all the time, and that some of these features can be fairly easy to miss. So in this post, we’ll highlight a handful of the features you might not be aware of, and that we think make it even easier to manage your infrastructure with Pulumi.

Let’s get started!

Feature 1: Stack Tags

The first thing we’ll cover is the Pulumi stack tags feature. Stack tags are key/value pairs that are associated with your Pulumi stack. You can use them for any number of purposes, from keeping things organized to even flagging whether or not it’s okay for a script to reclaim the stack’s resources.

While you can create, edit, and remove stack tags using the Pulumi CLI (see pulumi stack tag), you can also do so in the Pulumi Console:

Once set, you can quickly filter stacks on your organization’s Projects tab as well.

Feature 2: Link Updates to your CI/CD pipeline

Another helpful thing the Pulumi Service can do is link the Pulumi stack update to the CI/CD job or run that was used to perform it, as well as to the specific source commit:

When Pulumi updates a stack, it will store some information about the local machine state, such the current git SHA if the stack resides in a git repository. It is with this data that the Pulumi Service links to relevant services where possible.

If you’re curious to know all of the information Pulumi has for a stack update, you can navigate to the Environment tab for that update:

Feature 3: Custom Update Messages

One of the most useful pieces of additional data stored with each update is the message property. By default, Pulumi will use whatever the latest git commit message is. But did you know you can customize that message from the command line?

By passing the --message (or -m) flag to pulumi up, you can provide a custom update message, which can help explain the context for why you made a particular infrastructure change:

$ pulumi up --message "Release the hounds\!"

Feature 4: Viewing Resource Changes

Another thing you might have missed is that the Pulumi Service supports multiple views for a stack update’s logs.

The standard view mimics the Pulumi CLI’s output. However, you can switch to the Diff or Diagnostic views, too.

The Diff view displays all of the properties of modified resources, along with their previous/updated values. This can be helpful when trying to understand exactly what happened for a stack update.

The Diagnostic view just outputs so-called “diagnostic” messages from the Pulumi update. Most commonly, this is where you will see the output of things like calls to JavaScript’s console.log(...) or Python’s print(...) functions. It’ll also include the output of dependent tools, like the output when building Docker containers.

Feature 5: Stack Resource Visualization

Another cool feature of the Pulumi Service is being able to visualize a stack’s resources and their relationships to one another. This is a really important feature!

One of the advantages of using an infrastructure as code tool like Pulumi is that you aren’t just creating a flat list of resources. Instead, by the virtue of how the code and resource graph is structured, you are building a graph of resources.

This means that Pulumi has a semantic understanding of not only what resources are in your stack but also how they relate to one another.

If you navigate to a stack’s Resources tab, by default you see them listed in a table. You can filter this list by resource type, or by providing a custom filter:

However, in order to see the expanded resource graph, click the Graph View button. This will switch to rendering the resources as an interactive graph.

Notice here that the resources that are part of a component are grouped together. Also, you can double click on those nodes in the graph to expand or collapse them!

Feature 6: Drilling into Resource Properties

But you don’t need to just settle for a “top-level view” of your resources. You can click on any specific resource on the Resources tab, and see all of the properties of that resource.

This allows you to see all of the properties that Pulumi is tracking for that resource. It will also show you any relationships to other resources in the same stack.

The Pulumi Service also links you to the resource’s cloud provider’s console where possible, so you can see even more details.

Is your resource missing a link to the cloud provider console? With new services being launched from the myriad providers Pulumi supports every day, it’s hard to keep up. So if you would like us to add cloud provider links, please file an issue over in the pulumi/console-requests GitHub repository so we can update our database.

There are plenty of other features to check out in the Pulumi Service, and we have exciting improvements on the way! As always, feel free to stop by the Pulumi Community Slack to learn more, ask questions, or share anything cool you’re up to!

Improving the GitOps Pipeline with the Pulumi Operator

David Flanagan — Fri, 24 Dec 2021 17:08:06 +0000

Check out version 2.0 of the Pulumi Kubernetes Operator.

This time last year, I presented Applying the Law of Demeter to GitOps at GitOps Days 2020. The Law of Demeter is a design principle, proposed in 1988, which encourages loose coupling between systems. During this session, I wanted the audience to understand and be able to identify when their applications and continuous delivery pipelines have too much knowledge of the platform in which they’re going to run. As an industry, we’re seeing a great deal of momentum towards Platform Engineering and with this comes a Broca divide, a strict division of responsibilities: to build a platform and to consume a platform.

Instead, the platform will provide everything the application needs to function. We’ve seen a lot of this before, notably via the 12-factor manifesto.

Over the course of the last year I’ve collected my thinking more and more around this pattern, and I condensed it down into a single tweet:

#GitOps isn’t a decision of push vs pull for continuous delivery. It’s more deciding whether your application tells the environment how to deploy itself, or whether the environment tells your application how to run.

You want the latter. You want GitOps.

So what does this all mean in a more practical sense?

Ingress Example

Time and time again I see the same violation of the Law of Demeter with Kubernetes and GitOps: Ingress.

Application delivery teams require Ingress resources within Kubernetes to expose their applications to their customers. Often, Helm or Kustomize is used to provide bases that can be extended for each environment. We can see an example of the boundary violation in the Kustomize examples. I’m not calling out Kustomize here; Pulumi also has examples of this pattern.

Unfortunately, this pattern has become quite common. Leaking environment information down the stack is what we’re trying to avoid, but doing so can be rather tricky.

To understand this challenge in concrete terms, let’s look at an example of a simple Ingress resource:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: minimal-ingress
 annotations:
 nginx.ingress.kubernetes.io/rewrite-target: /
spec:
 rules:
 - host: staging.example.org
 http:
 paths:
 - path: /testpath
 pathType: Prefix
 backend:
 service:
 name: test
 port:
 number: 80

The violation here is host: staging.example.org inside of the application team resources. We now have a constraint binding the platform and the application together. If we continue down this path, the platform team can’t launch new environments without first reaching out to the application teams to ensure they have a base that can provide the correct domain name for these resources.

What we really want is for the platform team to be able to launch platforms whenever they want, increasing their ability to iterate and experiment with the platform. At the same time, by removing the bases and variance from the application teams, we ensure that they have a single blueprint for how to deploy their application. Win/Win.

Sadly, this is a difficult problem to solve; though not impossible.

Pulumi Operator

The Pulumi Operator hit major milestone in October this year, releasing version 1.0.

The Pulumi Operator allows us to leverage the power of Pulumi at the application team level for GitOps. This bond between Pulumi for building platforms and Pulumi for consuming platforms means we can have a strictly typed exchange of “knowns” between our platform and our application that happens at execution time.

Let’s imagine, in TypeScript, that we have the following versioned interface:

export interface Environment {
 name: string;
 baseDomain: string;
 serviceNamespace: string;
 ingressEnabled: boolean;
 networkPoliciesEnforced: boolean;
}

With this contract, the Platform Engineering team can ensure that every cluster, or namespace, has a ConfigMap called environment can be consumed by the Pulumi Operator. In this, the Pulumi program, the GitOps delivery pipeline from the application team, can begin to grok the environment to which it is being deployed. With this knowledge, it can begin to augment the manifests before they’re applied to the cluster.

Let’s go back to our Ingress example and see how this could be done.

First, let’s assume our Platform team have deployed a ConfigMap like such:

apiVersion: v1
kind: ConfigMap
metadata:
 name: environment
data:
 v1: |
 {
 "name": "production",
 "baseDomain": "pulumi.com",
 "serviceNamespace": "utils",
 "ingressEnabled": true,
 "networkPoliciesEnforced": true
 }

The specification for what an environment looks like will differ from organization to organization, but this hopefully provides enough context for some of the things you may wish to encapsulate this way.

Next, our application team defines their delivery via Pulumi, too. The first thing they do as part of the delivery is load the environment config. This data can then be used to augment any resources that need to be environment aware.

import * as k8s from "@pulumi/kubernetes";

const cm = k8s.core.v1.ConfigMap.get("environment", "environment", {});

export interface Environment {
 name: string;
 baseDomain: string;
 serviceNamespace: string;
 ingressEnabled: boolean;
 networkPoliciesEnforced: boolean;
}

const env = cm.data.apply((data): Environment => JSON.parse(data["v1"]));

const ingress = new k8s.networking.v1.Ingress(name,
 {
 metadata: {
 labels: labels,
 },
 spec: {
 rules: [
 {
 host: pulumi.interpolate`mysvc.${env.baseDomain}`,
 http: {
 paths: [
 {
 path: "/",
 backend: {
 serviceName: serviceName,
 servicePort: "http",
 }
 },
 ],
 },
 }
 ]
 }
 }
);

That’s it! We’ve got isolated delivery pipelines for both the platform team and the application team, leveraging GitOps and continuous reconciliation, without leaking any information across the stacks.

Pulumi is already seeing considerable adoption and success with platform teams, and I’m excited to see further adoption of the Pulumi Operator and application teams harnessing the power of GitOps with Pulumi.

Pulumi Kubernetes Operator 1.0

Luke Hoban — Tue, 12 Oct 2021 00:00:00 +0000

Check out version 2.0 of the Pulumi Kubernetes Operator.

Last year we released the Pulumi Kubernetes Operator, a new cloud-native way to manage and deploy cloud infrastructure using Pulumi from within your Kubernetes environment. Since then, we’ve worked with many Pulumi users who have adopted the Pulumi Kubernetes Operator at increasingly larger scales and for a wide variety of use cases. Today, we’re excited to make the 1.0 release of the Pulumi Kubernetes Operator available.

The Pulumi Kubernetes Operator defines a Kubernetes Custom Resource called pulumi.com/v1/Stack, which represents a Pulumi stack. The Pulumi stack can be authored in any supported Pulumi language (TypeScript, Python, Go, .NET) and can deploy and manage cloud infrastructure in any supported cloud (AWS, Azure, GCP, Kubernetes and 60+ additional cloud and SaaS providers). The Pulumi Kubernetes Operator triggers cloud deployments based on changes to the Stack Custom Resource or the resources it tracks.

As a result, the Pulumi Kubernetes Operator enables users to specify the desired state of their cloud infrastructure using resources managed directly in their Kubernetes cluster, which trigger creation, update and deletion of the detailed cloud infrastructure they need.

For example, the following Kubernetes resource can be created to deploy a Pulumi program which provisions cloud infrastructure to manage NGINX provided in the target Git repo and commit, and future updates to the commit or config settings will deploy updates to the infrastructure.

apiVersion: pulumi.com/v1
kind: Stack
metadata:
 name: nginx-k8s-stack-production
spec:
 stack: acmecorp/nginx/production
 projectRepo: https://github.com/acmecorp/pulumi-nginx
 commit: 2b0889718d3e63feeb6079ccd5e4488d8601e353

The Pulumi Kubernetes Operator can also be used along with Pulumi’s Kubernetes provider, which allows Pulumi programs to deploy resources to Kubernetes, either in the same cluster as the operator, or in another cluster. This can be used to package up complex Kubernetes infrastructure into more abstracted units, to mix Kubernetes and cloud infrastructure into a single unit of deployment, to define how application workloads are deployed into a cluster, or for a wide variety of additional use cases. The Pulumi Kubernetes provider has full support for the Kubernetes API available in every Pulumi language, along with newly improved support for Helm.

We’re at KubeCon North America 2021 this week - come meet us at booth SU8 if you are there! We also have two Kuberentes-focused webinars coming up where you can learn more about working with Pulumi and Kubernetes: Provision, Manage and Scale Kubernetes on Any Cloud (Oct 14th) and From Zero to Production in Kubernetes (Oct 21st).

What’s New in Pulumi Kubernetes Operator 1.0?

The 1.0 release includes many significant enhancements since the initial launch, as well as scalability, performance and quality improvements.

GitOps workflows with branch tracking and private Git repos

The operator now supports specifying a branch instead of a commit in a target Git repo. When a branch is specified, the operator will track any changes to what that branch is tracking, and trigger deployments automatically when new code is pushed. This is a “pull-based” model, where the operator watches the Git repo for changes. This ensures that the operator can be run in a private network environment that is not reachable from the internet (or via GitHub webhooks), and that the operator is fully in control of how these updates are discovered and processed.

This opens up a ton of new use cases for the Pulumi Kubernetes Operator, allowing the declarative model to be pushed from being purely within the Kubernetes resource, to allowing the exact commit to deploy to be specified in a Git repo branch. This allows infrastructure configuration to be defined once for a given environment, and then actively updated via git push. In this mode, the Pulumi Kubernetes Operator can be used in place of traditional CI/CD integrations or other cloud infrastructure deployment workflows.

The same stack as above can be changed to track a branch by using the branch option:

apiVersion: pulumi.com/v1
kind: Stack
metadata:
 name: nginx-k8s-stack-production
spec:
 stack: acmecorp/nginx/production
 projectRepo: https://github.com/acmecorp/pulumi-nginx
 branch: refs/heads/production

This will watch the production branch of the repo for Git pushes, and then trigger updates to the production stack of the NGINX deployment based on the new code pushed to that branch.

To help identify what commit was last deployed when tracking a branch, the Operator publishes information about the last update under .status.lastUpdate, including the lastAttemptedCommit property and lastSuccessfulCommit property.

Along with support for branch tracking, the Operator also now supports access to private Git repositories, critical for deploying your production infrastructure via the Pulumi Kubernetes Operator.

Building on the Pulumi Automation API

A couple months after we released the Pulumi Kubernetes Operator, we also released the initial version of the Pulumi Automation API. The Automation API provides an SDK for embedding Pulumi deployments inside your own software and systems, allowing you to build your own interfaces for managing cloud infrastructure. The design of the Automation API was heavily influenced by our experience building the Pulumi Kubernetes Operator, which was a perfect example of one of these custom interfaces - the Pulumi Kubernetes Operator embeds Pulumi deployments inside Kubernetes, offering a new interface (Kubernetes Stack resources) for managing cloud infrastructure.

Since then, we’ve moved the Pulumi Kubernetes Operator over to use the Pulumi Automation API, significantly reducing the size of the operator implementation, building on a robust foundation shared with dozens of other projects in the Pulumi ecosystem, and benefiting from many improvements and features added to the Pulumi Automation API. Earlier this year, the Pulumi Automation API also reached GA as part of the Pulumi 3.0 release.

Support for any state and secrets management backend

Pulumi supports a wide variety of state management and secrets management backends. Since the initial release of the Pulumi kubernetes Operator, we have added support for using not just the Pulumi Service backends, but also all of the other open source backend options, such as S3, Azure Blob and GCS for state storage, and KMS, KeyVault and Vault for secrets management.

apiVersion: pulumi.com/v1
kind: Stack
metadata:
 name: s3-bucket-stack
spec:
 backend: s3://my-s3-backend
 stack: acmecorp/operator-dev
 projectRepo: https://github.com/acmecorp/pytest1
 initOnCreate: true
 branch: refs/head/master
 config:
 aws:region: us-west-2

Worry-free Upgrades

The Pulumi Kubernetes Operator manages cloud infrastructure deployments which can take seconds to tens of minutes to complete. Although Pulumi ensures these deployments can be continued if interrupted, we want to ensure as little disruption to critical cloud infrastructure deployments as possible via the Operator. As a result, we have added a few key features to ensure that the Pulumi Kubernetes Operator can be safely upgraded to new versions without disrupting any deployments that are in-flight.

This includes capabilities like Leader Election, Graceful Shutdown and more.

Prometheus Metrics

The Pulumi Kubernetes Operator automatically manages a lot of details of Pulumi deployments, and as a result, it’s critical to be able to get insight into the operation of the Operator. Previously, this was available either via operator logs, or via the Pulumi Service. As part of the 1.0 release, the Operator is also now exporting Prometheus-compatible metrics for stacks_active and stacks_failing. These can be used to drive reporting and other workflows inside the Kubernetes cluster.

$ kubectl port-forward service/pulumi-kubernetes-operator-metrics 8383:8383 &
$ curl -s localhost:8383/metrics | grep "stacks_active\|stacks_failing"
# HELP stacks_active Number of stacks currently tracked by the Pulumi Kubernetes Operator
# TYPE stacks_active gauge
stacks_active 2
# HELP stacks_failing Number of stacks currently registered where the last reconcile failed
# TYPE stacks_failing gauge
stacks_failing{name="stack-test-aws-s3-commit-change-5wuczx",namespace="default"} 1

Documentation

We’ve added new documentation and enhanced existing documentation to ensure it’s easy to get started with the Pulumi Kubernetes Operator.

Full Stack Custom Resource API documentation is now available. And guides for deploying the Pulumi Kubernetes Operator and creating Stack resources are available for use either from kubectl/YAML or using the Pulumi Kubernetes Provider.

What’s Next

The 1.0 release of the Pulumi Kubernetes Operator is an important milestone, but there is still lots more planned to further expand the capabilities and scalability of the Operator. Some upcoming areas we’ll be investing:

Conclusion

The Pulumi Kubernetes Operator is a great way to deploy and manage cloud infrastructure from within your Kubernetes environment. With the 1.0 release, we’ve added many significant enhancements, as well as scalability, performance and quality improvements. You can get started with the Pulumi Kubernetes Operator today. Enjoy!

Preview of the Deploy Track at Cloud Engineering Summit 2021

Laura Santamaria — Wed, 15 Sep 2021 09:00:00 -0500

Cloud Engineering Summit 2021 is almost here! We’ve got a great line up this year.

Our tracks are built around the three pillars of cloud engineering: Build, Deploy, and Manage. I’m your track chair for the Deploy track, the track focused on automating and managing infrastructure. Deploy is all about unifying systems so everything in a cloud-based system is shipped together from the same automated, auditable process, reducing human error and improving quality across the board. That could mean focusing on automated testing and linting of infrastructure as code, providing shared services platforms for others to use in their pipelines, or exploring how unified and automated infrastructure changes engineering culture in an organization.

In no particular order, let’s go explore the Deploy track lineup.

Trying to get a handle on way too many environments across your pipelines? Adora Nwodo, Software Engineer at Microsoft, will explore bringing standard software engineering practices to solve the problem of maintaining many environments and configurations in a repeatable fashion. Adora has a very accessible way of explaining complex concepts in her blog and videos (find her at AdoraHacks), and I’m really looking forward to getting her perspective.

How can we improve the security of these unified processes? Alyssa Miller, Business Information Security Officer (BISO) at S&P Global Ratings, will discuss how threat modeling can improve CI/CD pipelines and general security. I’m excited to have Alyssa join us to help us understand how security should be part of the Deploy pillar; her Pitmaster’s Guide to Security is really good and I recommend watching it!

When you’re trying to share what you’re doing, you could share your code, but how do you share your infrastructure? Peter McKee, Head of Developer Relations and Advocacy at Docker, will demonstrate how to build into pipelines the ability to share features and changes in applications with others as a continuous preview. Peter’s a fantastic teacher who always has a great hands-on presentation in his back pocket, so I’m really looking forward to finding out what he has in store for us.

Data builds up while you’re building out, deploying, and managing cloud-based systems. How can you manage data while deploying your cloud infrastructure? Andy Dang, Co-Founder and Engineering Lead at WhyLabs, will talk about data integrity, observability, and monitoring with an open-source project called whylogs. Always love having a preview of an open-source project in the program, and I think data’s going to continue to be a big question for cloud engineering, so Andy’s talk was very welcome!

What about ARM-based systems? Apple certainly got into the game with their M1 chip, so it’s a platform you have to consider when doing deployments. Angel Rivera, Senior Developer Advocate at CircleCI, will explore how to make pipelines and workloads ARM compatible and capable of supporting ARM on various products. ARM-based systems are some of my favorites (yay Arduino and Raspberry Pi!), and they’re also really important considerations if you’re in the mobile or embedded space, so I can’t wait to hear Angel’s take.

How about some focus on doing all of this at scale? Jenna Pederson, Developer Advocate at AWS, will explore how bringing all of the best practices from software development to pipelines will help make working with cloud-based resources consistent and secure. I’m so happy Jenna could join us! Her content is wonderfully accessible and really explains technical concepts and complex systems clearly for all levels (which is my jam!). You don’t want to miss her talk!

Finally, how does all of this stuff work in a real life team? Evan Bradley, Software Engineer, and Tom Carrio, Senior Software Engineer, from Dynatrace are going to take us through bringing it all together with a real-life team, discussing the experiences they’ve had over time as they’ve implemented everything. I’m really excited to have folks bring their stories of what it’s all like so we can share pitfalls and solutions to move forward together. I think you’ll enjoy Evan and Tom’s talk, too!

Explore all the sessions, and don’t forget to register for Cloud Engineering Summit. I can’t wait to chat with you virtually at Cloud Engineering Summit 2021!

Unify app and infra delivery with Pulumi's CI/CD Assistant

Alex Mullans — Tue, 20 Apr 2021 00:00:01 -0700

Pulumi’s CI/CD Assistant helps you bring your infrastructure under version control and create a continuous integration and delivery pipeline that deploys changes to your cloud applications and infrastructure whenever you make a change in source control. Using CI/CD secures your production delivery while ensuring that every deployment is expressed in a committed Pulumi program and quickly reflected in your deployed infrastructure. With the CI/CD Assistant, it’s easier than ever before to set up version control and a CI/CD pipeline for your favorite CI/CD system, even if you’re new to CI/CD workflows.

Since its launch late last year, the CI/CD Assistant has been available to organizations using the Team and Enterprise offerings; today at PulumiUP, we announced that it’s now also available to all teams, including Team organizations. Get started with the CI/CD Assistant on any stack’s settings page, or learn more in the docs.

Get your cloud applications and infrastructure to production faster and safer

Building source code safely in version control and deploying using continuous integration and delivery (CI/CD) pipelines are best practices of cloud engineering. Using version control creates a full history of changes to Pulumi infrastructure as code programs you write: when the change was made, who made it, and what was changed. Version control systems also provide an easy mechanism to revert to an old infrastructure configuration: you can restore a past version of your infrastructure as code and redeploy it, effectively rolling back by rolling forward. Adding a CI/CD pipeline adds even more benefit by accelerating the deployment of changes. The CI/CD pipeline for your cloud code can deploy updates on every change, on a schedule, or in many CI/CD systems by an external event or workflow you’ve configured. A well-configured CI/CD pipeline is also much more secure: you can focus on restricting access to production deployments, generating temporary access keys for the CI/CD environment, and also also use the Pulumi Secrets feature to safeguard and minimize access to any and all secrets you use to deploy.

The CI/CD Assistant helps you realize all these benefits by getting version control and CI/CD configured for your infrastructure stacks. The Assistant walks you through a 4-step guided procedure:

Configuring version control, so Pulumi can show you the code history associated with each update
Configuring your CI/CD system to access Pulumi, including any secrets Pulumi safeguards for your stack
Creating a CI/CD workflow file that shows how to run Pulumi in your CI/CD system
Validating that everything was configured correctly

When you’ve configured version control and CI/CD on a Pulumi stack, you get additional helpful metadata in the Pulumi Service: each update shows the commit, branch, pull request (if applicable), and build number - all with links to your version control and CI/CD systems so you can dig into additional detail.

👉 Learn more

Support for many CI/CD providers

The CI/CD Assistant delivers on the Pulumi Cloud Engineering Platform’s promise of your cloud, your language, your way, by providing broad support for over a dozen of the most popular CI/CD services:

If you use a version control or CI/CD provider that’s not on this list, you can file a feature request. We’d love to hear from you!

👉 Request a new version control or CI/CD provider

Keep exploring

This announcement is just one part of the story. Watch the PulumiUP event for more.

👉 PulumiUP

CI/CD Pipelines for Kubernetes Apps with Pulumi & Codefresh

Sophia Parafina — Tue, 19 Jan 2021 00:00:00 +0000

Delivering modern applications is complicated and requires the coordination of many moving parts. Applications are frequently updated to implement new features and improve security and performance, translating to a better user experience for your customers. To further complicate matters, infrastructure must also be deployed and maintained simultaneously with applications to avoid conflicts or dependencies.

Containerized applications deployed on Kubernetes are particularly susceptible to a misalignment between developers who frequently push changes and operators who want to maintain a stable architecture. Continuous Integration builds and tests software and delivers it as packages. Continuous Delivery or Deployment deploys applications on infrastructure. Let’s take a look at how we can accomplish CI/CD for both applications and infrastructure.

Join us for a CI/CD with Kubernetes technical session on January 21st.

CI/CD and Pipelines

CD is sometimes called Continuous Delivery or Continuous Deployment. These have the same results, i.e., deploying software artifacts to production. However, Continuous Deployment is an automated process that deploys artifacts without human intervention. Continuous Delivery occurs when a person decides when to deploy an application. The reason could be a business decision, such as coordinating the release with a marketing campaign or having a final check by an engineer before release.

Continuous integration is an automated process that begins with a code push to a repository. From there, the code can be built and tested, and eventually released as an artifact. Your organization determines the CI process, and it can include manual testing of the code to catch corner cases.

A pipeline is the deployable unit path for CI/CD. A pipeline starts when code is committed to a repository like GitHub. The next step is a notification to a build system, such as Codefresh. The build system compiles the code and runs unit tests. If your code passes the unit tests, integration tests are the next step. If your code passes both unit and integration tests, the images will be created and pushed into a registry service in the case of containers. This is the simplest example of a pipeline, but you can do many more things such as security scans, check modules for CVEs (Common Vulnerabilities and Exploits), send Slack notifications, and run quality checks. A pipeline can be fully automated or have checkpoints that require approval before resuming.

Codefresh is a modern CI/CD platform specifically created for Kubernetes and containers. A pipeline in Codefresh is composed of a series of steps, where every step is a Docker container. This means that creating a pipeline is as easy as collecting your favorite Docker images from any public or private Registry. Integration with Pulumi is trivial as Pulumi is already offered in the form of a public image.

On the deployment front, Codefresh has native integration for Kubernetes, container registries, and Git providers. This means you can set up your cluster once in a central configuration and then make it available to all your pipelines by name. Codefresh is cloud-agnostic on all fronts. All major Git providers are supported, and deployment can happen on any compliant Kubernetes cluster.

Every Codefresh account comes with a built-in graphical dashboard that allows you to inspect your Kubernetes cluster not only on the service/pod level but also on the application (i.e., Helm releases). This means that with Codefresh, you don’t need to use multiple unrelated applications to understand if your deployment was successful.

A deployment that starts from a Git commit can be monitored with Codefresh from beginning to end, following all pipeline stages from application compilation/packaging to security scanning and infrastructure deployment.

See our dedicated documentation page on how to use Pulumi and Codefresh together.

Building Modern Infrastructure

Just like your application code, you can build your infrastructure with a CI/CD pipeline. Pulumi lets developers write infrastructure using their favorite language, such as Typescript, Python, or go. Developers can take advantage of abstractions, code reuse, refactoring, favorite development environments, and testing tools.

In addition to supporting multiple languages, you can build infrastructure on any cloud such as AWS, Azure, Google Cloud, and Kubernetes and take full advantage of their service offerings. Choose your toolchain and frameworks, and you’re set to deploy infrastructure on any major cloud provider and even hybrid and on-premises environments.

By leveraging infrastructure as code combined with software development practices, engineers can achieve greater productivity while enabling their development teams to be more “self-serve” with appropriate policies and guardrails in place. Pulumi brings cloud security engineering to the infrastructure and development teams. Using one common workflow that spans any cloud environment, whether public, private, or hybrid, security teams can enforce policies that accomplish cloud governance projects of all kinds — security, compliance, cost control, and more.

To see how infrastructure as code simplifies infrastructure deployment, let’s look at an example for deploying Kubernetes with python on AWS. The full example is available on Github.

The code breaks up the Kubernetes deployment into python modules:

vpc.py creates an AWS Virtual Private Cloud containing the cluster, security group, and networking.
iam.py creates roles for the cluster, and the nodegroup and policy attachments
utils.py creates a kubeconfig file that can be used with kubectl to manage Kubernetes

The main program (main.py) creates the Kubernetes cluster by calling the modules to deploy a cluster and a nodegroup with the policies applied. Using modules, we can make changes to the vpc, networking, roles, and policies and customize it to our application. In a CI/CD, all these changes would be pushed through a pipeline before deployment.

Although this example is written in Python for AWS, there are similar examples in other languages and cloud providers in Pulumi Examples.

Putting it Together

This article described how we could build both applications and infrastructure with CI/CD pipelines. To learn more about how Codefresh and Pulumi can automate Kubernetes deployments join us for a technical session on January 21st.

Amp-up GitLab Merge Requests With Pulumi

Praneet Loke — Wed, 26 Aug 2020 00:00:00 +0000

We are excited to announce the launch of native support for integrating GitLab Merge Requests with Pulumi. By integrating your GitLab Projects directly with Pulumi, you can now approve your merge requests with confidence.

New To Pulumi and GitLab CI/CD?

GitLab’s merge request pipelines runs a CI/CD workflow for building and testing your software. Pulumi makes a great addition to it, by allowing you to preview any infrastructure changes as part of your merge request flow.

Using the merge request pipeline feature with Pulumi can validate the changes to your infrastructure before they are applied. Once validated and approved through a peer-review process, you can apply your changes quickly and continuously with confidence while doing so, a process commonly referred to as GitOps for infrastructure.

To leverage GitOps for your infrastructure, the Pulumi CLI offers two commands: preview and up.

Concept	Pulumi command	Role
Merge requests pipeline	`pulumi preview`	Preview infrastructure changes to be merged.
Master branch pipeline	`pulumi up`	Update your infrastructure with latest changes

Learn how to use Pulumi in GitLab CI/CD with merge request pipelines and how to configure the GitLab project integration with Pulumi.

Already Using Both?

If you are already a user of either Pulumi and GitLab CI/CD, you know the importance of merge request pipelines and running Pulumi within them. With today’s announcement, you can now make any changes to your infrastructure in your merge request front and center, so you don’t miss any of those changes.

Here’s a preview of what that looks like 😉

Head over to our docs to learn how to configure the GitLab integration, whether for a single project or all of the projects under a GitLab Group.

GitLab provides lots of tools for streamlining your application delivery, and the new integration with Pulumi makes things even better. As always, we’d love to hear your feedback or suggestions for other improvements. Check out the Pulumi Community Slack.

Introducing the Pulumi Kubernetes Operator

Mike Metral — Wed, 12 Aug 2020 00:00:00 +0000

Check out version 2.0 of the Pulumi Kubernetes Operator.

Kubernetes developers and operators work together to manage workloads and to continuously ship software through CI/CD. These users have an affinity for automation and pipelines, and richer integration with Kubernetes is a growing theme across the cloud native ecosystem.

We’re excited to introduce the Pulumi Kubernetes Operator: a Kubernetes controller that deploys cloud infrastructure in Pulumi Stacks for you and your team.

These program stacks include virtual machines, block storage, managed Kubernetes clusters, API resources, serverless functions and more!

Overview

The Pulumi Kubernetes Operator is an extension pattern that enables Kubernetes users to create a Pulumi Stack as a first-class API resource, and use the StackController to drive the updates of the Stack until success.

Check out how to deploy the operator to a Kubernetes cluster using any of Pulumi’s supported languages, or through YAML manifests and kubectl.

You can also get started with Pulumi and create a new managed Kubernetes cluster on Amazon EKS, Google GKE, or Azure AKS if you don’t have an existing cluster.

How it Works

When the operator is deployed, the StackController waits for Stack CustomResources to be created, patched, or deleted in the cluster.

On these events, the Kubernetes controller invokes a reconciliation loop to process the request until it has reached success. For Stack creation or updates, this is a successful pulumi up. For deletions, it is a successful pulumi destroy and pulumi stack rm.

Stacks can be written in Typescript, Python, .NET, and Go, and span cloud providers such as Amazon AWS, Google GCP, and Microsoft Azure, as well as many cloud native services!

The following example showcases a typical Pulumi program in Typescript, that can be instantiated as a Stack.

import * as aws from "@pulumi/aws";

// Create an AWS resource (S3 Bucket)
const names = [];
for (let i = 0; i < 2; i++) {
 const bucket = new aws.s3.Bucket(`my-bucket-${i}`, {
 acl: "public-read",
 });
 names.push(bucket.id);
}

// Export the name of the buckets
export const bucketNames = names;

You can find more examples on GitHub of the types of infrastructure you can manage with Pulumi.

Check out Kubernetes Controllers work if you’d like to learn more.

Creating a Pulumi Stack in Kubernetes

The Stack CustomResourceDefinition (CRD) encapsulates an existing Pulumi infrastructure project in a Git repo, a specific commit SHA to deploy, and any additional settings to control the Pulumi update run.

These settings include your Pulumi API access token, environment variables, config and secrets, and lifecycle controls for the update.

Deploying an NGINX Stack on Kubernetes

Check out an example of creating a Kubernetes NGINX Deployment as a Stack by the Operator in its cluster.

Choose your preferred language below, or check out how to create Pulumi Stacks using kubectl.

import * as pulumi from "@pulumi/pulumi";
import * as k8s from "@pulumi/kubernetes";
import * as kx from "@pulumi/kubernetesx";

// Get the Pulumi API token.
const pulumiConfig = new pulumi.Config();
const pulumiAccessToken = pulumiConfig.requireSecret("pulumiAccessToken")

// Create the API token as a Kubernetes Secret.
const accessToken = new kx.Secret("accesstoken", {
 stringData: { accessToken: pulumiAccessToken },
});

// Create an NGINX deployment in-cluster.
const mystack = new k8s.apiextensions.CustomResource("my-stack", {
 apiVersion: 'pulumi.com/v1alpha1',
 kind: 'Stack',
 spec: {
 accessTokenSecret: accessToken.metadata.name,
 stack: "<YOUR_ORG>/k8s-nginx/dev",
 initOnCreate: true,
 projectRepo: "https://github.com/pulumi/examples",
 repoDir: "kubernetes-ts-nginx",
 commit: "e2e5eb426dbf5b57c50bba0f8eb54fe982ceddb1",
 destroyOnFinalize: true,
 }
});

import pulumi
from pulumi_kubernetes import core, apiextensions

# Get the Pulumi API token.
pulumi_config = pulumi.Config()
pulumi_access_token = pulumi_config.require_secret("pulumiAccessToken")

# Create the API token as a Kubernetes Secret.
access_token = core.v1.Secret("accesstoken", string_data={ "access_token": pulumi_access_token })

# Create an NGINX deployment in-cluster.
my_stack = apiextensions.CustomResource("my-stack",
 api_version="pulumi.com/v1alpha1",
 kind="Stack",
 spec={
 "access_token_secret": access_token.metadata["name"],
 "stack": "<YOUR_ORG>/k8s-nginx/dev",
 "init_on_create": True,
 "project_repo": "https://github.com/pulumi/examples",
 "repo_dir: "kubernetes-ts-nginx",
 "commit": "e2e5eb426dbf5b57c50bba0f8eb54fe982ceddb1",
 "destroy_on_finalize": True,
 }
)

using Pulumi;
using Pulumi.Kubernetes.ApiExtensions;
using Pulumi.Kubernetes.Core.V1;
using Pulumi.Kubernetes.Types.Inputs.Core.V1;

class StackArgs : CustomResourceArgs
{
 [Input("spec")]
 public Input<StackSpecArgs>? Spec { get; set; }

 public StackArgs() : base("pulumi.com/v1alpha1", "Stack")
 {
 }
}

class StackSpecArgs : ResourceArgs
{
 [Input("accessTokenSecret")]
 public Input<string>? AccessTokenSecret { get; set; }

 [Input("stack")]
 public Input<string>? Stack { get; set; }

 [Input("initOnCreate")]
 public Input<bool>? InitOnCreate { get; set; }

 [Input("projectRepo")]
 public Input<string>? ProjectRepo { get; set; }

 [Input("commit")]
 public Input<string>? Commit { get; set; }

 [Input("destroyOnFinalize")]
 public Input<bool>? DestroyOnFinalize { get; set; }
}

class MyStack : Stack
{
 public MyStack()
 {
 // Get the Pulumi API token.
 var config = new Config();
 var pulumiAccessToken = config.RequireSecret("pulumiAccessToken");

 // Create the API token as a Kubernetes Secret.
 var accessToken = new Secret("accesstoken", new SecretArgs
 {
 StringData =
 {
 {"accessToken", pulumiAccessToken}
 }
 });

 // Create an NGINX deployment in-cluster.
 var myStack = new Pulumi.Kubernetes.ApiExtensions.CustomResource("nginx", new StackArgs
 {
 Spec = new StackSpecArgs
 {
 AccessTokenSecret = accessToken.Metadata.Apply(m => m.Name),
 Stack = "<YOUR_ORG>/k8s-nginx/dev",
 InitOnCreate = true,
 ProjectRepo = "https://github.com/pulumi/examples",
 RepoDir = "kubernetes-ts-nginx",
 Commit = "e2e5eb426dbf5b57c50bba0f8eb54fe982ceddb1",
 DestroyOnFinalize = true,
 }
 });
 }
}

package main

import (
 "github.com/pulumi/pulumi-kubernetes/sdk/v2/go/kubernetes"
 apiextensions "github.com/pulumi/pulumi-kubernetes/sdk/v2/go/kubernetes/apiextensions"
 corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v2/go/kubernetes/core/v1"
 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v2/go/kubernetes/meta/v1"
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi"
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi/config"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // Get the Pulumi API token.
 c := config.New(ctx, "")
 pulumiAccessToken := c.Require("pulumiAccessToken")

 // Create the API token as a Kubernetes Secret.
 accessToken, err := corev1.NewSecret(ctx, "accesstoken", &corev1.SecretArgs{
 StringData: pulumi.StringMap{"accessToken": pulumi.String(pulumiAccessToken)},
 })
 if err != nil {
 return err
 }

 // Create an NGINX deployment in-cluster.
 _, err = apiextensions.NewCustomResource(ctx, "my-stack", &apiextensions.CustomResourceArgs{
 ApiVersion: pulumi.String("pulumi.com/v1alpha1"),
 Kind: pulumi.String("Stack"),
 OtherFields: kubernetes.UntypedArgs{
 "spec": map[string]interface{}{
 "accessTokenSecret": accessToken.Metadata.Name(),
 "stack": "<YOUR_ORG>/k8s-nginx/dev",
 "initOnCreate": true,
 "projectRepo": "https://github.com/pulumi/examples",
 "repoDir": "kubernetes-ts-nginx",
 "commit": "e2e5eb426dbf5b57c50bba0f8eb54fe982ceddb1",
 "destroyOnFinalize": true,
 },
 },
 }, pulumi.DependsOn([]pulumi.Resource{accessToken}))
 return err
 })
}

Deploying an Amazon EKS Stack

Check out an example of creating a new AWS VPC and Kubernetes cluster on AWS EKS as a Stack by the Operator.

import * as pulumi from "@pulumi/pulumi";
import * as k8s from "@pulumi/kubernetes";
import * as kx from "@pulumi/kubernetesx";

// Get the Pulumi API token and AWS creds.
const pulumiConfig = new pulumi.Config();
const pulumiAccessToken = pulumiConfig.requireSecret("pulumiAccessToken")
const awsAccessKeyId = pulumiConfig.require("awsAccessKeyId")
const awsSecretAccessKey = pulumiConfig.requireSecret("awsSecretAccessKey")
const awsSessionToken = pulumiConfig.requireSecret("awsSessionToken")

// Create the creds as Kubernetes Secrets.
const accessToken = new kx.Secret("accesstoken", {
 stringData: { accessToken: pulumiAccessToken},
});
const awsCreds = new kx.Secret("aws-creds", {
 stringData: {
 "AWS_ACCESS_KEY_ID": awsAccessKeyId,
 "AWS_SECRET_ACCESS_KEY": awsSecretAccessKey,
 "AWS_SESSION_TOKEN": awsSessionToken,
 },
});

// Create a Kubernetes cluster on AWS EKS.
const mystack = new k8s.apiextensions.CustomResource("my-stack", {
 apiVersion: 'pulumi.com/v1alpha1',
 kind: 'Stack',
 spec: {
 stack: "<YOUR_ORG>/pulumi-aws-eks/dev",
 projectRepo: "https://github.com/metral/pulumi-aws-eks",
 commit: "fc4ab1a3e49d48cf5c764cd8cd626879a90bcc45",
 accessTokenSecret: accessToken.metadata.name,
 config: {
 "aws:region": "us-west-2",
 },
 envSecrets: [awsCreds.metadata.name],
 initOnCreate: true,
 destroyOnFinalize: true,
 }
});

import pulumi
from pulumi_kubernetes import core, apiextensions

# Get the Pulumi API token.
pulumi_config = pulumi.Config()
pulumi_access_token = pulumi_config.require_secret("pulumiAccessToken")
aws_access_key_id = pulumi_config.require("awsAccessKeyId")
aws_secret_access_key = pulumi_config.require_secret("awsSecretAccessKey")
aws_session_token = pulumi_config.require_secret("awsSessionToken")

# Create the creds as Kubernetes Secrets.
access_token = core.v1.Secret("accesstoken", string_data={ "access_token": pulumi_access_token })
aws_creds = core.v1.Secret("aws-creds", string_data={
 "AWS_ACCESS_KEY_ID": aws_access_key_id,
 "AWS_SECRET_ACCESS_KEY": aws_secret_access_key,
 "AWS_SESSION_TOKEN": aws_session_token,
})

# Create a Kubernetes cluster on AWS EKS.
my_stack = apiextensions.CustomResource("my-stack",
 api_version="pulumi.com/v1alpha1",
 kind="Stack",
 spec={
 "stack": "<YOUR_ORG>/pulumi-aws-eks/dev",
 "project_repo": "https://github.com/metral/pulumi-aws-eks",
 "commit": "fc4ab1a3e49d48cf5c764cd8cd626879a90bcc45",
 "access_token_secret": access_token.metadata["name"],
 "config": {
 "aws:region": "us-west-2",
 },
 "env_secrets": [aws_creds.metadata["name"]],
 "init_on_create": True,
 "destroy_on_finalize": True,
 }
)

using System;
using Pulumi;
using Pulumi.Kubernetes.ApiExtensions;
using Pulumi.Kubernetes.Core.V1;
using Pulumi.Kubernetes.Types.Inputs.Core.V1;

class StackArgs : CustomResourceArgs
{
 [Input("spec")]
 public Input<StackSpecArgs>? Spec { get; set; }

 public StackArgs() : base("pulumi.com/v1alpha1", "Stack")
 {
 }
}

class StackSpecArgs : ResourceArgs
{
 [Input("accessTokenSecret")]
 public Input<string>? AccessTokenSecret { get; set; }

 [Input("stack")]
 public Input<string>? Stack { get; set; }

 [Input("initOnCreate")]
 public Input<bool>? InitOnCreate { get; set; }

 [Input("projectRepo")]
 public Input<string>? ProjectRepo { get; set; }

 [Input("commit")]
 public Input<string>? Commit { get; set; }

 [Input("destroyOnFinalize")]
 public Input<bool>? DestroyOnFinalize { get; set; }

 [Input("envSecrets")]
 public InputList<String>? EnvSecrets { get; set; }

 [Input("config")]
 public InputMap<String>? Config { get; set; }
}

class MyStack : Stack
{
 public MyStack()
 {
 // Get the Pulumi API token.
 var config = new Config();
 var pulumiAccessToken = config.RequireSecret("pulumiAccessToken");
 var awsAccessKeyId = config.Require("awsAccessKeyId");
 var awsSecretAccessKey = config.RequireSecret("awsSecretAccessKey");
 var awsSessionToken = config.RequireSecret("awsSessionToken");

 // Create the creds as Kubernetes Secrets.
 var accessToken = new Secret("accesstoken", new SecretArgs
 {
 StringData =
 {
 {"accessToken", pulumiAccessToken}
 }
 });
 var awsCreds = new Secret("aws-creds", new SecretArgs
 {
 StringData =
 {
 {"AWS_ACCESS_KEY_ID", awsAccessKeyId},
 {"AWS_SECRET_ACCESS_KEY", awsSecretAccessKey},
 {"AWS_SESSION_TOKEN", awsSessionToken}
 }
 });

 // Create an Kubernetes cluster on AWS EKS.
 var myStack = new Pulumi.Kubernetes.ApiExtensions.CustomResource("my-stack", new StackArgs
 {
 Spec = new StackSpecArgs
 {
 Stack = "<YOUR_ORG>/pulumi-aws-eks/dev",
 ProjectRepo = "https://github.com/metral/pulumi-aws-eks",
 Commit = "fc4ab1a3e49d48cf5c764cd8cd626879a90bcc45",
 AccessTokenSecret = accessToken.Metadata.Apply(m => m.Name),
 Config =
 {
 {"aws:region", "us-west-2"}
 },
 EnvSecrets = {awsCreds.Metadata.Apply(m => m.Name)},
 InitOnCreate = true,
 DestroyOnFinalize = true,
 }
 });
 }
}

package main

import (
 "github.com/pulumi/pulumi-kubernetes/sdk/v2/go/kubernetes"
 apiextensions "github.com/pulumi/pulumi-kubernetes/sdk/v2/go/kubernetes/apiextensions"
 corev1 "github.com/pulumi/pulumi-kubernetes/sdk/v2/go/kubernetes/core/v1"
 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v2/go/kubernetes/meta/v1"
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi"
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi/config"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // Get the Pulumi API token and AWS creds.
 config := config.New(ctx, "")
 pulumiAccessToken := config.Require("pulumiAccessToken")
 awsAccessKeyID := config.Require("awsAccessKeyId")
 awsSecretAccessKey := config.Require("awsSecretAccessKey")
 awsSessionToken := config.Require("awsSessionToken")

 // Create the creds as Kubernetes Secrets.
 accessToken, err := corev1.NewSecret(ctx, "accesstoken", &corev1.SecretArgs{
 StringData: pulumi.StringMap{"accessToken": pulumi.String(pulumiAccessToken)},
 })
 if err != nil {
 return err
 }
 awsCreds, err := corev1.NewSecret(ctx, "aws-creds", &corev1.SecretArgs{
 StringData: pulumi.StringMap{
 "AWS_ACCESS_KEY_ID": pulumi.String(awsAccessKeyID),
 "AWS_SECRET_ACCESS_KEY": pulumi.String(awsSecretAccessKey),
 "AWS_SESSION_TOKEN": pulumi.String(awsSessionToken),
 },
 })
 if err != nil {
 return err
 }

 // Create an Kubernetes cluster on AWS EKS.
 _, err = apiextensions.NewCustomResource(ctx, "my-stack",
 &apiextensions.CustomResourceArgs{
 ApiVersion: pulumi.String("pulumi.com/v1alpha1"),
 Kind: pulumi.String("Stack"),
 OtherFields: kubernetes.UntypedArgs{
 "spec": map[string]interface{}{
 "stack": "<YOUR_ORG>/s3-op-project/dev",
 "projectRepo": "https://github.com/metral/pulumi-aws-eks",
 "commit": "fc4ab1a3e49d48cf5c764cd8cd626879a90bcc45",
 "accessTokenSecret": accessToken.Metadata.Name(),
 "config": map[string]string{
 "aws:region": "us-west-2",
 },
 "envSecrets": []interface{}{awsCreds.Metadata.Name()},
 "initOnCreate": true,
 "destroyOnFinalize": true,
 },
 },
 }, pulumi.DependsOn([]pulumi.Resource{accessToken, awsCreds}))

 return nil
 })
}

CI/CD Scenarios

As you build out your Kubernetes workloads, you’ll need to implement update strategies to ship new software easily and reliably. Kubernetes API resources like Deployments are used to model stateless apps and operate their lifecycles e.g. recreate app replicas, or perform a rolling update.

A common approach that most teams rely on is the Blue / Green deployment. This starts with an initial version of the app, and then deploys an updated version alongside the initial version. Once ready, traffic is switched over from the initial version to the new version with no downtime.

In Kubernetes, Blue/Green is modeled by using a Service that load balances and selects a set of Deployment pods labeled “blue,” and then switching to a different selection of updated, “green” pods.

Similar to the Stack CustomResources in the previous sections, we can deploy a Stack of a Blue/Green Kubernetes app in Pulumi, and step through a sequence of its Git commits similar to how a CI/CD pipeline does. Additionally, we can use the Pulumi config system to parameterize settings and secrets on how to manage the Stack CustomResource project.

Deploying the active blue and passive green versions of the app can be done on the example Stack with the following steps.

pulumi config set --secret pulumiAccessToken <YOUR_PULUMI_API_TOKEN>
pulumi config set stackProjectRepo https://github.com/metral/pulumi-blue-green
pulumi config set stackCommit b19759220f25476605620fdfffeface39a630246
pulumi up -y

After deployment, we’ll test that the v1 / “blue” endpoint is live and running. The v2 / “green” app will be on standby.

while true; do curl http://34.83.25.150:80/version ; echo; sleep 1; done

{"id":1,"content":"current"}
{"id":1,"content":"current"}
{"id":1,"content":"current"}
{"id":1,"content":"current"}
...

When we’re ready to switch traffic, the blue/green transition is done by updating the config to the Git new commit of the selector swap, and running an update.

pulumi config set stackCommit f4bf2f7b54ec441d5af374933cca4ef09f2bce24
pulumi up -y

After a few seconds you should start seeing the generated traffic returning the new version in a clean cut-over with no dropped packets.

...
{"id":1,"content":"current"}
{"id":1,"content":"current"}
{"id":1,"content":"current"}
{"id":2,"content":"new"}
{"id":2,"content":"new"}
{"id":2,"content":"new"}
{"id":2,"content":"new"}
...

Check out the full walkthrough for a tutorial, and the video clip below for a demo.

Wrap-Up

The Pulumi Kubernetes Operator helps deploy Pulumi Stacks in your Kubernetes cluster, and naturally fits in with your CI/CD process.

We covered examples like deploying an NGINX Stack in the same cluster as the Operator, creating a new Kubernetes cluster in an AWS EKS Stack, and how Stacks can be abstracted for deployment strategies like Blue/Green.

There are many more examples of deploying infrastructure, containers, serverless, and Kubernetes across cloud providers and cloud native services.

Next Steps

Check out the GitHub repo to experiment deploying the operator and some test Stacks.

Learn more about how Pulumi works with Kubernetes, and Get Started if you’re new.

You can help to shape this experience directly by providing feedback on GitHub. We love to hear from our users!

You can explore more content by checking out PulumiTV on YouTube, work through Kubernetes tutorials to dive deeper, and join the Community Slack to engage with users and the Pulumi team.

Unlocking Spinnaker With Pulumi

Praneet Loke — Thu, 18 Jun 2020 00:00:00 +0000

Spinnaker is an open source, multi-cloud continuous delivery platform for releasing software changes with high velocity and confidence. This makes it a perfect fit for Pulumi to unleash the power of Spinnaker and its continuous delivery capabilities to all of the providers that Pulumi supports today and in the future.

We have written a developer guide showing you how to install the plugin in your Spinnaker instance and start using it right away.

Here’s a brief overview of the plugin.

Create a new Spinnaker pipeline with the Pulumi stage.

When you run a pipeline that has a Pulumi stage in it, you’ll be able to see the logs of the Pulumi CLI within your pipeline execution details window. Click the Console Output to view the logs.

Here’s a demo of the plugin:

If you have any questions or feedback about the plugin, please feel free to open an issue in the repo or join the conversation on our Community Slack as always.

Managing AWS Credentials on CI/CD: Part 3

Chris Smith — Thu, 14 May 2020 00:00:00 +0000

This article is the third part of a series on best practices for securely managing AWS credentials on CI/CD. In this article, we cover the last leg of the continuous delivery process to update your AWS resources and how to store sensitive data using Pulumi securely.

Recap

In part 1, we created a dedicated IAM User with limited privileges. Then in part 2, we set up a simple, serverless Pulumi program that periodically rotated the User’s access keys and updated the CI/CD system.

In this final post, we will use those AWS credentials to update cloud resources as part of a CI/CD workflow using Pulumi.

We will cover AWS’s IAM Roles and how they can be used to safely manage the access Pulumi has to your AWS account. We will also cover how sensitive data in your stack is stored within the Pulumi Service.

Assuming IAM Roles for Performing Updates

In the previous post, we automatically create and rotate the AWS access keys for an IAM User. So it seems natural to think that we should just use that same AWS access key to gain access to AWS resources. However, using the access keys for an IAM User has a few major drawbacks.

First, those access keys are long-lived. If we didn’t create that Pulumi program to rotate them automatically, the key would last forever. (Meaning that a bad actor, perhaps elsewhere, or far in the future, could take advantage of a compromised key.) Second, that access key could be used to do anything that the IAM User could. So you cannot differentiate the set of capabilities based on the task at hand. (For example, if updating the Pulumi stack website/staging, we might need a different set of permissions than if we were updating the stack website/production.)

The preferred alternative to using IAM User access keys is to “assume” an IAM Role. An IAM Role is similar to an IAM User in that it grants a set of policies and permissions. However, unlike an IAM User, an IAM Role does not have a password, nor can it be added to a Group and inherit any policies. Also, the credentials for using an IAM Role are always short-lived. (Lasting no more than 12 hours, and likely much, much less.)

Instead, an IAM Role defines a set of capabilities that can be performed by any principal capable of assuming it. So in other words, you can exchange your limited access IAM User’s credentials for a specialized IAM Role. The IAM Role then can be specifically tailored to a specific— wait for it— role to perform.

Don’t worry if that doesn’t all make sense at first; let’s look at a concrete example.

Desired End State

In our situation, we want to create an IAM Role WebsiteStackUpdaterRole that grants permissions necessary for Pulumi to update the website/production stack. We will custom tailor this role to only grant access to the set of resources and operations we know the stack needs.

We also want to restrict who can assume the WebsiteStackUpdaterRole, so that only specific users/accounts can access it. e.g., the IAM User cicd-bot but not IAM User dwight-over-in-sales.

And just to show a more real-world example, we’ll do this using multiple AWS accounts, where the website/production stack’s resources are housed in a “production” AWS account, and the cicd-bot IAM User is defined in a bastion AWS account.

Creating the IAM Role

The following snippet creates an IAM Role resource using Pulumi. The most important input property of which is the assumeRolePolicy document, which defines who can assume this role.

This is a situation where Pulumi’s ability to declare the policy document in code is super-useful, as it makes it much easier to reference other resources (e.g., another AWS account’s ID) as well as supporting inline comments.

// IAM Role used by cicd-bot to update the website stack.
export const websiteStackUpdaterRole = new aws.iam.Role("WebsiteStackUpdaterRole", {
 name: "WebsiteStackUpdaterRole",
 description: "Broad access for updating Pulumi stacks in the 'production' AWS account.",
 maxSessionDuration: 30 * 60, // 30 minutes
 assumeRolePolicy: {
 Version: "2012-10-17",
 Statement: [
 {
 Effect: "Allow",
 Principal: {
 // We grant the "bastion account", which defines IAM Users the ability to assume this role.
 // The IAM Policies in that other AWS account will then determine which users, groups, etc.
 // will be able to use the `sts:AssumeRole` action on this IAM Role.
 "AWS": `arn:aws:iam::${awsAccountIds.bastionAccount}:root`,
 },
 // But as an additional precaution against misconfiguration in the bastion account, we add
 // some additional checks to determine who can access this policy.
 Condition: {
 "ForAllValues:StringEquals": {
 "aws:PrincipalTag/access-to-production": "true",
 "sts:ExternalId": [
 "contoso/website/production",
 "contoso/website/staging",
 ],
 },
 },
 Action: "sts:AssumeRole",
 },
 ],
 },
});

There’s a lot packed into that IAM Role’s configuration.

First, the “principal” (who can assume the role) references a specific AWS account. This is an example of how you can associate one AWS account with another. (And is best practice for isolating data between different departments, etc.) We’ll cover how to configure the AWS bastion account side to use this role later.

The next thing to call out is the use of the Condition element of the policy document. To assume the role, we require two checks to be performed.

That the principal assuming the role (e.g., IAM User) has the tag "access-to-production" with value "true." This way, we will reject any IAM User in the bastion account who isn’t tagged with "access-to-production" from assuming this role.
We also verify that the “external ID” (an optional parameter passed when assuming an IAM Role) matches a well-known Pulumi stack name. This way, we can double-check that the entity using the IAM Role is doing so to use a specific stack. (This is less of a security measure and more of a general hygiene type requirement.)

IAM Policy Document

Now we have an IAM Role; we need to attach a policy document that defines what permissions are granted.

This is the most important step since it defines precisely what the CI/CD bot can able to do.

If we define the policy too broadly, then it is a general risk. For example, what if a compromised NPM package in your Pulumi program did something nefarious. If the policy is defined too narrowly, then your pulumi up might fail because the necessary permissions weren’t available. When in doubt, err on the side of having a more restrictive policy.

It can be advantageous to have an identical “staging” and “production” instance of the same Pulumi stack, so you can detect any permission issues before you update the user-facing production stack.

It can take some time to winnow down the specific set of permissions necessary since that is ultimately specific to the Pulumi program. For example, if the stack governs the networking configuration, then you can simply not deny to the s3:* actions.

Unfortunately, Pulumi doesn’t currently support generating a reasonable IAM Policy document for your stack. For a good introduction to some of the “worst-case scenarios” that could arise from a misconfigured IAM Policy, see this blog post from BishopFox.

The following is a hypothetical IAM policy that restricts the operations performed to just those for the AWS CloudFront and S3 products. (Which might be sufficient for updating a CDN-based website hosted on AWS, but likely not much else.)

// Policy document defining the permissions granted to the WebsiteStackUpdaterRole.
export const websiteUpdaterPolicyDocument: aws.iam.PolicyDocument = {
 Version: "2012-10-17",
 Statement: [
 // Grant access to all AWS S3 actions, but scoped to just the specific S3 buckets
 // used for the given Pulumi stack.
 {
 Sid: "GrantS3Access",
 Effect: "Allow",
 Action: "s3:*",
 Resource: [
 // Content bucket.
 "arn:aws:s3:::website-bucket",
 "arn:aws:s3:::website-bucket/*",
 // S3 bucket which stores the access logs.
 "arn:aws:s3:::website-logs-bucket",
 "arn:aws:s3:::website-logs-bucket/*",
 ],
 },
 // Grant access to update a CloudFront distribution. For more information, see:
 // https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/access-control-managing-permissions.html
 {
 Sid: "GrantCloudFrontAccess",
 Effect: "Allow",
 Action: [
 "cloudfront:CreateDistribution",
 "cloudfront:DeleteDistribution",
 "cloudfront:GetDistribution",
 "cloudfront:GetDistributionConfig",
 "cloudfront:ListDistributions",
 "cloudfront:UpdateDistribution",
 "cloudfront:ListCloudFrontOriginAccessIdentities",
 ],
 // The specific CloudFront distribution used in the stack.
 Resource: "arn:aws:cloudfront::0123456789:distribution/A123B1CABWZZ",
 },
 ],
};

const websiteUpdaterPolicy = new aws.iam.Policy("WebsiteUpdaterRolePolicy", {
 name: "WebsiteUpdaterRolePolicy",
 description: "Policy granting the permissions needed for updating the website/production Pulumi stack",
 policy: websiteUpdaterPolicyDocument,
}, resourceOptions);

To actually associate the IAM Policy document with the new IAM Role, we need to create a PolicyAttachment associating the two.

const rolePolicyAttachment = new aws.iam.RolePolicyAttachment("WebsiteStackUpdaterRolePolicyAttachment", {
 role: websiteStackUpdaterRole,
 policyArn: websiteUpdaterPolicy.arn,
});

Reminder, we defined the IAM Role in our production AWS account, which allows any principal in the AWS bastion account (with the right tag, external ID) to assume that role. We then need to update the bastion account to grant the sts:AssumeRole action on that IAM Role so it can be used! That isn’t covered in this post.

Assuming the IAM Role

Finally! With everyone in place, we can now use the IAM Role.

The following is a set of commands you can run on your CI/CD worker to exchange the low-privilege IAM User credentials for the specific IAM Role we just created in the production AWS account. But there are many alternatives for programmatically assuming an IAM Role, such as using the AWS_PROFILE environment variable) or the assume-role tool.

# Use the AWS credentials passed to the CI system and available as environment variables.
# These are the low-privilege credentials that automatically get rotated.
aws configure set aws_access_key_id ${AWS_ACCESS_KEY_ID}
aws configure set aws_secret_access_key ${AWS_SECRET_ACCESS_KEY}
aws configure set region "${AWS_REGION:-us-west-2}"

# Assume the desired IAM Role. Note how we pass the stack name in the --external-id parameter,
# as required to assume the role.
readonly CREDS_JSON=$(aws sts assume-role \
 --role-arn "${ARN_OF_IAM_ROLE_TO_ASSUME}" \
 --role-session-name "<useful description of the context, such as CI/CD job ID>" \
 --external-id "${TARGET_STACK}")

# The `aws sts assume-role` command just returns a JSON blob. So we now tease out the actual
# access key and set the corresponding environment variables.
export AWS_ACCESS_KEY_ID=$(echo "${CREDS_JSON}" | jq ".Credentials.AccessKeyId" --raw-output)
export AWS_SECRET_ACCESS_KEY=$(echo "${CREDS_JSON}" | jq ".Credentials.SecretAccessKey" --raw-output)
export AWS_SESSION_TOKEN=$(echo "${CREDS_JSON}" | jq ".Credentials.SessionToken" --raw-output)

# If everything worked correctly, then running the get-caller-identity command will refer to
# the IAM Role and not our IAM User.
echo "Assumed AWS identity:"
aws sts get-caller-identity

Storing Sensitive Data using Pulumi

As a developer, the most critical security threat is access to your cloud account since that’s ultimately where your or your customer’s data is stored. But beyond that, there are places where Pulumi stores sensitive data.

Pulumi goes to great lengths to safeguard your data. Here we review where Pulumi interacts with your sensitive data and what you can do to safeguard it better.

Secrets in Configuration

A Pulumi stack can have configuration data associated with it, which allows you to parameterize cloud resources. For example, a machine-size configuration key might determine which class of virtual machine to use. A dev stack might only need a "t2.small" but your production instance might need something more powerful.

Configuration data is stored in a file like Pulumi.website-production.yaml should be checked in along with the source code for a stack, so that builds and stack updates are reproducible.

However, what if those configuration settings contain secrets? Like you need to store an API key to use a 3rd party API.

Pulumi supports encrypting sensitive configuration data. You just need to add the --secret flag.

# Securely add an API key to the stack's configuration.
pulumi config set api-key "hunter2" --secret

For stacks hosted on the Pulumi Service, the default is that your configuration data is encrypted using a key specific to your stack. (So the ciphertext stored in the Pulumi.yaml file is safe to check into your source tree, since it cannot be copied/decrypted for another stack.)

Secrets in Checkpoint Files

Pulumi keeps track of your cloud resources in a something called a checkpoint file, and that too might contain sensitive information. For example, a Pulumi resource might have a "password" output property.

Pulumi has support to mark that resource output as “secret” and make sure that it is encrypted within the checkpoint file. (So if you were to look at the checkpoint file contents via pulumi stack export, you would not be able to recover the data.

Just like for secret configuration values, the default for stacks hosted on the Pulumi Service is to encrypt this data using a key that Pulumi manages and is specific to your stack.

Custom Secrets Providers

We at Pulumi take great care in safeguarding your data, especially configuration or checkpoint data that is marked as sensitive. However, you might feel more comfortable if your stack’s data were encrypted using a key that you controlled. (So even if the data stored in the Pulumi Service were available, it would be useless without your specific key.)

If that’s the case, then you can use a configurable secrets provider, and swap out the default “Pulumi Service managed” encryption scheme for your own. (And we won’t take it personally, promise.)

When you create a new stack using pulumi stack init, you can optionally specify a --secrets-provider flag. That will determine where and how secrets get managed on your stack.

For example, to use your own KMS key for encrypting data, you can pass the secrets provider "awskms://alias/ExampleAlias?region=us-east-1". The Pulumi command-line will work and behave the same. Except that whenever it needs to encrypt or decrypt some data, it will refer to the custom provider.

As you would expect, if you do use a custom secrets provider and lose access to your encryption key, then there is no way to recover the encrypted data stored on your stack.

For more information on custom secret providers, see Peace of Mind with Cloud Secret Providers or Managing Secrets with Pulumi.

Wrapping Up

Securely managing any sensitive data is challenging to get right, especially if there are many moving parts, such as when coordinating access between your AWS account, CI/CD provider, and your Pulumi stack.

In this blog series, we walked through the best practices in creating, securing, and managing access to an AWS account using IAM Users and Roles. We then reviewed the specifics of where and how Pulumi stores sensitive data, in case you want to take more control over your data.

If you have any other questions, corrections, or other ideas you’d like to discuss about Pulumi, Continuous Delivery, or security, please join the conversation over on the Pulumi Community Slack.

Managing AWS Credentials on CI/CD: Part 2

Chris Smith — Thu, 26 Mar 2020 00:00:00 +0000

This article is the second part of a series on best practices for securely managing AWS credentials on CI/CD. In this article, we go in-depth on providing AWS credentials securely to a 3rd party and introduce a Pulumi program to automate rotating access keys.

NOTE: These recommendations do not apply if you are running a CI/CD system within your AWS account, e.g., running a Jenkins server on EC2 or using AWS CodeDeploy. In those cases, please refer to AWS’s documentation for how to assume IAM Roles when running on an EC2 instance instead.

Provide IAM credentials to your CI/CD system

In the first post in our series, we created a dedicated IAM User to perform updates to AWS resources within your CI/CD system. The next step is to pass the AWS access keys for that user to your CI/CD system.

We need to take great caution. AWS’s documentation states, “Do not provide your access keys to a third party”.

Providing AWS access keys to any system is dangerous because of the risk that a bad actor could obtain those keys and do something nefarious. Even if your CI/CD service takes great care to protect your secrets, those AWS credentials could be inadvertently exposed in debugging output, system logs, or in some other way.

This is one of the main advantages of performing continuous deployment from within your AWS account. For example, running Jenkins on an EC2 instance you manage, or using the AWS CodeDeploy service. Because when performing a deployment from within an AWS compute environment, you can use built-in mechanisms to obtain credentials securely.

However, there are many reasons to use a hosted service for performing your CI/CD, such as developer productivity, ease of use, performance, or simply because you don’t want to recreate your existing deployment workflows.

Ultimately, it is essential to understand the risks involved and how to mitigate them.

Properly Storing Credentials

If you choose to provide your CI/CD with credentials, the most important thing is to mark them as secret. Your CI/CD provider typically has built-in support for handling “secure variables” or other sensitive information. As opposed to general, configuration data or environment variables.

User-supplied configuration values are typically write-only and only accessible by the CI/CD worker jobs at runtime. For example, Travis CI has encrypted environment variables or secure environment variables in CircleCI.

Please consult your CI/CD provider’s documentation for how to pass and store sensitive information appropriately.

Why not use a “secrets manager” service?

A common question when discussing is how to pass access keys to a CI/CD system securely:

Rather than giving your CI/CD provider AWS credentials, why not have your CI/CD system obtain credentials from a specialized “secrets manager” service?

In other words, if you choose not to trust your CI/CD system with this data, can you instead trust some other system dedicated for securely storing and retrieving sensitive information?

It sounds like a good idea on the surface but doesn’t seem to make your data any more secure.

Abstractly, the difference here is that the credentials are provided on-demand, rather than being available to the CI/CD job when it starts (and stored via the CI/CD provider). Instead, your CI/CD job would obtain credentials from the “secrets manager” only when needed.

There are some advantages to this approach, such as providing a clear audit trail for access and more control over the distribution of sensitive information.

However, the secrets provider system needs to be presented with some form of credentials. And those credentials need to be available to your CI/CD environment. So using a secrets manager leaves you in the same place you started, i.e., needing to provide sensitive data to your CI/CD provider.

Also, by adding a dependency on a secrets manager, you introduce additional risks. Not only do you need to be even more security-conscious about that secrets manager, but it also needs to be highly available. Any outage for that service would mean that you would be unable to perform deployments!

So it does not seem that using a secrets manager to dole out AWS credentials to your CI/CD system is a good practice to follow. Or, at the very least, makes some tradeoffs without fundamentally making your approach to CI/CD any more or less secure.

Instead, it is better to focus on other ways to reduce the risk if those AWS credentials do get exposed.

Automate Rotating and Revoking AWS Credentials

When providing credentials to a 3rd party, rather than hoping it will be 100% secure forever (which is impossible), we can make those credentials volatile. If we regularly invalidate and rotate the credentials supplied to the CI/CD system, we can dramatically reduce the impact of any accidental disclosure. Even if the IAM User’s credentials were leaked, by the time they were discovered and used, they would no longer be valid.

The AWS security blog describes how to rotate access keys for IAM Users, by using the following steps:

Create a second access key in addition to the one in use.
Update all your applications to use the new access key.
Change the state of the previous access key to inactive.
Verify everything is still working as expected.
Delete the inactive access key.

That all sounds simple enough, but it’s certainly tedious. And if you need to repeat the process across dozens, if not hundreds of different CI/CD pipelines, you need a better solution. Fortunately, Pulumi provides an excellent and extensible way for writing a serverless program to automate rotating credentials.

The next few sections describe a simple infrastructure application for automating AWS IAM credential rotation. You can see the full application on GitHub at chrsmith/pulumi-aws-travis-cicd-demo.

AWS Credential Rotator 9000

The affectionally titled AWS Credential Rotator 9000 is a simple, serverless Pulumi application for rotating AWS access keys.

It creates an AWS lambda function that’s triggered on a regular schedule, e.g., every 30 minutes, and performs the next step in the sequence for rotating access keys as outlined above.

First, it creates a new access key and pushes the new value out. On the next iteration, it marks the older access key as “inactive.” On the following iteration, the inactive key is deleted. The process repeats, generating a new key and removing the inactive key.

Periodically Invoking an AWS Lambda

The heart of the application is triggering it to execute on a fixed interval. Thankfully this is super-easy to do using Pulumi since it allows you to seamlessly blend your “cloud infrastructure” with “code” in a natural way. The user guide for Pulumi Crosswalk for AWS has more information on serverless eventing if you would like to learn more.

The following snippet is the core part of the credential rotator app. We define a function to handle the logic of key rotation in rotateIAMUserKeys. Then we create an AWS Lambda resource named lambda. Finally, the triggerSchedule resource invokes our lambda on a fixed schedule, thereby ensuring that the key rotation process goes on indefinitely. You can see the full code here.

async function rotateIAMUserKeys(iamUser: aws.iam.User, onNewCredentials: (key: string, secret: string) => Promise<void>) {
 ...
}

const lambda = new aws.lambda.CallbackFunction<aws.cloudwatch.EventRuleEvent, void>(
 "keyRotatorLambda",
 {
 callback: async (e) => {
 await rotateIAMUserKeys(userToRotate, async (newKey: string, newSecret: string) => {
 console.log("onNewCredentials callback called!");
 console.log(`A new AWS access key "${newKey}" has been created.`);
 });
 },

 role: lambdaRole,
 runtime: "nodejs10.x",
 });

const triggerSchedule = aws.cloudwatch.onSchedule(
 "keyRotatorScheduler", "rate(1 hour)", lambdaCallbackFn);

When writing reusable infrastructure components in Pulumi however, it is helpful to organize things into a custom resource.

For example, we can bundle together the AWS Lambda, CloudWatcn schedule, and the associated IAM policies into a single conceptual resource AccessKeyRotator. Bundling resources allows for the code reuse.

const rotator = new AccessKeyRotator("rotator", {
 interval: config.require("rate"),
 user: user,
 credentialPusher: demoTravisCIPusher,
})

Pushing Credentials

With the mechanics of updating an AWS access key out of the way, the next step is to notify dependent systems what the new access key should be. The key rotator app has a CredentialPusher abstraction to provide a pluggable way for you to send credentials to where they need to be. The example on GitHub only supports updating a Travis CI project (See credential-pusher-travis.ts.), but it could be easily extended to support other CI/CD systems.

The following snippet shows how the CredentialPusher is configured in the key rotator app. We load a Travis CI access token from the Pulumi stack’s configuration, and hard-code the specific set of projects and encrypted environment variables to store the new access key.

// A "credential pusher" is the component that pushes new AWS IAM credentials out to 3rd parties
// as the older ones get rotated. For demonstration purposes, this updates the Travis CI settings
// for the chrsmith/pulumi-aws-travis-cicd-demo repo. But you can imagine another implementation
// that pushes the new IAM credentials to GitLab CI, or updating multiple CI/CD pipelines.

const demoTravisCIPusher = new CredentialPusher(
 new TravisCIPusher(),
 {
 accessKey: config.require("travis-ci-token"),
 projects: [
 {
 project: "chrsmith/pulumi-aws-travis-cicd-demo",
 // In the Travis CI configuration for that GitHub repo, there are two
 // environment variables for storing the AWS credentials. So whenever the
 // AWS credentials get rotated, the job's configuration settings will be
 // updated to reflect the new values.
 accessKeyIDLocation: "AWS_ACCESS_KEY_ID",
 secretAccessKeyLocation: "AWS_SECRET_ACCESS_KEY",
 }
 ],
 });

Demo

To demonstrate the access keys rotation, we can examine the log files generated from AWS Lambda. These can be accessed from the command-line using the pulumi logs command.

Here’s a summary of the output for clarity:

 START RequestId: 9406913e-a9d0-46a4-b861-2efdee508b2b Version: $LATEST
 9406913e-a9d0-46a4-b861-2efdee508b2b INFO IAM User has 2 keys:
 9406913e-a9d0-46a4-b861-2efdee508b2b INFO - AKIASHIVKXX3QEQNZNX5 [Active] Tue Mar 24 2020 12:37:03 GMT+0000 (Coordinated Universal Time)
 9406913e-a9d0-46a4-b861-2efdee508b2b INFO - AKIASHIVKXX3SA2K3ME7 [Inactive] Tue Mar 24 2020 09:37:03 GMT+0000 (Coordinated Universal Time)
 9406913e-a9d0-46a4-b861-2efdee508b2b INFO Deleting older, inactive access key AKIASHIVKXX3SA2K3ME7 [Inactive]
 9406913e-a9d0-46a4-b861-2efdee508b2b INFO Key rotation step complete.
 END RequestId: 9406913e-a9d0-46a4-b861-2efdee508b2b

 START RequestId: 05644e4a-81a4-4197-8ad7-fc3517998270 Version: $LATEST
 05644e4a-81a4-4197-8ad7-fc3517998270 INFO IAM User has 1 keys:
 05644e4a-81a4-4197-8ad7-fc3517998270 INFO - AKIASHIVKXX3QEQNZNX5 [Active] Tue Mar 24 2020 12:37:03 GMT+0000 (Coordinated Universal Time)
 05644e4a-81a4-4197-8ad7-fc3517998270 INFO Created new key key AKIASHIVKXX3YX4IXRVH
 05644e4a-81a4-4197-8ad7-fc3517998270 INFO Pushing out the new key to 3rd party services...
 05644e4a-81a4-4197-8ad7-fc3517998270 INFO Pushing new credentials to Travis CI project 'chrsmith/pulumi-aws-travis-cicd-demo'
 05644e4a-81a4-4197-8ad7-fc3517998270 INFO Updating env var 'AWS_ACCESS_KEY_ID' (b246ab3b-c4cb-e76f-a6fc-50f4cbf451c0)
 05644e4a-81a4-4197-8ad7-fc3517998270 INFO Updated AWS access key ID. Got response code (200)
 05644e4a-81a4-4197-8ad7-fc3517998270 INFO Updating env var 'AWS_SECRET_ACCESS_KEY' (ba9ac4be-fbb3-4da9-c5b8-35a1afe9e02b)
 05644e4a-81a4-4197-8ad7-fc3517998270 INFO Updated AWS secret access key. Got response code (200)
 05644e4a-81a4-4197-8ad7-fc3517998270 INFO Key rotation step complete.
 END RequestId: 05644e4a-81a4-4197-8ad7-fc3517998270

 START RequestId: ca204959-32a7-44a8-ba20-738d139542ba Version: $LATEST
 ca204959-32a7-44a8-ba20-738d139542ba INFO IAM User has 2 keys:
 ca204959-32a7-44a8-ba20-738d139542ba INFO - AKIASHIVKXX3YX4IXRVH [Active] Tue Mar 24 2020 15:37:03 GMT+0000 (Coordinated Universal Time)
 ca204959-32a7-44a8-ba20-738d139542ba INFO - AKIASHIVKXX3QEQNZNX5 [Active] Tue Mar 24 2020 12:37:03 GMT+0000 (Coordinated Universal Time)
 ca204959-32a7-44a8-ba20-738d139542ba INFO Invalidating older access key AKIASHIVKXX3QEQNZNX5
 ca204959-32a7-44a8-ba20-738d139542ba INFO Key rotation step complete.
 END RequestId: ca204959-32a7-44a8-ba20-738d139542ba

 START RequestId: de753fb4-0720-4f4f-a7aa-ad6b9d4c8832 Version: $LATEST
 de753fb4-0720-4f4f-a7aa-ad6b9d4c8832 INFO IAM User has 2 keys:
 de753fb4-0720-4f4f-a7aa-ad6b9d4c8832 INFO - AKIASHIVKXX3YX4IXRVH [Active] Tue Mar 24 2020 15:37:03 GMT+0000 (Coordinated Universal Time)
 de753fb4-0720-4f4f-a7aa-ad6b9d4c8832 INFO - AKIASHIVKXX3QEQNZNX5 [Inactive] Tue Mar 24 2020 12:37:03 GMT+0000 (Coordinated Universal Time)
 de753fb4-0720-4f4f-a7aa-ad6b9d4c8832 INFO Deleting older, inactive access key AKIASHIVKXX3QEQNZNX5 [Inactive]
 de753fb4-0720-4f4f-a7aa-ad6b9d4c8832 INFO Key rotation step complete.
 END RequestId: de753fb4-0720-4f4f-a7aa-ad6b9d4c8832

As you can see, the AWS credentials for the IAM User are automatically updated every few hours. And whenever a new access key is created, the value is pushed out to the impacted Travis CI projects automatically.

Wrapping Up

In this post, we covered some of the things to consider when providing AWS credentials to your CI/CD system. (In short, be cautious and follow best practices.)

We then showed how relatively easy it is to stand up a Pulumi infrastructure application for automatically rotating AWS access keys and updating your CI/CD system. By rotating credentials, you can limit the impact if the value is inadvertently disclosed.

At this point, we’ve now given access keys to a CI/CD system for a low-privilege AWS IAM User account. But we still can’t do anything with it. The IAM User whose credentials we have given to the CI/CD system doesn’t have access to your production AWS account.

In the next post, we’ll go into the details about IAM Roles, and they can be used to securely and temporarily gain access to additional resources. This is how our low-privilege IAM User can access needed to update production data for your CI/CD pipelines.

Managing AWS Credentials on CI/CD: Part 1

Chris Smith — Thu, 12 Mar 2020 00:00:00 +0000

Continuous delivery requires providing highly sensitive credentials to your deployment pipeline. Understanding the risks, mitigations, and best practices for handling those credentials can be difficult. In this guide, we describe the best practices for providing AWS credentials to a CI/CD system and to securely automate updating your cloud infrastructure using Pulumi.

You’ve heard about Infrastructure as Code and decided to try your hand at using Pulumi to manage cloud infrastructure using familiar programming languages. Naturally, you’re curious and eager to start updating your cloud infrastructure in your CI/CD pipeline – but then panic hits… Is updating your cloud infrastructure secure? If you start doing cloud deployments in your CI/CD system, doesn’t that mean you need to give it your production credentials? Aren’t those the most sensitive things your company has?!?

Take a deep breath. It’s going to be OK. You can securely provide AWS credentials to your CI/CD system and practice continuous delivery. This is the first post in a series going in-depth on how to do just that.

The goal of this article series is to give you a clear understanding of AWS credential management and how that relates to using Pulumi within a CI/CD environment. Once you have the AWS credentials in-place, you can then follow our Continuous Delivery guide for configuring your specific CI/CD service, whether you want to use CircleCI, GitLab CI, or Travis CI.

NOTE: These recommendations do not apply if you are running your own CI/CD system within your AWS account, e.g., running a Jenkins server on EC2 or using AWS CodeDeploy. In those cases, please refer to AWS’s documentation for how to assume IAM Roles when running on an EC2 instance instead.

Overview

The recommendations in this series describe a general “one-size fits most” approach for credential management, which requires a minimal amount of work to configure and maintain. Depending on your specific environment, needs, and constraints, there may be a better alternative for your use case.

Create a dedicated IAM User for your CI/CD

The first step for securely automating CI/CD is to create a dedicated IAM User for use in your CI/CD pipelines. (Sometimes referred to as a “robot account.”)

The following code snippet shows how to create a new AWS IAM User using Pulumi. (The code is in TypeScript, but you could another language like Python, C#, or Go as well.)

const user = new aws.iam.User("cicdUser", {
 name: "cicd-bot",
 tags: {
 "purpose": "Account used to perform Pulumi stack updates on CI/CD.",
 }
});

By default, that IAM User doesn’t have permissions to do anything. The recommended best practice is to use groups to assign permissions to IAM Users. So next, we will create a “Pulumi Stack updaters” IAM Group, and add our robot account as a member.

const group = new aws.iam.Group("pulumiStackUpdaters", {
 name: "PulumiStackUpdaters",
});

const groupMembership = new aws.iam.GroupMembership("cicdUserMembership", {
 group: group.name,
 users: [ user.name ],
});

Group Permissions

The next part is where things start to get tricky. We now need to grant permissions to this IAM Group’s members so they can access various APIs and resources. (For example, if your Pulumi program will need to create or update EC2 instances, then you will need to provide AWS credentials that contain the ec2:StopInstances or ec2:StartInstances action.)

But rather than grant the newly created cicd-bot IAM User those permissions, we will instead only grant them access to the sts:AssumeRole action. This is an AWS API that allows an IAM User to gain temporary credentials that have a different set of permissions associated with them. (That is, the permissions associated with an IAM Role.)

We’ll go into more detail about the exact difference between an IAM User and an IAM Role in the next post in the series since it isn’t entirely obvious why one would be more secure than the other.

const currentAwsIdentity = aws.getCallerIdentity();

const groupPolicy = new aws.iam.GroupPolicy("pulumiStackUpdatersPolicy", {
 group: group.name,
 policy: {
 Version: "2012-10-17",
 Statement: [{
 Action: [
 // Allow anybody (i.e., members of the group) to call the sts:AssumeRole API.
 // This allows them to "assume the role" of a more permissive IAM Role
 // when they go to update a stack later.
 "sts:AssumeRole",
 ],
 Effect: "Allow",
 // This is the set of resources that the "sts:AssumeRole" operation could be
 // performed on, which is to say any IAM role in the current AWS account.
 Resource: pulumi.interpolate `arn:aws:iam::${currentAwsIdentity.accountId}:role/*`,
 Sid: ""
 }],
 },
});

That’s it for the first post in our series. We’ve taken the first step and created an IAM User specifically for performing updates to AWS resources using Pulumi.

In the next post, we will review over the security concerns when passing AWS credentials to your CI/CD provider, as well as show a Pulumi program that you can use to rotate the AWS credentials for a user automatically.

Continuous Delivery on Octopus Deploy using Pulumi

Sophia Parafina — Tue, 22 Oct 2019 00:00:00 +0000

Continuous delivery is about making changes in your application and getting them into production securely, quickly, and consistently. Pulumi’s infrastructure as code approach uses source code to model cloud resources, making it ideal for continuous delivery. Your infrastructure code can share the same process as your application code including running unit and integration tests, performing code reviews via Pull Requests, and examining your infrastructure using linters or static analysis tools. Like your application, your cloud infrastructure can be validated and tested before deploying to production. Pulumi can integrate into any CI/CD system (such as Jenkins, Azure DevOps, CircleCI, TravisCI, Codefresh, and others), so let’s take a look at how to implement a deployment server, using Octopus as an example.

Octopus Deploy is a deployment automation server that integrates with your existing build pipeline or through a REST API. You can select the environments for deploying your software, and who has rights to complete a deployment. For example, the QA team can deploy to the Test environment but not to Development or Production. Different team members may trigger deployments, and the deployment process is always consistent.

Continuous Delivery with Octopus Deploy

Deployment targets are grouped in environments in Octopus. Environments represent different stages of the deployment pipeline that your software passes through on its way to a release. A common practice is to create unique Development, Test, and Production environments. Octopus Deploy lets you deploy the correct versions of your software, with the right configuration for the appropriate environment.

Pulumi’s guide to implementing Octopus Deploy walks you through the process of implementing an environment for deploying a Python flask application on AWS Fargate. Before starting, make sure you have met the prerequisites.

Octopus environments are analogous to Pulumi stacks. If you have Development, Test, and Production Octopus environments, you would create a Pulumi stack for each environment. The application being deployed is a Pulumi project. Octopus uses packages that contain the source code bundled in a supported format such as .zip files, gzip or bzip tar files, Java archive formats such as .jar or .war, and Docker images. Pulumi apps can be packaged in the desired format and extracted onto a Octopus worker that the Pulumi CLI can access.

In Octopus, a Process is made up of the steps needed to execute a project. To deploy a Pulumi package, additional steps are needed to extract the code. You can add a step to a process by using the Octopus menu. In the Process section, select Add Step to add the package. You will need to specify the package feed, the package id and a package name. The step can be further customized by adding a script that specifies where to extract the package.

To deploy your Pulumi package, select Add Step again and search for the Pulumi template from the Community Library Steps. Fill out the template with the appropriate configuration parameters for your stack. Next, create a release from by selecting Releases and click Create Release and follow the instructions.

Using a CI/CD to build and deploy Pulumi applications can be a straightforward process. Try deploying a Pulumi application with Octopus Deploy with our continuous delivery guide.

Continuous Delivery with GitLab and Pulumi on Amazon EKS

Nishi Davidson — Wed, 22 May 2019 00:00:00 +0000

In this blog, we will work through an example that shows how to use Pulumi to enable GitLab-based continuous delivery with your Kubernetes workloads on Amazon EKS. This integration will work just as seamlessly for any Kubernetes cluster, including Azure AKS or Google GKE, using the relevant Pulumi libraries for Azure and Google Cloud.

Prerequisites

An account on https://app.pulumi.com with an organization.
The latest pulumi CLI. Installation instructions are here.
A bare repository. Set the remote URL to be your GitLab project.

Concepts in Pulumi

Organization of Pulumi Projects and Pulumi Stacks

All users in the Pulumi service will start with the hierarchy of an organization. This can be a specific GitHub, GitLab or Atlassian organization or your solo organization. Inside each organization, users create Pulumi projects and stacks.

Pulumi projects and stacks are flexible to accommodate the diverse needs across teams, applications, and infrastructure scenarios. Just like Git repos that work with varying approaches Pulumi projects and stacks allow you to organize your code within them. Immediate options include:

Monolithic project/stack structure: A single project defines the infrastructure and application resources for an entire vertical service as represented in the image below:

Micro-stacks project/stack structure: A project broken into separately managed smaller projects, often across different dimensions as represented in the image below:

Working with Inter-Stack Dependencies with the latter option is more suited in a production setup giving users more flexibility and boundaries between their teams. We will use this structure in our example below. For more information on Pulumi projects and stacks, please refer to our organizaing documentation.

Use Tags to group Pulumi Stacks as Environments:

Pulumi Stacks have associated metadata in the form of key/value tags.
You can assign custom tags to stacks (when logged into the Pulumi Service backend to customize how stacks are listed in the Pulumi Service.
- In our example below we have two environments prod and dev.
- To group stacks by environment we assign custom tags environment: prod and environment: dev to the respective stacks
- In the Pulumi Service, you’ll be able to group stacks by tag: environment:dev and tag: environment:prod.

Please read more about managing stack tags in Pulumi.

Let’s now work through our example with GitLab Pipelines.

Please replace the highlighted <org-name-in-pulumi> below with your Pulumi organization name to run through the steps successfully.

GitLab Pipeline by Environment

We created a GitLab Group called pulumi-gitlab.
We created three GitLab projects called sample-iam, sample-eks and sample-k8sapp. These projects match the project names in the Pulumi Service.
We have two pipelines: environment:dev and environment:prod. In the two pipelines, we have a total of six pulumi stacks:
- <org-name-in-pulumi>/sample-iam/dev, and <org-name-in-pulumi>/sample-iam/prod.
- <org-name-in-pulumi>/sample-eks/dev and <org-name-in-pulumi>/sample-eks/prod.
- <org-name-in-pulumi>/sample-k8sapp/dev and <org-name-in-pulumi>/sample-k8sapp/prod.
<org-name-in-pulumi>/sample-iam/dev stack will trigger the downstream stack <org-name-in-pulumi>/sample-eks/dev provided the cycle of pulumi preview and pulumi deploy completes without any failure.

Similarly, <org-name-in-pulumi>/sample-eks/dev will trigger the downstream stack <org-name-in-pulumi>/sample-k8sapp/dev provided the cycle of pulumi preview and pulumi deploy completes without any failure.
To use Pulumi within GitLab CI, there are a few environment variables you’ll need to set for each build.
- The first is PULUMI_ACCESS_TOKEN, which is required to authenticate with pulumi.com in order to perform the preview or update. You can create a new Pulumi access token specifically for your CI/CD job on your Pulumi Account page.
- Next, you will also need to set environment variables specific to your cloud resource provider. For example, if your stack is managing resources on AWS, AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

Create Pulumi stacks and push the files to your GitLab project

If you run pulumi from any branch other than the master branch, you will hit an error that the PULUMI_ACCESS_TOKEN environment variable cannot be accessed. You can fix this by specifying a wildcard regex to allow specific branches to be able to access the secret environment variables. Please refer to the GitLab documentation to understand this better.

First we set up three Pulumi stacks: sample-iam; sample-eks and sample-k8sapp with group stack tag: environment:dev

Step 1: Create the pulumi stack “sample-IAM” and set stack tag “key:value” = “environment:dev”. We update the index.ts file with the relevant code block as shown below and run pulumi up.

$ pulumi new aws-typescript --dir <org-name-in-pulumi>/sample-iam/dev
$ cd <org-name-in-pulumi>/sample-iam/dev

Let’s run pulumi up with the following index.ts file.

import * as aws from "@pulumi/aws";
function createIAMRole(name: string): aws.iam.Role {
 // Create an IAM Role
 return new aws.iam.Role(`${name}`, {
 assumeRolePolicy: `{
 "Version": "2012-10-17",
 "Statement":[
 {
 "Sid": "",
 "Effect": "Allow",
 "Principal": {
 "AWS": "arn:aws:iam::153052954103:root"
 },
 "Action": "sts:AssumeRole"
 }
 ]
 }`,
 tags: {
 "clusterAccess": `${name}-usr`,
 },
 });
}

// Administer automation role for use in pipelines, e.g. gitlab CI, Teamcity, etc.
export const automationRole = createIAMRole("automationRole");
export const automationRoleArn = automationRole.arn;

We then group the stack by initializing it with a new stack tag “key:value” = “environment:prod” and run pulumi up with the same index.ts file.

$ pulumi up

# Initialize new pulumi stack in the format:
# pulumi stack init <org-name-in-pulumi>/<project>/<stack>
$ pulumi stack init <org-name-in-pulumi>/sample-iam/prod
$ pulumi stack tag set environment prod
$ pulumi up

Step 2: Create the pulumi stack sample-eks and set stack tag “key:value” = “environment:dev”.

$ pulumi new aws-typescript --dir <org-name-in-pulumi>/sample-eks/dev
$ cd <org-name-in-pulumi>/sample-eks/dev

Update the index.ts file with the relevant code block as shown below:

import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";
import * as eks from "@pulumi/eks";
import * as k8s from "@pulumi/kubernetes";
import * as pulumi from "@pulumi/pulumi";

const env = pulumi.getStack();
const iamstack = new pulumi.StackReference(`<org-name-in-pulumi>/sample-iam/${env}`);

const automationRoleArn = iamstack.getOutput("automationRoleArn")

/* * Single step deployment of EKS cluster with the most important variables and simple function to create two namespaces */

const vpc = new awsx.ec2.Vpc("vpc", {});
const cluster = new eks.Cluster("eks-cluster", {
 vpcId : vpc.id,
 subnetIds : vpc.publicSubnetIds,
 instanceType : "t2.medium",
 nodeRootVolumeSize: 200,
 desiredCapacity : 1,
 maxSize : 2,
 minSize : 1,
 deployDashboard : false,
 vpcCniOptions : {
 warmIpTarget : 4,
 },
 roleMappings : [
 // Map IAM role arn "automationRoleArn" to the k8s user with name "automation-usr", e.g. gitlab CI
 {
 groups : ["pulumi:automation-grp"],
 roleArn : automationRoleArn,
 username : "pulumi:automation-usr",
 },
 ],
});

export const clusterName = cluster.eksCluster.name;

/* * Single Step deployment of k8s RBAC configuration */

new k8s.rbac.v1.Role("automationRole", {
 metadata: {
 name: "automationRole",
 namespace: "automation",
 },
 rules: [{
 apiGroups: ["*"],
 resources: ["*"],
 verbs: ["*"],
 }]
}, {provider: cluster.provider});

new k8s.rbac.v1.RoleBinding("automation-binding", {
 metadata: {
 name: "automation-binding",
 namespace: "automation",
 },
 subjects: [{
 kind: "User",
 name: "pulumi:automation-usr",
 apiGroup: "rbac.authorization.k8s.io",
 }],
 roleRef: {
 kind: "Role",
 name: "automationRole",
 apiGroup: "rbac.authorization.k8s.io",
 },
}, {provider: cluster.provider});

export const kubeconfig = cluster.kubeconfig.apply(JSON.stringify)

Let’s download the additional npm packages for EKS and Kubernetes and run pulumi up and initialize a new stack tag “key:value” = “environment:prod” to run pulumi up with the same index.ts file.

$ npm install --save @pulumi/eks @pulumi/kubernetes
$ pulumi up
# Initialize new pulumi stack in the format:
# pulumi stack init <org-name-in-pulumi>/<project>/<stack>
$ pulumi stack init <org-name-in-pulumi>/sample-eks/prod
$ pulumi stack tag set environment prod
$ pulumi up

Step 3: Create the pulumi stack “sample-k8sapp” and set stack tag “key:value” = “environment:dev”. We will then update the index.ts file with the relevant code block as shown below:

$ pulumi new aws-typescript --dir <org-name-in-pulumi>/sample-k8sapp/dev
$ cd <org-name-in-pulumi>/sample-k8sapp/dev

import * as aws from "@pulumi/aws";
import * as docker from "@pulumi/docker";
import * as k8s from "@pulumi/kubernetes";
import * as pulumi from "@pulumi/pulumi";

const env = pulumi.getStack();
const eksCluster = new pulumi.StackReference(`<org-name-in-pulumi>/sample-eks/${env}`);

const kubeconfig = eksCluster.getOutput("kubeconfig");

const k8sProvider = new k8s.Provider("eks-cluster", {
 kubeconfig: kubeconfig,
 });

/* * Single step deployment of one docker container in ECR */

function getImageRegistry(repo: aws.ecr.Repository) {
 return repo.registryId.apply(async registryId => {
 if (!registryId) {
 throw new Error("Expected registry ID to be defined during push");
 }
 const credentials = await aws.ecr.getCredentials({ registryId: registryId });
 const decodedCredentials = Buffer.from(credentials.authorizationToken, "base64").toString();
 const [username, password] = decodedCredentials.split(":");
 if (!password || !username) {
 throw new Error("Invalid credentials");
 }
 return {
 server: credentials.proxyEndpoint,
 username: username,
 password: password,
 };
 });
}

const ecr1 = new aws.ecr.Repository("breathe");
const image1 = new docker.Image("breathe", {
 imageName: ecr1.repositoryUrl,
 build: {
 context: "./app",
 cacheFrom: true,
 },
 registry: getImageRegistry(ecr1),
});

// Declare the docker container based deployment
const appLabels = { app: appName };
const breathecontainer = new k8s.apps.v1beta1.Deployment(appName, {
 spec: {
 selector: { matchLabels: appLabels },
 replicas: 1,
 template: {
 metadata: { labels: appLabels },
 spec: { containers: [{ name: appName, image: image1.imageName }] }
 }
 },
 }, { provider: k8sProvider });

We download the additional npm packages for EKS and Kubernetes and run pulumi up and group a new stack by initializing it with a new stack tag “key:value” = “environment:prod” and run pulumi up with the same index.ts file.

$ npm install --save @pulumi/kubernetes @pulumi/docker
$ pulumi up

# Initialize new pulumi stack
$ pulumi stack init <org-name-in-pulumi>/sample-eks/prod
$ pulumi stack tag set environment prod
$ pulumi up

Using GitLab Pipelines with the six Pulumi stacks in `environment:dev` and `environment:prod`

GitLab pipelines are configured using .gitlab-ci.yml files in the root of each repository. GitLab Silver and above is capable of running pipelines that cross project boundaries, so we will be using that to construct our pipeline.

All three .gitlab-ci.yml files that we use are very similar in structure. The base one,sample-iam, looks like this:

image:
 name: pulumi/pulumi:v0.17.10
 entrypoint:
 - '/usr/bin/env'
 - 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'

stages:
 - preview
 - update
 - downstream

Pulumi Preview:
 stage: preview
 script:
 - npm ci
 - pulumi stack select pulumi/sample-iam/$DEPLOY_ENVIRONMENT
 - pulumi preview

Pulumi Update:
 stage: update
 script:
 - npm ci
 - pulumi stack select pulumi/sample-iam/$DEPLOY_ENVIRONMENT
 - pulumi update --skip-preview

Update EKS:
 stage: downstream
 trigger: pulumi-gitlab/sample-eks

This file describes a three-stage pipeline for the sample-iam project:

First, we run a preview for the requested deployment environment, failing the pipeline if the preview fails.
If the preview was successful, we run pulumi update, which deploys the IAM changes.
Finally, we trigger the pipeline in pulumi-gilab/sample-eks, which triggers the next pipeline in our pipeline daisy chain illustrated in the above image.

Despite being powerful, conceptually this setup is quite simple and doesn’t require much code to get right.

Upon a successful update, each tier’s pipeline will trigger a pipeline for the tiers that depend on it. Pulumi’s StackReference feature ensures that the dependent tiers receive new copies of the outputs exported from the IAM stack, so the deployment flows naturally through the pipeline!

This brings us to the end of our CD solution with Pulumi and GitLab on Amazon EKS. For more examples, refer to the Pulumi examples repository. Refer to my previous post on Amazon EKS and k8s RBAC in Pulumi.

CI/CD Made Easy with Pulumi and Azure Pipelines

Praneet Loke — Mon, 06 May 2019 00:00:00 +0000

Azure DevOps is very popular among teams that want a single place to manage their development pipelines, Git repositories, builds, releases, and test plans. Pulumi’s open-source tools are a great choice for developers and operators deploying infrastructure as code on Azure. With these two tools at hand, adopting CI and CD for your Azure infrastructure is just a few steps away for you and your teams.

To make it easy to use Pulumi with Azure, we are announcing an open-source task extension for Azure Pipelines! The task extension will manage the installation of the Pulumi CLI, and run the Pulumi commands you specify against your stack.

You can install the task extension directly from the Visual Studio Marketplace. Click the “Get It Free” button to install the extension into your Azure DevOps organization. If you do not have the permissions to add it to your organization, please contact your org admin to have the extension installed. The source code for the open source task extension can be found on GitHub.

To use the Pulumi task extension in Azure Pipelines, there are two options. The full example is in the source repo. For contrast, without this task extension, here’s how you would achieve the same, but with scripts.

OPTION 1: Using the Classic Editor Console

Pasted below is a typical view of the Editor User Interface. Click on the "+" button next to the job. As visible in the screenshot, the job is called “Agent job 1”. The agent pool you choose does not matter. The Pulumi task can run on both Linux as well as Windows images.

Search for the Pulumi task by typing in the filter input on the right. Click the Add button.

Fill in the required information. You can learn about creating service connections for Azure here.

If you want to understand all the commands available with the Pulumi CLI, click on the information icon (i) next to Pulumi Command, and click on the help link to go to our quick reference page.

Option 2: Using a YAML File

Here’s an example of the Pulumi task extension defined in YAML configuration.

- task: Pulumi@0
 inputs:
 azureSubscription: '<your-service-connection-name>'
 command: "up"
 args: "--yes --skip-preview"
 cwd: "./"
 # Replace this with your org's name.
 stack: "<org-name>/pulumi-az-pipelines-task-example/dev"

Moving to Production

Once you’ve configured your Pulumi task extensions for your Azure Pipelines, you can now more easily integrate Pulumi into your CI/CD pipeline, and take advantage of previews to infrastructure changes in pull requests, push-to-deploy, and ultimately removing the friction for your DevOps.

As always, we’d love to hear what you think. Say 👋 in the Pulumi Community Slack or drop us a line.

Want to learn more? Check out this post from Mikhail Shilkov for a detailed look at deploying applications on Azure with Pulumi. Also, keep an eye on our blog posts this week for more posts about Azure.

Testing Your Infrastructure as Code with Pulumi

Joe Duffy — Wed, 17 Apr 2019 00:00:00 +0000

Some parts of this blog post are out-of-date. Please refer to our Testing Guide for the updated overview and tutorials.

Using Pulumi and general purpose languages for infrastructure as code comes with many benefits: leveraging existing skills and knowledge, eliminating boilerplate through abstraction, and using the same ecosystem of tools like IDEs and linters that your team already knows and loves. In general, these are all attributes of software engineering, which not only make us more productive, but also improve the quality of our code. It’s only natural, therefore, that using general purpose languages unlocks another important software engineering practice: testing.

In this article, we will see the many ways in which Pulumi lets us test our infrastructure as code.

Why Test Your Infrastructure?

Before jumping into the details, it’s worth asking: Why test your infrastructure, anyway? There are many reasons to do this, however here are a few examples:

Unit testing individual functions or pieces of logic in your program
Ensuring that the program’s desired infrastructure state meets certain constraints
Catching common mistakes, like forgetting to encrypt your storage buckets, or exposing virtual machines unprotected to the open Internet
Ensuring that the provisioned infrastructure has certain desired attributes
Doing runtime testing of the application logic that is running inside of your compute infrastructure, to ensure that it is healthy after your infrastructure has been provisioned

As we see, there’s a broad spectrum of infrastructure testing options available. Pulumi has mechanisms available for testing at each point along this spectrum. Let’s dig in and see how.

Unit Testing

Pulumi programs are authored in a general purpose language like JavaScript, Python, TypeScript, or Go. The full power of each language is available, including access to tools and libraries for that runtime, including testing frameworks. Pulumi is also multi-cloud, meaning the full underlying platform for each cloud is available too, and therefore available for testing.

(Despite being multi-language and multi-cloud, for the purposes of this article let’s assume we are using JavaScript, with Mocha for testing, and targeting AWS. Python unittest, Go’s test framework, or any of your favorite unit test frameworks in your language of choice would do, and of course Pulumi works great with Azure, Google Cloud, and Kubernetes, among others.)

As we’ve seen, there are several reasons you may want to test your infrastructure code. One of them is basic unit testing. Because your code may have functions – say, to compute CIDR blocks, dynamically compute names and tags, etc. – you probably want to test them. Doing this is the same as writing ordinary unit tests against any program in your language as choice.

Going one step further, you may want to test the way your program allocates resources. To illustrate, let’s imagine we want to create a simple EC2-based webserver, but want to ensure:

Instances have a Name tag.
Instances must not use an inline userData script – we must use an AMI (image).
Instances must not have SSH open to the Internet.

Our running example is loosely based on our aws-js-webserver example:

index.js:

let aws = require("@pulumi/aws");

let group = new aws.ec2.SecurityGroup("web-secgrp", {
    ingress: [
 { protocol: "tcp", fromPort: 22, toPort: 22, cidrBlocks: ["0.0.0.0/0"] },
 { protocol: "tcp", fromPort: 80, toPort: 80, cidrBlocks: ["0.0.0.0/0"] },
    ],
});

let userData =
`#!/bin/bash echo "Hello, World!" > index.html nohup python -m SimpleHTTPServer 80 &`;

let server = new aws.ec2.Instance("web-server-www", {
    instanceType: "t2.micro",
    securityGroups: [ group.name ], // reference the group object above
    ami: "ami-c55673a0" // AMI for us-east-2 (Ohio),
    userData: userData // start a simple web server
});

exports.group = group;
exports.server = server;
exports.publicIp = server.publicIp;
exports.publicHostName = server.publicDns;

This is a basic Pulumi program: it simply allocates an EC2 security group and instance. Notice, however, that we are violating all three of the rules stated above – let’s write some tests!

Writing the Tests

The overall structure and scaffolding of our tests will look like any ordinary Mocha testing:

ec2tests.js

let assert = require("assert");
let mocha = require("mocha");
let pulumi = require("@pulumi/pulumi");
let infra = require("./index");

describe("Infrastructure", function() {
    let server = infra.server;
    describe("#server", function() {
 // TODO(check 1): Instances have a Name tag.
 // TODO(check 2): Instances must not use an inline userData script.
    });
    let group = infra.group;
    describe("#group", function() {
 // TODO(check 3): Instances must not have SSH open to the Internet.
    });
});

Now let’s implement our first test: ensuring that instances have a Name tag. To verify this we simply need to grab hold of the EC2 instance object, and check the relevant tags property:

// check 1: Instances have a Name tag.
it("must have a name tag", function(done) {
 pulumi.all([server.urn, server.tags]).apply(([urn, tags]) => {
     if (!tags || !tags["Name"]) {
     done(new Error(`Missing a name tag on server ${urn}`));
     } else {
      done();
     }
    });
});

This looks like a normal test, with a few noteworthy pieces:

Since we’re querying resource state before we’ve done a deployment, our tests are always effectively running in “plan” (or “preview”) mode. Thus, there are many properties whose values will simply not resolve yet or will be undefined. This includes any output properties computed by your cloud provider. That’s fine for these tests – we’re checking for valid inputs anyway. We’ll revisit this later when it comes to integration testing.
Because all Pulumi resource properties are “outputs” – since many of them are computed asynchronously – we need to use the apply method to get access to the values. This is a lot like promises and the then function.
Because we are using multiple properties, mostly just so we can show the resource URN in the error message, we need to use the pulumi.all function to join all of them.
Finally, since these outputs are resolved asynchronously, we need to use Mocha’s built-in asynchronous test capability, with the done callback, or by returning a promise.

After we’ve gotten through that setup, we get access to the raw inputs as plain JavaScript values. The tags property is a map, so we just make sure it is (1) not falsey, and (2) not missing an entry for the Name key in the map. This is very basic, but we can check anything!

Now let’s write our second check. It’s even easier:

// check 2: Instances must not use an inline userData script.
 it("must not use userData (use an AMI instead)", function(done) {
 pulumi.all([server.urn, server.userData]).apply(([urn, userData]) => {
 if (userData) {
 done(new Error(`Illegal use of userData on server ${urn}`));
 } else {
 done();
 }
    });
});

And finally, let’s write our third check. It’s a bit more complex because we’re searching for ingress rules associated with a security group – of which there may be many – and CIDR blocks within those ingress rules – of which there may also be many. But it’s still easy:

// check 3: Instances must not have SSH open to the Internet.
it("must not open port 22 (SSH) to the Internet", function(done) {
 pulumi.all([ group.urn, group.ingress ]).apply(([ urn, ingress ]) => {
 if (ingress.find(rule =>
 rule.fromPort == 22 && rule.cidrBlocks.find(block => block === "0.0.0.0/0"))) {
     done(new Error(`Illegal SSH port 22 open to the Internet (CIDR 0.0.0.0/0) on group ${urn}`));
 } else {
     done();
 }
    });
});

That’s it – now let’s actually run the tests!

Running the Tests

For the most part, we can run the tests in the usual way in your test framework of choice. There is one thing specific to Pulumi we need to cover first, however.

Usually you run your Pulumi programs with the pulumi CLI. That configures the language runtime, monitors execution from the engine so that resource operations can be recorded and turned into a plan, and so on. There is one problem, however: When running under the watchful eye of your test framework, this connection to the CLI, and Pulumi’s engine, will not exist.

To work around this problem, we simply have to mock three pieces of information:

Your project name, which can be supplied with the PULUMI_NODEJS_PROJECT environment variable (or more generally PULUMI_<LANGUAGE>_PROJECT in other languages).
Your stack name, which can be supplied with the PULUMI_NODEJS_STACK environment variable (or more generally PULUMI_<LANGUAGE>_STACK in other languages).
Your stack configuration variables. These can be supplied using the PULUMI_CONFIG environment variable, and its format is a JSON map of configuration key/value pairs.

The program will emit some warning messages indicating the connection to the CLI/engine is not available at runtime. This is important to know, because your program won’t actually be deploying anything – which can come as a surprise if that’s not what you meant to do! To tell Pulumi that’s what you intended, you can set PULUMI_TEST_MODE to true.

Imagine we want a project name of my-ws, stack name of dev, and AWS region of us-west-2. The command line to run your Mocha tests would therefore be:

$ PULUMI_TEST_MODE=true \
 PULUMI_NODEJS_STACK="my-ws" \
 PULUMI_NODEJS_PROJECT="dev" \
 PULUMI_CONFIG='{ "aws:region": "us-west-2" }' \
 mocha tests.js

Running this will tell us that we have three failing tests, as we had hoped!

 Infrastructure
    #server
1) must have a name tag      2) must not use userData (use an AMI instead)
    #group
3) must not open port 22 (SSH) to the Internet
  0 passing (17ms)
  3 failing
 1) Infrastructure
       #server
 must have a name tag:
Error: Missing a name tag on server
urn:pulumi:my-ws::my-dev::aws:ec2/instance:Instance::web-server-www
 2) Infrastructure
       #server
 must not use userData (use an AMI instead):
Error: Illegal use of userData on server
urn:pulumi:my-ws::my-dev::aws:ec2/instance:Instance::web-server-www
 3) Infrastructure
       #group
 must not open port 22 (SSH) to the Internet:
Error: Illegal SSH port 22 open to the Internet (CIDR 0.0.0.0/0) on group

Let’s fix our program to comply:

"use strict";

let aws = require("@pulumi/aws");

let group = new aws.ec2.SecurityGroup("web-secgrp", {
    ingress: [
 { protocol: "tcp", fromPort: 80, toPort: 80, cidrBlocks: ["0.0.0.0/0"] },
    ],
});

let server = new aws.ec2.Instance("web-server-www", {
    tags: { "Name": "web-server-www" },
    instanceType: "t2.micro",
    securityGroups: [ group.name ], // reference the group object above
    ami: "ami-c55673a0" // AMI for us-east-2 (Ohio),
});

exports.group = group;
exports.server = server;
exports.publicIp = server.publicIp;
exports.publicHostName = server.publicDns;

And then rerun our tests:

 Infrastructure
    #server
      ✓ must have a name tag
      ✓ must not use userData (use an AMI instead)
    #group
      ✓ must not open port 22 (SSH) to the Internet
 3 passing (16ms)

They all passed – huzzah! ✓✓✓

Deployment Testing

This style of testing is powerful and lets us do white box testing of the innards of how our infrastructure code works. This style of testing is somewhat limited in what we can check, however. The tests run on an in-memory representation of Pulumi’s deployment plan – before actually performing the deployment – and so cannot test anything the deployment itself. For that, Pulumi has an integration test framework. The two approaches work great together!

Pulumi’s integration test framework is written in Go and is how we test much of our internal code. If the earlier unit test approach took more of a white box testing approach, the integration test framework is very much black box. (There are also options for thorough internals-oriented testing.) This framework has been built to take a directory containing a full Pulumi program and drive various lifecycle operations against it – such as deploying a new stack from scratch, updating it with variations, and tearing it down afterwards, potentially multiple times. We run these in checkin suites, on a recurring basis (such as nightly), and as stress tests.

(We have an open work item which will bring similar integration testing capabilities to each native language SDK. You can use the Go integration test framework no matter the language your Pulumi program is written in, however.)

By running a program through this integration test framework, you can ensure:

Your project’s code is syntactically well formed and runs without errors.
Your stack’s configuration and secrets settings work and are interpreted correctly.
Your project can be successfully deployed to your cloud provider of choice.
Your project can be successfully updated from its starting state to N other states.
Your project can be successfully destroyed and removed from your cloud provider.

As we will see soon, you can also leverage this framework to perform runtime validation.

A Basic Integration Test

To see this in action, we’ll look from the pulumi/examples repo, since our team and community here at Pulumi use that to validate our own examples in pull requests, checkins, and nightlies.

The following is a simplified test of our example that provisions an S3 bucket and some objects in Pulumi:

example_test.go:

package test

import (
    "os"
    "path"
    "testing"

    "github.com/pulumi/pulumi/pkg/testing/integration"
)

func TestExamples(t *testing.T) {
    awsRegion := os.Getenv("AWS_REGION")
    if awsRegion == "" {
 awsRegion = "us-west-1"
    }
    cwd, _ := os.Getwd()
    integration.ProgramTest(t, &integration.ProgramTestOptions{
 Quick: true,
 SkipRefresh: true,
 Dir: path.Join(cwd, "..", "..", "aws-js-s3-folder"),
 Config: map[string]string{
     "aws:region": awsRegion,
 },
    })
}

This test runs through a basic lifecycle of stack creation, updating, and destruction, for the aws-js-s3-folder example. It takes about a minute or so to report success:

$ go test .
PASS
ok ... 43.993s

There are many, many options to control the behavior of these tests. For a full set of options, see the ProgramTestOptions data structure. For instance, you can configure a Jaeger endpoint for tracing (Tracing), tell the harness to expect failure for negative testing (ExpectFailure), apply a series of “edits” to the program for a sequence of update state transitions (EditDirs), and more. Let’s see how to use these to do validate that we deployed what we expected.

Validating the Shape of Resources

All of the above integration ensures that our program “works” – as in, it does not crash. But what if we want to validate properties of the resulting stack? For example, that certain kinds of resources did (or did not) get provisioned, and that they have certain attributes.

The ExtraRuntimeValidation option for ProgramTestOptions allows us to look at the post-deployment state that Pulumi has captured, so that we can do extra validation on it. This includes a full snapshot of the resulting stack’s state, including configuration, exported outputs, all resources and their property values, and all inter-resource dependencies.

To see a basic example of this, let’s check that our program creates a single S3 Bucket:

integration.ProgramTest(t, &integration.ProgramTestOptions{
 // as before...
 ExtraRuntimeValidation: func(t *testing.T, stack integration.RuntimeValidationStackInfo) {
     var foundBuckets int
     for _, res := range stack.Deployment.Resources {
 if res.Type == "aws:s3/bucket:Bucket" {
      foundBuckets++
 }
     }
     assert.Equal(t, 1, foundBuckets, "Expected to find a single AWS S3 Bucket")
 },
})

Now when we run go test, it will not only run through the battery of lifecycle tests but will also, after successfully standing up the stack, perform the extra validation against the resulting state.

Runtime Testing

All of the testing thus far has been exclusively about deployment behavior and the Pulumi resource model. What if you want to test that your provisioned infrastructure is actually working, however? For example, that our VM is alive, our S3 bucket contains what we expect, and so on.

You may have already guessed how to accomplish some of this: the ExtraRuntimeValidation option for ProgramTestOptions is the perfect opportunity to do this. At this point, you are running an arbitrary Go test, with access to the full state of your program’s resources. This state includes information such as VM IP addresses, URLs, and everything you need to actually interact with your resulting cloud applications and infrastructure to perform verification.

For instance, our test program above exports the bucket’s websiteEndpoint property, as the stack output named websiteUrl, which is a full URL that we can use to fetch the bucket’s configured index document. Although we could just as well have dug through the state file to find the bucket and read this property directly, many times our stacks will export useful properties like this that we can use conveniently for verification:

    integration.ProgramTest(t, &integration.ProgramTestOptions{
     // as before ...
 ExtraRuntimeValidation: func(t *testing.T, stack integration.RuntimeValidationStackInfo) {
     url := "http://" + stack.Outputs["websiteUrl"].(string)
     resp, err := http.Get(url)
     if !assert.NoError(t, err) {
 return
     }
     if !assert.Equal(t, 200, resp.StatusCode) {
 return
     }
     defer resp.Body.Close()
     body, err := ioutil.ReadAll(resp.Body)
     if !assert.NoError(t, err) {
 return
     }
     assert.Contains(t, string(body), "Hello, Pulumi!")
 },
})

Like our previous runtime validation checks, this will run in the harness right after the stack is stood up, all in response to a simple go test invocation. This just scratches the surface of what’s possible – anything you can write in code using Go’s test capabilities is within reach!

Continuous Infrastructure Integration

It’s wonderful to be able to easily run these on demand, on a laptop, when you’re iterating on infrastructure changes and want to verify them before submitting for code review. But we and many of our customers run infrastructure testing at various points in the development lifecycle:

In each pull request opened, to validate before allowing it to be merged.
In response to each commit, to double check it was merged correctly.
On a recurring basis, such as nighty or even weekly for extended testing.
As part of more exotic and thorough test programs, such as performance or stress testing, which typically run over a longer period of time, execute many things in parallel, and/or deploy the same program multiple times.

For each of these, Pulumi supports integration with your favorite continuous integration system. This gives you similar continuous integration testing coverage for your infrastructure as you might already enjoy for your application software or in other programming domains.

Pulumi supports your existing CI systems. Here are a few of those supported:

Please refer to the Continuous Delivery documentation for a more comprehensive guide.

Ephemeral Environments

A very powerful capability this unlocks is the ability to spin up ephemeral environments solely for purposes of acceptance testing. Pulumi’s concept of projects and stacks is designed to make it very easy to stand up entirely isolated and independent environments, and to tear them down, all in either a few easy CLI gestures, or by using the integration testing framework.

If you are using GitHub, Pulumi offers a GitHub App that helps to glue together your Pull Request workflow with this sort of acceptance testing run inside of your CI pipelines. Simply install the App into your GitHub repos, and Pulumi in your CI, and your Pull Requests will light up with infrastructure previews, updates, and test results seamlessly:

By leveraging Pulumi for your core acceptance test workflow, you’ll unlock new automation capabilities that improve your team’s productivity and confidence in the quality of changes.

In Summary

In this post, we saw that by using general purpose programming languages, we unlock many software engineering practices we’ve benefited from in our application development. This includes unit testing, integration testing, and even the intersection of both to do extensive runtime testing. It’s easy to run these on demand and harness them in your CI system of choice.

Pulumi is open source, free to use, and works with your favorite languages and clouds – give it a try today!

Getting Started on Google Cloud Platform with Pulumi

Luke Hoban — Tue, 09 Apr 2019 00:00:00 +0000

Google Cloud is one of the most exciting cloud platforms available today, with a breadth of powerful infrastructure services from Google Container Engine (GKE) and Google Cloud Functions to Cloud Firestore and Cloud Spanner.

Pulumi is the most productive tooling available today for teams building cloud applications and infrastructure, in your favorite languages. Add them together, and teams can easily take maximum advantage of Google Cloud Platform’s rich features, productively, with a combined platform that makes it easy to collaborate, share, and reuse.

In this post, we’ll highlight some of the top features of Pulumi that make it a joy to work with Pulumi + Google Cloud:

Infrastructure as code for the Full Google Cloud Platform
Working with Kubernetes and Google Container Engine (GKE)
Super Simple Serverless with Google Cloud Functions
Continuous Delivery with Google Cloud Build
Managing Deployment with the Pulumi Service

Infrastructure as Code for the Full Google Cloud Platform

Pulumi lets you define and deploy infrastructure as code using your favorite programming languages, including JavaScript, Python, and Go. Pulumi provides access to the full Google Cloud Platform – every resource (including beta resources!) is available directly in your programming language and IDE. This delivers the software engineering benefits of your programming language, combined with the reliable and repeatable deployments of infrastructure as code.

In this example, we define a new network, firewall, and compute instance, using Python:

import pulumi
from pulumi_gcp import compute

disk = {
    'initializeParams': {
        'image': "centos-cloud/centos-7-v20190116"
    }
}

addr = compute.address.Address(resource_name='poc')

network = compute.Network("network")
network_interface = [
    {
        'network': network.id,
        'accessConfigs': [{'nat_ip': addr.address}],
    }
]

firewall = compute.Firewall("firewall", network=network.self_link, allows=[{
    'protocol': "tcp",
    'ports': ["22", "80"]
}])

instance = compute.Instance('poc', name='poc', boot_disk=disk, machine_type="f1-micro",
                            network_interfaces=network_interface)

pulumi.export('external_ip', addr.address)

Infrastructure can be deployed and updated using the pulumi CLI, with each update driving toward the desired state declared by the Pulumi program. Before each update, you can see a preview of the changes that will be made to your target infrastructure, and only after confirming will it proceed (with a full audit history):

$ pulumi update
Previewing update (luke):
     Type Name Plan Info
     pulumi:pulumi:Stack gcp-instance-nginx-luke
 ~ ├─ gcp:compute:Firewall firewall update [diff: ~allows]
 +- └─ gcp:compute:Instance poc replace [diff: ~bootDisk,name]
Resources:
    ~ 1 to update
    +-1 to replace
    2 changes. 3 unchanged
Do you want to perform this update? Yes
...

Learn more about working with Pulumi and Google Cloud infrastructure in this GCE Web Server tutorial.

Working with Kubernetes and Google Container Engine (GKE)

Pulumi also makes it easy to deploy resources to Kubernetes with access to the full Kubernetes API, across all object types and versions, using the same programming language and IDE experience. You no longer have to switch tooling at the boundary between GCP and Kubernetes. Instead, Pulumi gives you a single programming model for both.

In addition to the preview and auditing mentioned above, the Pulumi CLI also provides rich status and repeatable deployments to Kubernetes through its support for dependency-based orchestration of deployments:

And if you need a Kubernetes cluster, spinning one up with GKE using Pulumi is a breeze. Best of all you have easy access to the full breadth of configurability you might need: networking configuration, multiple NodePools, or clusters in multiple regions/zones. And you can easily add additional GCP infrastructure like managed databases, DNS configuration, or other resources needed to have a fully functional Kubernetes compute environment.

This example spins up a GKE cluster and uses Pulumi’s configuration system for easy parameterization:

from pulumi import Config
from pulumi_gcp.container import Cluster

# Read in some configurable settings for our cluster:
config = Config(None)

# nodeCount is the number of cluster nodes to provision. Defaults to 3 if unspecified.
NODE_COUNT = config.get('node_count') or 3
# nodeMachineType is the machine type to use for cluster nodes. Defaults to n1-standard-1 if unspecified.
# See https://cloud.google.com/compute/docs/machine-types for more details on available machine types.
NODE_MACHINE_TYPE = config.get('node_machine_type') or 'n1-standard-1'
# username is the admin username for the cluster.
USERNAME = config.get('username') or 'admin'
# password is the password for the admin user in the cluster.
PASSWORD = config.require('password')

# Now, actually create the GKE cluster.
k8s_cluster = Cluster('gke-cluster',
    initial_node_count=NODE_COUNT,
    node_version='latest',
    min_master_version='latest',
    master_auth={ 'username': USERNAME, 'password': PASSWORD },
    node_config={
        'machine_type': NODE_MACHINE_TYPE,
        'oauth_scopes': [
            'https://www.googleapis.com/auth/compute',
            'https://www.googleapis.com/auth/devstorage.read_only',
            'https://www.googleapis.com/auth/logging.write',
            'https://www.googleapis.com/auth/monitoring'
        ],
    },
)

Learn more about using Pulumi with Kubernetes and GKE in this Hello GKE tutorial.

Super Simple Serverless with Google Cloud Functions

Serverless has never been simpler than it is with Pulumi + Google Cloud Functions. Thanks to Pulumi’s “magic functions”, we can write callbacks in our language of choice, using a natural event-driven style, and Pulumi will handle the rest.

For example, want to serve a simple HTTP API with no fixed costs? It’s just a few lines of code:

import * as gcp from "@pulumi/gcp";

let greeting = new gcp.cloudfunctions.HttpCallbackFunction("greeting", (req, res) => {
    res.send(`Greetings from ${req.body.name || 'Google Cloud Functions'}!`);
});

export const url = greeting.httpsTriggerUrl;

Or perhaps a pubsub topic that runs some custom code on every message received:

// Create a PubSub Topic
let requests = new gcp.pubsub.Topic("requests");
// Print out a log message for every message on the Topic
requests.onMessagePublished("newMessage", (data) => {
    console.log(Buffer.from(data.data, "base64").toString());
});

And for quick inner loop development, the pulumi logs command supports pulling logs for your Google Cloud Functions from StackDriver, aggregating all compute logs in your program into one unified stream:

$ pulumi logs
Collecting logs for stack dev since 2019-04-08T12:02:48.000-07:00.
 2019-04-08T12:37:34.000-07:00[ newMessage-9d41c7a] Function execution started
 2019-04-08T12:37:34.000-07:00[ newMessage-9d41c7a] {"hello": "world"}
 2019-04-08T12:37:34.000-07:00[ newMessage-9d41c7a] Function execution took 70 ms, finished with status: 'ok'

Learn more about Pulumi support for Google Cloud Functions in this Simple Serverless Slackbot example.

Continuous Delivery with Google Cloud Build

Because Pulumi is infrastructure-as-code, the projects you deploy with Pulumi are robust and repeatable. Spinning up a testing or staging environment is simple, and continuously deploying infrastructure changes from commits and pull requests to your code is as easy as adding a few lines to your CI/CD system.

For teams working with Google Cloud Build, Pulumi deployments can be integrated into your continuous delivery with just a few steps. And for teams working with GitHub, Pulumi’s GitHub App gives you rich feedback on what changes will be deployed as part of a Pull Request, so you can see and approve infrastructure changes before they happen:

Learn more about integrating Pulumi with CI/CD in our continuous delivery documentation.

Managing Deployment with the Pulumi Service

In addition to the pulumi CLI for deploying and updating GCP infrastructure, Pulumi is always connected to the Pulumi Service, which provides a rich interface for viewing the history of your stacks and deployments, management of access controls for teams, and details of all resources under management.

With deep links into the Google Cloud Console, Pulumi provides quick access to all resources in your project.

And because Pulumi works with both Google Cloud and Kubernetes resources at the same time, it’s just as easy to deep link into resources in the Kubernetes console as well!

You can check out the Pulumi Service at app.pulumi.com and create your first Pulumi GCP project with just a few clicks, with an array of New Project options to get you up and running quickly.

Next Steps

Here at Pulumi, we love working with GCP and Kubernetes, and we are constantly working on ways to make Pulumi the most productive (and most fun!) way to program the Google Cloud Platform.

Pulumi is free and open source. You can get started with Pulumi today.

Here are a few resources to learn more about working with Pulumi and GCP:

Pulumi now supports Atlassian Identity

Praneet Loke — Wed, 30 Jan 2019 00:00:00 +0000

Today we added support for yet another developer favorite product, Atlassian Bitbucket. You can now sign-up for a Pulumi account with an Atlassian identity. This also means you can connect your Atlassian identity with an existing Pulumi account.

This work follows on from the support for GitLab identity and also the ability to connect identities, eliminating the need for users to create multiple accounts on Pulumi.

This helps users with repos across the major version control systems to seamlessly import their GitHub Organizations and GitLab Groups - and now Atlassian Bitbucket Teams - into a single Pulumi account. Of course, you don’t need to connect identities. You can always create separate account for each of your identities, if that’s what you want to do.

Creating a new Pulumi account using your Atlassian identity

To get started using your Atlassian identity, you can navigate to app.pulumi.com and select the Atlassian button to authenticate. If you are already logged-in to Pulumi, you should first logout and then head back to the Pulumi Service.

Connecting an existing Pulumi account to your Atlassian identity

If you want to connect your Atlassian identity to an existing Pulumi account, then navigate to app.pulumi.com and once logged in, head to Account Settings to choose the option to connect your Atlassian identity.

Bitbucket Team-backed Organizations

A Bitbucket Team helps you organize users into user groups and therefore, assign each of them a permission to your repositories. By adding a Bitbucket Team-backed organization, you can invite the same members to collaborate on Pulumi.

This allows you to set permissions on Stacks owned by your organization. You can learn more about Teams in the Pulumi and how those permissions get applied too.

CI Integration

Did you also know that the Pulumi CLI can be used in most, if not all, CI/CD workflows? Check out our documentation for integrating with several popular CI/CD systems. Get up and running with Pulumi in CI on GitLab, Travis, Azure DevOps and more. Don’t see docs for a particular CI system? Let us know or better yet file an issue here.

Having trouble? Questions? Join our community Slack or send us an email.

Getting to ChatOps with Pulumi Webhooks

Chris Smith — Wed, 23 Jan 2019 00:00:00 +0000

Today we are delighted to announce the availability of Webhooks on Pulumi. Webhooks are a very common mechanism to enable teams to be notified or react to events. In Pulumi’s case, this means: notifications of infrastructure changes (be it on Kubernetes, AWS, or any other cloud); responding to those changes as part of ‘ChatOps’; or other build pipelines, to improve the delivery of cloud native infrastructure.

Pulumi Webhooks are available for the Team and Enterprise editions of Pulumi. If you’re keen to try them out, start a trial of Team Edition here.

ChatOps — the idea of conversation-driven collaboration — is an increasingly common pattern, and as such we’ve gotten a lot of requests to integrate Pulumi with cloud services such as Slack, GitHub, and Twillio to facilitate that. These requests are rooted in users wanting to have more awareness of changes being made to their cloud infrastructure or wanting to build extensions to automate continuous delivery pipelines. Pulumi Webhooks offer the flexibility to design these notification and response systems as required for your team.

Using Pulumi Webhooks

You can attach webhooks directly to stacks or organizations. Stack webhooks will send notifications whenever changes to a stack are made (pulumi up) or simply previewed (pulumi preview). Webhooks attached to an organization will be sent the events for every stack within the organization.

When events occur, we’ll send a HTTP POST request to any registered listeners. Webhooks can then be used to send notifications to Slack or SMS, start running automated tests, or maybe even update another stack!

We’ve been using Pulumi webhooks internally for raising awareness when our test and production environments get updated. A common sight after submitting some code is:

Building a Pulumi Webhook handler using… Pulumi!

Pulumi is perfect for setting up cloud infrastructure, and so what better tool to use for handling notifications from Pulumi than Pulumi!

In this example, we’ll create a simple Pulumi program that stands up a webhook handler that will receive webhook requests and echo them to Slack. You can find the full code in the examples repo, under aws-ts-pulumi-webhooks.

The code is very straight forward, thanks to the @pulumi/cloud package.

import * as pulumi from "@pulumi/pulumi";
import * as cloud from "@pulumi/cloud";

import * as slack from "@slack/client";

const config = new pulumi.Config();

const stackConfig = {
 slackToken: config.require("slackToken"),
 slackChannel: config.require("slackChannel"),
};

const webhookHandler = new cloud.HttpEndpoint("pulumi-webhook-handler");

webhookHandler.get("/", async (_, res) => {
 res.status(200).end("🍹 Pulumi Webhook Responder🍹");
});

webhookHandler.post("/", async (req, res) => {
 const webhookID = req.headers["pulumi-webhook-id"];
 const webhookKind = req.headers["pulumi-webhook-kind"];
 console.log(`Received webhook from Pulumi ${webhookID} [${webhookKind}]`);

 const payload = <string>req.body.toString();
 const prettyPrintedPayload = JSON.stringify(JSON.parse(payload), null, 2);
 const client = new slack.WebClient(stackConfig.slackToken);
 await client.chat.postMessage({
 channel: stackConfig.slackChannel,
 text: `Pulumi Service Webhook (`${webhookKind}`)` + "```" + prettyPrintedPayload + "```",
 as_user: true,
 });

 res.status(200).end(`posted to Slack channel ${stackConfig.slackChannel}`);
});

export const url = webhookHandler.publish().url;

But to stand up the cloud application, we’ll need to create the stack and set some configuration. And finally deploy it using pulumi up.

$ pulumi stack init pulumi-webhook-handler
Created stack 'pulumi-webhook-handler'
$ pulumi config set cloud:provider aws
$ pulumi config set aws:region us-west-2
$ pulumi config set slackChannel #ops-notificatinos
$ pulumi config set slackToken ****** --secret
$ pulumi up --yes
Updating (robot-co/pulumi-webhook-handler):
... snip ...
Outputs:
url: "https://6tfwcc3vg3.execute-api.us-west-2.amazonaws.com/stage/"
Resources:
~ 1 updated
15 unchanged
Duration: 4s
Permalink: https://app.pulumi.com/robot-co/deploytron-9000/updates/6

The stack emits the URL to the webhook handler in the stack’s outputs. That URL is what will receive and process Pulumi webhooks.

With the infrastructure in-place, we just need to register the webhook in the Pulumi Service. Organization administrators can do so under the organization’s SETTINGS tab.

That’s it! As stacks are created, updated, and so on, the webhook handler will be notified and you can start building any custom actions you can think of.

There are three kinds of webhook events.

Event Type	Trigger
`stack`	Fired whenever a stack is created or deleted within an organization.
`stack_update`	Fired when a stack is updated. (Be it from `pulumi up`, `pulumi refresh`, or `pulumi destroy`.)
`stack_preview`	Fired whenever changes to a stack are previewed.

Tell us what you think

Get started with Pulumi Webhooks on the Pulumi app, and let us know how you’re using them so we can continue to extend their capabilities. We’d love to hear your ideas, as well as any feedback you have on the Pulumi Community Slack.

Upcoming AWS + Pulumi Webinar on Feb 5

Erin Xue — Wed, 09 Jan 2019 00:00:00 +0000

Pulumi is hosting a webinar with AWS Fargate on February 5th, 10AM PST (register here). We’ll be chatting about how to implement cloud native infrastructure across your organization using AWS and Pulumi: general purpose programming languages to deliver everything from VMs to Kubernetes to Serverless.

Register Now

Registration link. Specifically, you will learn:

How using general-purpose programming languages provides familiar and powerful programming concepts to infrastructure-as-code challenges
How Pulumi can help build a library of code packages to enhance efficiency when implementing standard policies, network best practices, and more
Best practices for becoming a deployment-focused organization, rather than configuration-focused, to continuously deliver new cloud-native infrastructure with containerization

Joining us is Pulumi customer Learning Machine. Adopting Pulumi helped Learning Machine reduce 25,000 LoC of ad-hoc DevOps scripts to 500 LoC of JavaScript, improving time to ship code, time to provision and onboard new customers, and unify their Development and DevOps teams and techniques.

Download the case study

This webinar will be useful to developers, System Architects, IT managers and DevOps managers. (register here).

Ask any questions you have to our speakers, Trevor Hansen (Startup Partner Solutions Architect), Luke Hoban (CTO, Pulumi), and Kim Hamilton Duffy (CTO & Principal Architect, Learning Machine).

To learn more about the webinar, check out our e-book and case study.

Connecting multiple identities to Pulumi

Praneet Loke — Fri, 14 Dec 2018 00:00:00 +0000

Hot on the heels of our GitLab sign-in support, we’ve just released support for multiple identities for a single Pulumi account in the Pulumi Service. Previously, you could only sign-up for a new Pulumi account using a GitHub or GitLab identity. Starting today, you can connect your Pulumi account with additional identities, beyond what you first signed-up with.

For example, if you signed-in to Pulumi with your GitHub account, you could only add GitHub organizations to the Pulumi Service, or collaborate with other GitHub users. With multiple identities, you can now also connect your Pulumi account to a GitLab identity too!

To connect an identity, first sign-in as usual on https://app.pulumi.com, then go to your Profile page as shown below.

Based on the example, on the Profile page, you would see a new button to connect a GitLab account. Conversely, if you signed-up originally using your GitLab account onto Pulumi, then you would see a button to connect your GitHub account.

Connecting multiple identities to Pulumi has advantages:

You can then add your GitHub organizations or GitLab groups, to collaborate with the respective members of those organizations using Pulumi.
A single Pulumi account connected to additional identities allows for more ways to sign into the Pulumi Service.

However, you don’t have to associate multiple identities with the same Pulumi account. Whenever you sign-in with a new identity for the first time on app.pulumi.com, a new Pulumi account is created. You can then choose to add additional identities at a later time.

Welcoming GitLab users to Pulumi

Praneet Loke — Mon, 03 Dec 2018 00:00:00 +0000

We are very pleased to announce that Pulumi now supports signing-in with your GitLab account. Don’t have an account on GitLab yet? Head on over to https://gitlab.com and sign-up for an account.

Why GitLab?

GitLab is a fully-integrated DevOps service that not only provides git repository management but also allows you to collaborate using issue boards, setup build and deployment pipelines. It provides an end-to-end solution that starts from your code and ends with your customers, all without ever having to leave GitLab.

And with Pulumi on GitLab, you truly never have to leave GitLab, not even to provision your infrastructure to run your services on the cloud.

Learn more about GitLab.

Log into app.pulumi.com using the GitLab Sign-in.
Click on the organizations dropdown menu and select Add an organization.
On the next page, you may select your GitLab group, based on which a Pulumi organization will be created.
Once the organization is created, you can create stacks in your newly created organization, so that other members of your organization can collaborate with you.

Inviting members in your GitLab group to use Pulumi

Select the organization you created from the organization dropdown menu. You should see a People tab. Select the People tab.

You will see the People tab if you added the organization on Pulumi or if you are an Owner of the GitLab group on gitlab.com.

Members must have an active and valid role in your GitLab group, so that you may invite them.

From the People tab, you can invite additional members in your group to also use Pulumi. By inviting members in your group to join Pulumi, you can collaborate with them more easily and share stacks, just like you would collaborate on a project on GitLab.

Running Pulumi on GitLab

Pulumi can be run in many CI/CD environments. Pulumi can be easily integrated in your CI pipeline on GitLab, too.

Deep-linking to your GitLab projects, branches and commits

When you run Pulumi in a GitLab-based repo, the pulumi CLI recognizes the correct metadata based on your repository’s remote configuration and uses that information to enhance your experience on app.pulumi.com.

Here’s an example of a project hosted on gitlab.com.

Here’s the same project with some activity having links to the commit-hash, and the branch in which this stack was updated.

FAQ

Click the GitLab sign-in button on <app.pulumi.com>, Pulumi will redirect you to gitlab.com where GitLab will present you with the sign-in options to login into your GitLab account. At that point, you may choose any of the sign-in options GitLab provides to sign-in.

I already have an account on Pulumi. Will signing-in with my GitLab identity create a new account?

Yes. Signing-in with a GitLab account will create a new account. That means, your stacks and activity will stay with the other account. You can migrate them by performing a pulumi stack export from your source stack and then importing it using pulumi stack import in a new stack in your GitLab-based account.

If you would like to add your GitLab identity to your existing Pulumi account, you can do so by connecting your GitLab identity from your Pulumi account’s profile page.

One of the benefits of using <app.pulumi.com> is to track the state of your stacks. When you are running the pulumi CLI on your machine, you can login into your account by typing pulumi login. There are two options for you to complete the login process. You can either create an Access Token on <app.pulumi.com> or simply press ENTER to let the CLI launch the browser.

If you would like to let the CLI launch the browser, ensure that you are already signed-in using GitLab at <app.pulumi.com> using your machine’s default browser. This way, when the browser is launched by the CLI, your Pulumi account based on your GitLab identity would be automatically used.

Reusable CI/CD components with CircleCI Orbs for Pulumi

Chris Smith — Wed, 07 Nov 2018 00:00:00 +0000

This morning CircleCI announced the launch of CircleCI Orbs which enable you to create reusable components for CircleCI workflows. Orbs enable you to simplify your CI/CD configuration by reusing existing orb jobs or commands, in much the same way Pulumi enables you to simplify the delivery of your cloud native infrastructure by sharing and reusing existing components.

Pulumi is proud to be a CircleCI technology partner, and we were excited to get a head start on seeing how orbs could make it easier to take Pulumi into production within CircleCI. The Pulumi Orbs for CircleCI are available today for you to start using.

Pulumi Orbs

You can see the Pulumi orbs in-action in the following CircleCI config.yml. It’s a simple job to build and update a JavaScript-based stack.

version: 2.1
Orbs:
pulumi: pulumi/pulumi@1.0.0
jobs:
build:
docker:
- image: circleci/node:7.10
working_directory: ~/repo
steps:
- checkout
- pulumi/login
- run:
command: |
npm install
npm run build
- pulumi/update:
stack: website-prod

First, the Pulumi orbs (and fixed version) are referenced by the name pulumi/pulumi@1.0.0. Then, the two Pulumi orbs pulumi/login and pulumi/update are invoked.

CircleCI orbs can be parameterized, so you can set the version of the Pulumi client to download, whether the update should skip its preview, and so on. (Full orb reference)

With other CI/CD systems, you’d typically need to write one-off Bash scripts to download the client from https://get.pulumi.com, add it to the current $PATH and so on. Orbs allow for simpler, more declarative CI/CD workflows.

The source code for the Pulumi orbs is available on GitHub, if you have any suggestions or feedback.

Other Integrations

Of course Pulumi’s integration with CircleCI doesn’t end with orbs.

Newer versions of the Pulumi command-line client will start associating CircleCI build metadata with stack updates, so you you will soon see links from app.pulumi.com to any stack updates or previews that happened during a CircleCI workflow.

Also, if you are using CircleCI with GitHub, you can consider installing the Pulumi GitHub application. The Pulumi GitHub app will surface the results of any previews or updates from your CI/CD on the source GitHub pull request. It’s always good to know if a pull request is going to lead to changes to your cloud infrastructure!

We are continuing to make Pulumi the best tool for the “Continuous Deployment” part of CI/CD workflows, and with the release of CircleCIs Orbs, it’s just that much easier.

Having trouble? Questions? Join our community Slack or drop us a line.

Links:

Get Started with Circle CI Orbs

Continuous Delivery to Any Cloud using GitHub Actions

Joe Duffy — Wed, 17 Oct 2018 00:00:00 +0000

Today we announced our partnership with GitHub on the new GitHub Actions feature. We are super excited about this bold and innovative technology, especially as it relates to Pulumi, and CI/CD more broadly. We truly believe that Pulumi plus GitHub Actions delivers the easiest, most capable, and friction-free way to achieve continuous delivery of cloud applications and infrastructure, no matter your cloud – AWS, Azure, Google Cloud, Kubernetes, or even on-premises. In this post, we’ll dig deeper to see why, and how to get up and running. It’s refreshingly easy!

GitHub Actions + Pulumi == ❤️

Pulumi lets you write cloud applications and infrastructure in your favorite language. This includes containers, serverless, and even VM-based infrastructure. Combined with GitHub Actions, we can get continuous deployment to any cloud – AWS, Azure, Google Cloud, Kubernetes, or even on-premises – all straight from GitHub.

GitHub Actions gives us a way to run compute in GitHub’s cloud, triggered by any GitHub event – commits, pull requests, and more. This enables us to run Pulumi in response to these events, giving us the benefits of infrastructure as code seamlessly integrated into a lovable workflow. We can edit any files – application- or infrastructure-related – to do end-to-end continuous delivery, without any of the YAML, tedious bash scripting, or manual, error-prone steps.

This capability enables some exciting scenarios:

Deploy cloud changes anytime a commit is made, including
- Provisioning entirely new cloud environments
- Updating existing infrastructure, like scaling things up or modifying config
- Rebuilding and redeploying app containers to your cluster
Preview deployments before they’re actually made, integrated with Pull Requests
Use Git branches to achieve GitOps-style promotion from test/staging to production

To see things in action, check out the video shown at the GitHub Universe keynote today:

In this short video, we see a Ruby on Rails application that is continuously deployed to our Kubernetes cluster (full code available here). It’s just a standard Rails application, with a Dockerfile, and a complete Pulumi infrastructure specification, which are just a collection of TypeScript modules. (We could have used Python or Go.) This includes a fully functioning Google Kubernetes cluster, hosted PostgreSQL database (for easier management!), Kubernetes definitions, and automatic building and publishing of our Rails Docker image to the Docker Hub, which then gets auto-deployed to our cluster. And it works across three entirely separate testing, staging, and production environments!

Notice that it’s all ordinary code, so that it works great with GitHub, in addition to our favorite IDEs and tools.

After committing our changes, Pulumi takes it from there. Deployments can be previewed, diffed, and are recorded so that you’ll always know who changed what, when, and why – all very “Git-like.” Pulumi’s GitHub App adds to this and enables “GitOps” so that teams can propose, approve, and promote code from “staging” to “production” using pull requests (more on that below).

We’ve chosen GKE here as an illustration because it’s amazing how powerful such a simple example can be, but this works just as well if you’re using AKS, EKS, on-premises clusters – and of course works for non-Kubernetes scenarios, like VMs, ECS, and Serverless applications.

All of this with just a handful of TypeScript and GitHub Actions set up to work with Pulumi – despite being quite capable, it’s super easy to get all of this up and running. Let’s see how!

Getting Up and Running

The full sequence of steps is available in our GitHub Actions Getting Started Guide.

In summary, using GitHub Actions with Pulumi is as easy as signing up for Pulumi (if you haven’t already), creating one or more workflow files in your repo (see here for examples), configuring secrets, and adding a Pulumi program. Afterwards, you can simply Commit and Pull Request away to deploy All The Things! GitHub Actions will fire and run Pulumi tasks appropriately; you’ll know it’s working if you see Pulumi actions show up alongside your PRs and commits:

Two sorts of actions, defined the examples referenced above, are

Pulumi Preview runs pulumi preview in response to a Pull Request, showing what would happen if the PR were merged into the target branch.
Pulumi Up runs pulumi up on the target branch, in response to a commit on that branch.

If either of these fail, you’ll see the standard GitHub Checks red “x”, whereas success will give a green “checkmark”; click on either to get the full Pulumi log output:

GitHub Actions’ Logs page, plus Pulumi’s Console, together give you total insight into a deployment’s status, including the Docker build logs, Kubernetes status updates, and more – so you’ll always have the full scoop on what’s going on, good or bad, in your cloud deployments.

Pulumi’s GitHub App

GitHub Actions work with Pulumi without needing any extra configuration beyond the workflow file and GitHub Secrets described above. That’s why it’s so easy!

If you are going to use this setup in a real team setting, you’ll probably also want to use Pulumi’s GitHub App. Simply by installing it into your repo, and combined with the above, you’ll instantly get improved GitHub Checks API integration and, more importantly, context added by the Pulumi bot to your Pull Requests about what a deployment will do before it’s even done, as shown above.

Note that this is optional with GitHub Actions, but worthwhile – it’s much easier for your teams to collaborate on deployments, including knowing whether a change might lead to downtime before it’s even triggered. “GitOps” ahoy!

Learning More

There is so much more fun to have, and we’re just getting started. Look for more in the weeks to come. In the meantime, here are some follow up links in case you want to learn more about GitHub Actions and Pulumi:

To learn more, install Pulumi and then check out our Getting Started guide.
In addition to the keynote video, we have two other videos that you might enjoy watching to learn more:
- In this video, we see the GitOps workflow for promoting from staging to production.
- In this longer-form video, we explore more nuances of how the entire experience works.
The Ruby on Rails App, deployed to Kubernetes, from the keynote, is available for easy forking here.

This is an exciting day for developers, DevOps engineers, and SREs alike. The combination of GitHub Actions and Pulumi gives us all an easy, automated solution for continuous deployment of cloud applications and infrastructure to any cloud, purely using code and Git. What’s best about it is that it’s fun in the same way programming is fun.

If you’re not yet in the GitHub Actions private beta, but want to try Pulumi, head on over to our Pulumi Getting Started guide. You’ll be running CLI commands, but we think that can be a lot of fun too!

We want to thank our GitHub partners for the bold and innovative work that went into GitHub Actions, and for letting us in on the action – we can’t wait to see what you, the Pulumi community, builds using this magical combination!

Announcing Pulumi 0.15

Luke Hoban — Wed, 15 Aug 2018 00:00:00 +0000

This blog post refers to an outdated version of Pulumi (0.15). For current features and capabilities, please refer to the latest documentation and current release notes.

Just over a month ago we publicly launched Pulumi, a new cloud native development platform. The response has been overwhelming and we’ve been hard at work responding to your feedback ever since.

Today, we are excited to release Pulumi 0.15 and make it available to download. This release includes improvements across the entire Pulumi development experience. Pulumi supports more platforms (Kubernetes and OpenStack, is faster (Parallelism, simpler (native TypeScript support), richer (serverless frameworks for Azure and Google Cloud), and is more deeply integrated into the application lifecycle (GitHub App for CI/CD integration).

In this post, we’ll take a quick tour of these new features. Stay tuned for follow up blog posts to dive deeper into individual topics.

Kubernetes

Pulumi can now deploy and manage Kubernetes resources using the same familiar programming model supported for AWS, Azure, and Google Cloud Platform. It’s also possible to mix resources from multiple clouds in the same program, such as using an S3 Bucket or RDS database from your Kubernetes-managed containers.

This release includes dozens of features that make it simple and productive to create and manage Kubernetes applications and resources and to perform production-grade deployments. The @pulumi/kubernetes package provides access to the full Kubernetes API, ensuring you can deploy any Kubernetes application with Pulumi.

New support for new helm.v2.Chart(...) allows deploying existing Helm charts via Pulumi, ensuring you can coordinate deployment of charts as part of a larger Pulumi deployment.

Complex Kubernetes resources (like Deployments and Services) also provide rich status updates so that you have complete visibility into the state of a deployment while it is in progress, including incremental progress reporting, and significant work was put into making it easy to diagnose and robustly recover from deployment failures.

The Pulumi Kubernetes provider can be used to deploy to any Kubernetes cluster, whether cloud hosted (EKS, AKS, GKE) or local (Minikube, Docker for Mac) or on-premise. And because Pulumi is already great at deploying cloud resources, the Pulumi Kubernetes provider can be combined with, for example, the Pulumi Azure provider to deploy and manage both the cluster and Kubernetes resources that should be installed into the cluster.

import * as azure from "@pulumi/azure";
import * as k8s from "@pulumi/kubernetes";
import * as helm from "@pulumi/kubernetes/helm";

// Create an Azure Kubernetes Service cluster
const resourceGroup = new azure.core.ResourceGroup("aks", { location: "West US" });
const kubernetesService = new azure.containerservice.KubernetesCluster("kubernetes", {
/* ... */
});

// Create a Pulumi Kubernetes provider configured to deploy to the AKS cluster above
export const azk8s = new k8s.Provider("azk8s", {
 kubeconfig: kubernetesService.kubeConfigRaw,
});

// Deploy a Helm chart into the cluster
const kibana = new helm.v2.Chart("kibana", {
 repo: "stable",
 chart: "kibana",
 version: "0.8.0",
 values: { service: { type: "LoadBalancer" } },
}, { providers: { kubernetes: azk8s } });

Check out the Kubernetes overview docs, the API documentation and the pulumi-kubernetes GitHub project for additional details.

Parallelism

Pulumi now runs deployments in parallel by default. To accomplish this, the Pulumi engine tracks all dependencies across cloud resources in your program, maximizing the amount of work that can proceed in parallel, while also ensuring all resource are created, updated, replaced, or destroyed in the correct order.

In some extreme cases, this new parallelism support can improve performance by 2-10x, but even more importantly, parallelism provides a performance boost for practically every Pulumi deployment.

First Class Providers

Pulumi programs can now create multiple instances of a resource provider, and can initialize an instance of a resources provider using outputs of other resources.

For example, a Pulumi program can now deploy resources into multiple AWS regions by creating providers targeting each region. This can be used for resources that must be deployed in a certain region (e.g. Lambda@Edge or ACM certificates), or just for managing multi-region deployments from a single Pulumi program.

This also enables scenarios where a Pulumi program deploys some base infrastructure, and then wants to manage resources deployed on top of that using a different provider. For example, deploy a GKE cluster, and then configure a Kubernetes provider to deploy Kubernetes resources on top of the cluster. Or even then deploy OpenFaaS into the Kubernetes cluster, and use the Pulumi OpenFaaS provider to deploy functions.

You might have already noticed, however we already used this feature in the Kubernetes example earlier!

Native TypeScript Support

Pulumi has supported JavaScript since inception, and as a result, it’s been possible to use Pulumi via TypeScript to get the benefits of earlier error checking and richer tooling for cloud infrastructure and application development. However, TypeScript code had to previously be manually compiled to JavaScript prior to doing a Pulumi deployment, which added some friction for users who wanted to use TypeScript with Pulumi.

In this release, Pulumi now natively supports TypeScript, so you do not need to run tsc explicitly before deploying. (We often forget to do this too!) Simply run pulumi up, and the program will be recompiled on the fly before running it.

New templates are now available via pulumi new which use the new, simpler, TypeScript support.

Using Imports from Runtime Functions

Pulumi supports creating cloud functions (Lambdas, Azure Functions, Google Cloud Functions) using JavaScript callbacks in your Pulumi application. This provides a nice and simple way to write the little pieces of code that you need to integrate into your cloud application and version with your infrastructure using real functions.

We heard feedback from early adopters who wanted to be able to import NPM packages in their Pulumi program more easily, and then use them within one of these cloud functions. We’ve enabled this now, so that the common way of structuring code works as expected, and so that writing idiomatic JavaScript or TypeScript “just works.”

For example, the code below uses the axios NPM package to make an HTTP request inside an AWS Lambda invoked by an AWS API Gateway (in just a few lines of code!).

import * as axios from "axios";
import * as cloud from "@pulumi/cloud-aws";

const api = new cloud.API("api");
api.get("/", async (req, res) => {
 const statusText = (await axios.default.get("https://www.pulumi.com")).statusText;
 res.write(`GET https://www.pulumi.com/ == ${statusText}`).end();
});

export const url = api.publish().url;

OpenStack

Thanks to a contribution by Fraser Waters (@Frassle), Pulumi now also supports OpenStack. This allows users targeting an OpenStack API from their public or private cloud provider to use Pulumi to author and manage infrastructure deployments.

For example, a VM can be deployed to OVH with just the following:

const os = require("@pulumi/openstack");
const instance = new os.compute.Instance("test", {
 flavorName: "s1-2",
 imageName: "Ubuntu 16.04",
});

export const instanceIP = instance.accessIpV4;

Check out the API documentation and the pulumi-openstack GitHub project for additional details. Huge thanks to Fraser for his work on this!

Serverless Functions in Google Cloud and Azure

Particularly with serverless functions being real functions, we had a lot of interest in broadening our existing support beyond just AWS serverless. We are happy to announce better serverless functionality for Google Cloud and Azure.

On Azure, the new @pulumi/azure-serverless package makes it easy to work with serverless functions, and has initial support for hooking up to Blob storage event sources:

import * as azure from "@pulumi/azure";
import * as serverless from "@pulumi/azure-serverless";

const storageAccount = new azure.storage.Account("images-container", { /* ... */ });
serverless.storage.onBlobEvent("newImage", storageAccount, (context, blob) => {
 context.log(context);
 context.log(blob);
 context.done();
}, { containerName: "folder", filterSuffix: ".png" });

export const storageAccountName = storageAccount.name;

On Google Cloud, the new gcp.serverless.Function provides an easy way to create a Google Cloud Function from a JavaScript callback in a Pulumi program. Thanks to Mikhail Shilkov (@mikhailshilkov) for contributing this feature!

import * as gcp from "@pulumi/gcp";
let f = new gcp.serverless.Function("f", {}, (req, res) => {
 res.send(`Hello ${req.body.name || 'World'}!`);
});

export const url = f.function.httpsTriggerUrl;

GitHub App for CI/CD Integration

Pulumi already works with your favorite CI/CD systems to accomplish automated and continuous deployments of cloud infrastructure and applications. This is how Pulumi deploys and manages our own infrastructure that runs <pulumi.com>, and is how our most engaged users adopt Pulumi in their own teams.

Our mission is to make Pulumi the easiest way to get code to the cloud. To that end, we are launching a new Pulumi GitHub App which bridges the gap between GitHub (source code, pull requests) and Pulumi (cloud resources, stack updates). By installing the Pulumi GitHub App into your GitHub organization , and then running Pulumi as part of your existing CI/CD process, you will receive integrated stack updates and previews in your pull requests. This allows you to see the potential impact a change would have on your cloud infrastructure before merging the code, and collaborate with your teammates when deploying application and infrastructure changes.

The Pulumi GitHub App is still in preview as we work to support more CI systems and extend its capabilities. For information on how to install it and configure it with your CI system, please read the documentation.

Summary

We’re excited about all the new features in this release and the new scenarios they enable for the Pulumi community . If you are new to Pulumi, download the tools and get started today, or join us in Slack. A big thanks to all the users and contributors who have helped shape this release – we can’t wait to see what you build next !

Special Thanks

Fraser Waters (@Frassle)
Mikhail Shilkov (@mikhailshilkov)
James Nugent (@jen20)
Thomas Schersach (@Tirke)
JT (@Jtango18)
Itay Maman (@imaman)