Pulumi Blog: Cloud engineering

Scan AWS GovCloud and more partitions with Pulumi Insights

Alejandro Cotroneo — Tue, 14 Apr 2026 09:00:00 +0000

Pulumi Insights account scanning now supports every AWS partition. If your workloads run in GovCloud, China, the European Sovereign Cloud, or one of the ISO intelligence-community clouds, you can get the same resource discovery, cross-account search, and AI-assisted insights that commercial accounts already have.

Supported partitions

AWS Standard (Commercial)
AWS GovCloud (US)
AWS ISO (US)
AWS ISOB (US)
AWS ISOF (US)
AWS ISOE (Europe)
AWS European Sovereign Cloud
AWS China

You can also exclude specific regions from discovery — useful when regions are disabled by SCPs or fall outside an audit’s scope.

Set it up

In the Pulumi Cloud console:

Go to Accounts → Create account.
Select AWS as the provider.
Under Add your configuration, pick the target partition.
Supply credentials via a Pulumi ESC environment. The OIDC trust policy uses the partition-appropriate ARN prefix (arn:aws-us-gov:, arn:aws-cn:, etc.).

For IAM and ESC setup, see the Insights accounts docs. Log in to Pulumi Cloud to get started.

Maximize Business Outcomes In Your Multi-Cloud Environment

Marlowe Fenne — Fri, 16 May 2025 00:00:00 +0000

Overcoming the cost and complexity of multi-cloud environments and managing infrastructure more effectively is a perpetual challenge. For thousands of companies growing Platform Engineering skills, the solution to these challenges may be closer than you think.

Hundreds of thousands of developers across the globe already rely on Pulumi to drive more scalable, repeatable deployments in large self-managed production environments. By combining the power of Infrastructure as Code (IaC) with automation, compliance, and robust security within a Pulumi-hosted environment, you can achieve new levels of performance, visibility and security while reducing risk. Here are a few examples:

Speed - Streamlined Operations

Pulumi Cloud accelerates time to market. It automates deployments, manages state, and centralizes secrets, eliminating the complexities of DIY IaC backends. Its transactional state management prevents issues like outages caused by mismanaged state files, offering a secure, reliable platform for scaling operations.

Materialize reduced costs and accelerated time to market by making developer onboarding 75% faster.

Scale - Improved Developer Productivity

By supporting modern programming languages like Python, TypeScript, and Go, Pulumi Cloud allows developers to define infrastructure using familiar tools, speeding up onboarding and simplifying workflows. Predefined policy packs and dynamic secrets further reduce operational overhead.

BMW Group used Pulumi to build a scalable and resilient hybrid cloud implementation that could handle more than eleven thousand developers.

Security - Enhanced Security and Compliance

With features like Pulumi CrossGuard for policy enforcement and Pulumi ESC for centralized secrets management, Pulumi Cloud ensures adherence to security standards like SOC2 and HIPAA. This eliminates the risks associated with improperly stored credentials or manually enforced policies.

Tetrate uses Pulumi ESC to stop secrets sprawl and attain compliance more efficiently.

Savings - Cost Efficiency

Pulumi Cloud’s AI-driven insights identify stale infrastructure and optimize resource use, reducing waste and cutting costs. Its managed service eliminates the need for dedicated backend engineers, freeing resources for core business innovation.

Unity Technologies reduced deployment times by 80%, from weeks to hours, significantly improving their time to market.

By choosing Pulumi Cloud, IT leaders can focus on driving innovation, securing infrastructure, and achieving scalability without the hidden costs and risks of managing IaC backends manually. It’s a comprehensive solution for modern cloud management.

“Pulumi lets us manage multi-cluster Kubernetes infrastructure efficiently, all in one stack. It has hugely benefited productivity and our service’s reliability. Without Pulumi, we know that scaling and maintaining Materialize would be much harder for the team.”

– Paul Hemberger, Engineering Manager for Materialize’s cloud team

Is your team ready to move from a self-managed environment to Pulumi cloud? Explore your options with these resources, and enroll in our workshop series that’s customizable for your cloud and business environment.

Your Perfect Infrastructure May Not Be So Perfect

Simen A. W. Olsen — Mon, 16 Dec 2024 10:43:07 +0000

Guest Article: Simen A. W. Olsen from Bjerk, is here to share his lessons learned on why designing the perfect architecture for your future needs might be a mistake

I remember standing in front of our engineering team in 2018, proudly presenting what I believed was the future-proof architectural design for our new distributed system. The diagrams were immaculate, the technology choices were cutting-edge, and the scalability patterns were ready for any possible future scenario.

I was basically the Leonardo da Vinci of system design… if Leonardo had been really into Kubernetes and had a concerning addiction to coffee. But six months later, that “future-proof” architecture had become a constraint rather than an enabler, and my masterpiece was looking more like a finger painting done by a caffeinated raccoon.

This experience taught me something crucial: trying to build the perfect system that anticipates every future need is often worse than creating a system designed to change quickly. It’s like trying to predict what your kid will want to be when they grow up and pre-buying all the necessary equipment. Congrats, you now own a space suit, a stethoscope, and a dragon costume — and they decided to become a software engineer anyway.

The Over-Planning

Many teams fall into a common trap: they try to design systems that anticipate every possible future requirement. This happens even in agile teams, where we convince ourselves we need to “get the architecture right” before we can start iterating. You know, because nothing says “agile” like spending three months in a room drawing boxes and arrows while muttering “microservices” under your breath like it’s a magic spell.

In 2008, Netflix faced a choice: build the perfect data center that could handle all their anticipated future needs, or move to the cloud with a simpler architecture that could evolve. They chose the latter, focusing on making their system easy to change rather than trying to make it perfect. Smart move — unlike those my past self made who probably would’ve insisted on building a data center capable of streaming to Mars, just in case Elon asked nicely.

You might also like:

The Core Principles of Change-Ready Architecture

Through both failures and successes, I’ve identified three principles that define truly adaptable architecture:

Embrace simplicity. I think it makes sense to start with the simplest architecture that could possibly work for your current needs. Complexity should be earned, not presumed. If your architecture diagram looks like a plate of spaghetti that’s been hit by lightning, you might be doing it wrong.
Make change cheap. Instead of trying to avoid change, make it inexpensive. This means investing in automated testing, continuous deployment, and monitoring. When change is cheap, you don’t need to fear it.
Learn through action. Rather than trying to predict the future, build mechanisms that help you learn quickly about real needs. It includes feature toggles (Protip: Try Unleash.), A/B testing, and robust monitoring of how your system is actually being used. You know, actual data, not just what that one loud guy in planning insists will definitely happen.

The rise of AI and machine learning systems has made one thing clear: we can’t predict how our systems will need to evolve. The most successful teams aren’t those that try to build the perfect AI architecture upfront, but those that can rapidly experiment and adapt their systems based on real-world feedback.

The biggest pushback I hear is, “But what if we need to scale?” or “What about future requirements?” These fears often drive teams to over-architect their solutions. But here’s the reality: the cost of changing a simple system is usually lower than the cost of maintaining an over-engineered one. The key is understanding that good architecture isn’t about predicting the future — it’s about making future changes as painless as possible.

Conclusion

That over-engineered system I was so proud of in 2018? Its most significant flaw wasn’t in what it got wrong about the future — it was that it tried too hard to be right about the future in the first place. It’s like bringing a fully packed suitcase to a first date. Today, I know that the best architecture isn’t one that anticipates every need, but one that makes it easy to respond to needs as they emerge.

The next time you’re tempted to design for every possible future scenario, remember: the goal isn’t to build a perfect system, but to build one that’s perfectly easy to change. And if someone tells you they’ve designed the perfect future-proof architecture, they’re either lying, or they’ve discovered time travel — and in that case, they should be sharing lottery numbers, not system designs.

You might also like:

Dependent Stack Updates with Pulumi Deployments

Evan Boyle — Tue, 20 Jun 2023 00:00:00 +0000

As infrastructure projects grow in size and complexity, you need to decompose infrastructure into smaller stacks to limit the blast radius of errors, extract and reference common layers like networking, and limit access to sensitive components. This comes with a coordination cost as you now need to figure out how to detect and propagate changes to downstream stacks in your dependency graph. Today we’re announcing two features that can help you manage this complexity by automatically updating dependent stacks:

Deployment Webhook Destinations for triggering downstream stack updates automatically
The Pulumi Auto Deploy Package for easily expressing dependencies and configuring downstream updates via Deployment Webhooks.

The following example shows automatic deployment of stacks with the following dependency graph
a
├── b
│ ├── d
│ ├── e
│ └── f
└── c
Whenever a node in the graph is updated, all downstream nodes will be automatically updated
via a webhook triggering Pulumi Deployments.

Automatically updating dependent stacks and providing a way to write Pulumi programs that can express nested stack updates are two of the most requested features across the Pulumi project, and we’re excited to enable both of these via Pulumi Cloud and Pulumi Deployments.

Deployment Webhook Destinations

Pulumi Webhooks allow you to pick one or more event types on a stack (i.e. update succeeded, or refresh failed) and automatically deliver an event to a destination such a slack channel or a webserver you’ve deployed with custom code. Now you can choose Pulumi Deployments as a webhook destination. When an update completes on the parent stack, a deployment will automatically be triggered on the destination stack. This makes it easy to keep dependent stacks up to date. For instance, you can use Deployment Webhook Destinations to configure every successful update on your networking stack to trigger downstream updates on your database and compute stacks.

import * as pulumi from "@pulumi/pulumi";
import * as pulumiservice from "@pulumi/pulumiservice";

const databaseWebhook = new pulumiservice.Webhook("databaseWebhook", {
 organizationName: "org",
 projectName: "network",
 stackName: "prod",
 format: pulumiservice.WebhookFormat.PulumiDeployments,
 payloadUrl: "database/prod",
 active: true,
 displayName: "deploy-database",
 filters: [pulumiservice.WebhookFilters.UpdateSucceeded],
});
const computeWebhook = new pulumiservice.Webhook("computeWebhook", {
 organizationName: "org",
 projectName: "database",
 stackName: "prod",
 format: pulumiservice.WebhookFormat.PulumiDeployments,
 payloadUrl: "compute/prod",
 active: true,
 displayName: "deploy-compute",
 filters: [pulumiservice.WebhookFilters.UpdateSucceeded],
});

using System.Collections.Generic;
using System.Linq;
using Pulumi;
using PulumiService = Pulumi.PulumiService;

return await Deployment.RunAsync(() =>
{
 var databaseWebhook = new PulumiService.Webhook("databaseWebhook", new()
 {
 OrganizationName = "org",
 ProjectName = "network",
 StackName = "prod",
 Format = PulumiService.WebhookFormat.PulumiDeployments,
 PayloadUrl = "database/prod",
 Active = true,
 DisplayName = "deploy-database",
 Filters = new[]
 {
 PulumiService.WebhookFilters.UpdateSucceeded,
 },
 });

 var computeWebhook = new PulumiService.Webhook("computeWebhook", new()
 {
 OrganizationName = "org",
 ProjectName = "database",
 StackName = "prod",
 Format = PulumiService.WebhookFormat.PulumiDeployments,
 PayloadUrl = "compute/prod",
 Active = true,
 DisplayName = "deploy-compute",
 Filters = new[]
 {
 PulumiService.WebhookFilters.UpdateSucceeded,
 },
 });

});

import pulumi
import pulumi_pulumiservice as pulumiservice

database_webhook = pulumiservice.Webhook("databaseWebhook",
 organization_name="org",
 project_name="network",
 stack_name="prod",
 format=pulumiservice.WebhookFormat.PULUMI_DEPLOYMENTS,
 payload_url="database/prod",
 active=True,
 display_name="deploy-database",
 filters=[pulumiservice.WebhookFilters.UPDATE_SUCCEEDED])
compute_webhook = pulumiservice.Webhook("computeWebhook",
 organization_name="org",
 project_name="database",
 stack_name="prod",
 format=pulumiservice.WebhookFormat.PULUMI_DEPLOYMENTS,
 payload_url="compute/prod",
 active=True,
 display_name="deploy-compute",
 filters=[pulumiservice.WebhookFilters.UPDATE_SUCCEEDED])

package main

import (
 "github.com/pulumi/pulumi-pulumiservice/sdk/go/pulumiservice"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 _, err := pulumiservice.NewWebhook(ctx, "databaseWebhook", &pulumiservice.WebhookArgs{
 OrganizationName: pulumi.String("org"),
 ProjectName: pulumi.String("network"),
 StackName: pulumi.String("prod"),
 Format: pulumiservice.WebhookFormatPulumiDeployments,
 PayloadUrl: pulumi.String("database/prod"),
 Active: pulumi.Bool(true),
 DisplayName: pulumi.String("deploy-database"),
 Filters: pulumiservice.WebhookFiltersArray{
 pulumiservice.WebhookFiltersUpdateSucceeded,
 },
 })
 if err != nil {
 return err
 }
 _, err = pulumiservice.NewWebhook(ctx, "computeWebhook", &pulumiservice.WebhookArgs{
 OrganizationName: pulumi.String("org"),
 ProjectName: pulumi.String("database"),
 StackName: pulumi.String("prod"),
 Format: pulumiservice.WebhookFormatPulumiDeployments,
 PayloadUrl: pulumi.String("compute/prod"),
 Active: pulumi.Bool(true),
 DisplayName: pulumi.String("deploy-compute"),
 Filters: pulumiservice.WebhookFiltersArray{
 pulumiservice.WebhookFiltersUpdateSucceeded,
 },
 })
 if err != nil {
 return err
 }
 return nil
 })
}

name: auto-deploy-demo
runtime: yaml
description: A simple auto-deploy example
resources:
 databaseWebhook:
 type: pulumiservice:Webhook
 properties:
 organizationName: org
 projectName: network
 stackName: prod
 format: pulumi_deployments
 payloadUrl: database/prod
 active: true
 displayName: deploy-database
 filters:
 - update_succeeded
 computeWebhook:
 type: pulumiservice:Webhook
 properties:
 organizationName: org
 projectName: database
 stackName: prod
 format: pulumi_deployments
 payloadUrl: compute/prod
 active: true
 displayName: deploy-compute
 filters:
 - update_succeeded

Expressing Stack Dependencies with Pulumi Auto Deploy

We’ve also shipped a new Pulumi package called Pulumi Auto Deploy. It lets you simply express dependencies between stacks, and takes care of creating and updating the necessary Deployment Webhooks under the hood.

import * as autodeploy from "@pulumi/auto-deploy";
import * as pulumi from "@pulumi/pulumi";

/**
 *
 * The following example configures automatic deployment of stacks with the following dependency graph:
 a
 ├── b
 │ ├── d
 │ ├── e
 │ └── f
 └── c
 * Whenever a node in the graph is updated,
 * all downstream nodes will be automatically updated via a webhook triggering Pulumi Deployments.
 */

const organization = pulumi.getOrganization();
const project = "dependency-example"

export const f = new autodeploy.AutoDeployer("auto-deployer-f", {
 organization,
 project,
 stack: "f",
 downstreamRefs: [],
});

export const e = new autodeploy.AutoDeployer("auto-deployer-e", {
 organization,
 project,
 stack: "e",
 downstreamRefs: [],
});

export const d = new autodeploy.AutoDeployer("auto-deployer-d", {
 organization,
 project,
 stack: "d",
 downstreamRefs: [],
});

export const c = new autodeploy.AutoDeployer("auto-deployer-c", {
 organization,
 project,
 stack: "c",
 downstreamRefs: [],
});

export const b = new autodeploy.AutoDeployer("auto-deployer-b", {
 organization,
 project,
 stack: "b",
 downstreamRefs: [d.ref, e.ref, f.ref],

});

export const a = new autodeploy.AutoDeployer("auto-deployer-a", {
 organization,
 project,
 stack: "a",
 downstreamRefs: [b.ref, c.ref],
});

using System.Collections.Generic;
using System.Linq;
using Pulumi;
using AutoDeploy = Pulumi.AutoDeploy;

/**
 *
 * The following example configures automatic deployment of stacks with the following dependency graph:
 a
 ├── b
 │ ├── d
 │ ├── e
 │ └── f
 └── c
 * Whenever a node in the graph is updated,
 * all downstream nodes will be automatically updated via a webhook triggering Pulumi Deployments.
 */

return await Deployment.RunAsync(() =>
{
 var projectVar = "dependency-example";

 var organization = "pulumi";

 var f = new AutoDeploy.AutoDeployer("f", new()
 {
 Organization = organization,
 Project = projectVar,
 Stack = "f",
 DownstreamRefs = new[] {},
 });

 var e = new AutoDeploy.AutoDeployer("e", new()
 {
 Organization = organization,
 Project = projectVar,
 Stack = "e",
 DownstreamRefs = new[] {},
 });

 var d = new AutoDeploy.AutoDeployer("d", new()
 {
 Organization = organization,
 Project = projectVar,
 Stack = "d",
 DownstreamRefs = new[] {},
 });

 var c = new AutoDeploy.AutoDeployer("c", new()
 {
 Organization = organization,
 Project = projectVar,
 Stack = "c",
 DownstreamRefs = new[] {},
 });

 var b = new AutoDeploy.AutoDeployer("b", new()
 {
 Organization = organization,
 Project = projectVar,
 Stack = "b",
 DownstreamRefs = new[]
 {
 d.Ref,
 e.Ref,
 f.Ref,
 },
 });

 var a = new AutoDeploy.AutoDeployer("a", new()
 {
 Organization = organization,
 Project = projectVar,
 Stack = "a",
 DownstreamRefs = new[]
 {
 b.Ref,
 c.Ref,
 },
 });

});

import pulumi
import pulumi_auto_deploy as auto_deploy

'''

The following example configures automatic deployment of stacks with the following dependency graph:
 a
 ├── b
 │ ├── d
 │ ├── e
 │ └── f
 └── c
Whenever a node in the graph is updated,
all downstream nodes will be automatically updated via a webhook triggering Pulumi Deployments.
'''

project_var = "dependency-example"
organization = pulumi.get_organization()
f = auto_deploy.AutoDeployer("f",
 organization=organization,
 project=project_var,
 stack="f",
 downstream_refs=[])
e = auto_deploy.AutoDeployer("e",
 organization=organization,
 project=project_var,
 stack="e",
 downstream_refs=[])
d = auto_deploy.AutoDeployer("d",
 organization=organization,
 project=project_var,
 stack="d",
 downstream_refs=[])
c = auto_deploy.AutoDeployer("c",
 organization=organization,
 project=project_var,
 stack="c",
 downstream_refs=[])
b = auto_deploy.AutoDeployer("b",
 organization=organization,
 project=project_var,
 stack="b",
 downstream_refs=[
 d.ref,
 e.ref,
 f.ref,
 ])
a = auto_deploy.AutoDeployer("a",
 organization=organization,
 project=project_var,
 stack="a",
 downstream_refs=[
 b.ref,
 c.ref,
 ])

package main

import (
 "github.com/pulumi/pulumi-auto-deploy/sdk/go/autodeploy"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 /**
 *
 * The following example configures automatic deployment of stacks with the following dependency graph:
 a
 ├── b
 │ ├── d
 │ ├── e
 │ └── f
 └── c
 * Whenever a node in the graph is updated,
 * all downstream nodes will be automatically updated via a webhook triggering Pulumi Deployments.
 */

 pulumi.Run(func(ctx *pulumi.Context) error {
 projectVar := "dependency-example"
 organization := "pulumi"
 f, err := autodeploy.NewAutoDeployer(ctx, "f", &autodeploy.AutoDeployerArgs{
 Organization: pulumi.String(organization),
 Project: pulumi.String(projectVar),
 Stack: pulumi.String("f"),
 DownstreamRefs: pulumi.StringArray{},
 })
 if err != nil {
 return err
 }
 e, err := autodeploy.NewAutoDeployer(ctx, "e", &autodeploy.AutoDeployerArgs{
 Organization: pulumi.String(organization),
 Project: pulumi.String(projectVar),
 Stack: pulumi.String("e"),
 DownstreamRefs: pulumi.StringArray{},
 })
 if err != nil {
 return err
 }
 d, err := autodeploy.NewAutoDeployer(ctx, "d", &autodeploy.AutoDeployerArgs{
 Organization: pulumi.String(organization),
 Project: pulumi.String(projectVar),
 Stack: pulumi.String("d"),
 DownstreamRefs: pulumi.StringArray{},
 })
 if err != nil {
 return err
 }
 c, err := autodeploy.NewAutoDeployer(ctx, "c", &autodeploy.AutoDeployerArgs{
 Organization: pulumi.String(organization),
 Project: pulumi.String(projectVar),
 Stack: pulumi.String("c"),
 DownstreamRefs: pulumi.StringArray{},
 })
 if err != nil {
 return err
 }
 b, err := autodeploy.NewAutoDeployer(ctx, "b", &autodeploy.AutoDeployerArgs{
 Organization: pulumi.String(organization),
 Project: pulumi.String(projectVar),
 Stack: pulumi.String("b"),
 DownstreamRefs: pulumi.StringArray{
 d.Ref,
 e.Ref,
 f.Ref,
 },
 })
 if err != nil {
 return err
 }
 _, err = autodeploy.NewAutoDeployer(ctx, "a", &autodeploy.AutoDeployerArgs{
 Organization: pulumi.String(organization),
 Project: pulumi.String(projectVar),
 Stack: pulumi.String("a"),
 DownstreamRefs: pulumi.StringArray{
 b.Ref,
 c.Ref,
 },
 })
 if err != nil {
 return err
 }
 return nil
 })
}

name: auto-deploy-demo
runtime: yaml
description: A simple auto-deploy example
variables:
 project: dependency-example
 # TODO: update once https://github.com/pulumi/pulumi-yaml/issues/461 is fixed
 organization: pulumi
resources:
 f:
 type: auto-deploy:AutoDeployer
 properties:
 organization: ${organization}
 project: ${project}
 stack: f
 downstreamRefs: []
 e:
 type: auto-deploy:AutoDeployer
 properties:
 organization: ${organization}
 project: ${project}
 stack: e
 downstreamRefs: []
 d:
 type: auto-deploy:AutoDeployer
 properties:
 organization: ${organization}
 project: ${project}
 stack: d
 downstreamRefs: []
 c:
 type: auto-deploy:AutoDeployer
 properties:
 organization: ${organization}
 project: ${project}
 stack: c
 downstreamRefs: []
 b:
 type: auto-deploy:AutoDeployer
 properties:
 organization: ${organization}
 project: ${project}
 stack: b
 downstreamRefs:
 - ${d.ref}
 - ${e.ref}
 - ${f.ref}
 a:
 type: auto-deploy:AutoDeployer
 properties:
 organization: ${organization}
 project: ${project}
 stack: a
 downstreamRefs:
 - ${b.ref}
 - ${c.ref}

Just declare an AutoDeployer resource for each stack, and any downstream stacks will automatically be updated whenever a stack in its dependency chain is updated.

Automate your deployment pipeline

Use Pulumi Deployments to trigger downstream stack updates automatically when an upstream stack changes. Keep your dependent infrastructure in sync without manual runbooks.

Get started

Scale Infrastructure with Software

At Pulumi, we build tools to help you scale your infrastructure footprint with software instead of just humans. Deployment Webhook Destinations and pulumi-auto-deploy give you the tools to keep all of your dependent infrastructure up to date with automation instead of manual runbooks and human-driven operations.

Check out these resources to get started today:

Check out the Pulumi Deployments documentation and learn about configuring Deployment Settings for your stacks.
Get started with the Pulumi Auto Deploy Package.
Learn more about Pulumi Cloud Webhooks.
Join us in the #pulumi-deployments community slack channel.

🚀🚀🚀 Happy building! 🚀🚀🚀

Review Stacks: Collaborate in the Cloud

Evan Boyle — Wed, 14 Jun 2023 00:00:00 +0000

Today we’re excited to announce Review Stacks – dedicated cloud environments that get created automatically every time a pull request is opened, all powered by Pulumi Deployments. Open a pull request, and Pulumi Deployments will stand up a stack with your changes and the Pulumi GitHub App will add a PR comment with the outputs from your deployment. Merge the PR and Pulumi Deployments will destroy the stack and free up the associated resources. It has never been simpler to pick up an unfamiliar codebase, make changes to both application and infrastructure code, and share a live environment for review with your teammates.

Ephemeral environments for front-end development and static websites have become commonplace in recent years, but infrastructure and complex systems have been unsupported up until this point. Review Stacks enable you to iterate on both application code changes and infrastructure code changes at the same time. Making changes to HTML, CSS, Next.js, Go APIs, Pulumi configuration, subnet configuration, databases, and load balancers are all possible with Review Stacks. No more fumbling with out-of-date READMEs and developer documentation. Just open a pull request and you can start testing changes against everything from simple static websites to API servers, microservices, data pipelines, Kubernetes clusters, and any other piece of infrastructure across Pulumi’s 100+ cloud providers.

Review Stacks manage the full lifecycle of your cloud development environment including creating it when the PR is opened, updating it every time a new commit is pushed, and destroying and cleaning up all cloud resources when the pull request is merged or closed. They even integrate with GitHub checks so that you can gate your pull requests on successful Review Stack deployments.

Pulumi customers already using Review Stacks find that it saves their developers time. Having a standardized process for development environments and change management makes it easier to onboard new developers, and load-balance teammates onto unfamiliar projects and codebases. It also makes it substantially easier for developers to work on multiple feature branches and bug fixes at the same time, as they don’t have to worry about manually maintaining and juggling development environments.

“Pulumi allows every engineer to build and update infrastructure instead of only our infrastructure team, which enables us to ship faster than if we were using a domain-specific language,” said Aaron Gibralter, head of engineering, Mighty. “With Pulumi Deployments, we could easily set up a standardized CI/CD infrastructure workflow for every engineer that immediately makes them productive. Review Stacks automates the work of managing our own ephemeral infrastructure and makes our lives easier.”

Review Stacks also provide an avenue for optimizing cloud costs. Rather than keeping development infrastructure running 24/7, Review Stacks only exist as long as an engineer works on a feature. Since Review Stacks are flexible and built on Pulumi IaC, it is also possible to configure it to deploy to multi-tenant environments that share development infrastructure to further streamline costs.

“At GitHub, we are constantly seeking to improve the developer experience and our partner ecosystem plays a big role in that.” said Matthew Manning, Senior Business Development Manager for GitHub Actions, “The launch of Pulumi Review Stacks with GitHub makes it easier for teams to review and test their products before shipping which ultimately improves the developer experience.”

Configuring Review Stacks

Configuring Review Stacks is a simple three-step process:

Create a new stack, by convention named pr, and corresponding Pulumi.pr.yaml configuration file - this config will be copied into every review stack that gets created, and can even be modified within a PR.
Configure Deployment Settings for the stack - this specifies how to acquire source code, cloud credentials and more when deploying via Pulumi Deployments.
Set the pullRequestTemplate Deployment Setting to true - this indicates that all pull requests against this stack’s branch should reference this stack as a Review Stack template.

You can use an existing stack as a Review Stack template, as long as it has Deployment Settings configured. This will result in Review Stacks being deployed into the same cloud account. If you want to separate the cloud resources in your production stack from the resources created via Review Stacks then you can create a separate stack and template that references a different cloud account (AWS, Azure, GCP, etc).

Review Stacks and Deployment Settings can be configured via the Pulumi Cloud console, the Pulumi Cloud REST API, or within a Pulumi Program using the Pulumi Cloud Resource Provider.

Pulumi Cloud UI

It is just one click to turn on Review Stacks via the Pulumi Cloud console.

REST API

You can programmatically configure Review Stacks and Deployment Settings at scale across thousands of projects using the Deployments REST API.

curl -i -XPOST -H "Content-Type: application/json" -H "Authorization: token $PULUMI_ACCESS_TOKEN" \
--location "https://api.pulumi.com/api/stacks/org/project/stack/deployments/settings" \
-d '{
"gitHub":{
"pullRequestTemplate": true
}
}'

Pulumi Cloud Resource Provider

You can use Pulumi to manage and code review Deployment Settings and Review Stacks with the Pulumi Cloud Provider.

import * as pulumiservice from "@pulumi/pulumiservice";

const deploymentSettings = new pulumiservice.DeploymentSettings("deploymentSettings", {
 organization: pulumi.getOrganization(),
 project: "your project",
 stack: "your stack",
 github: {
 deployCommits: true,
 previewPullRequests: true,
 pullRequestTemplate: true,
 repository: "pulumi/deployment-automation",
 },
 sourceContext: {
 git: {
 branch: "refs/heads/main",
 repoDir: "pulumi-pet-shop",
 },
 },
});

Collaborate in the Cloud

At Pulumi, our engineering team writes a ton of code. We have hundreds of repositories with programs managing thousands of stacks and tens of thousands of cloud resources. These tools and services power production workloads, internal tools, business intelligence and process, and more. And of course we use Pulumi to ship all of it. When you’re an engineer working in one of these repositories, making a code change is the easy part. The hard part is figuring out how to get a tight dev loop going—in particular, faithfully verifying your changes before merging and deploying them. Are there instructions in the README for running locally? A Docker Compose file? Or maybe standalone dev environments? Can I figure out what config and upstream stack references are required to stand up a fresh dev stack? And what about service dependencies? Pulumi makes it incredibly easy to build in the cloud and write prolific amounts of code, however, with that increase in development velocity comes a need to standardize the entire process by which we develop, test, review, and ship our code.

We’ve heard similar stories from hundreds of customers. From startup teams of five to enterprises of thousands, they’re all looking for the same thing - so we went ahead and built it. With Review Stacks and Pulumi Deployments, you can ship with greater velocity, confidence, and collaboration.

Check out these resources and get started today:

Review Stacks documentation
Learn more about Pulumi Deployments
Join us in the #pulumi-deployments community slack channel

🚀🚀🚀 Happy building! 🚀🚀🚀

IaC Best Practices: Applying Stack References

Scott Lowe — Fri, 31 Mar 2023 00:00:00 +0000

This is the fourth post in a series of blog posts focused on Zephyr Archaeotech Emporium—our fictional company—and their use of Pulumi to manage their online retail store. In the first three posts, you saw how Zephyr’s initial use of Pulumi changed as the company grew, and how the use of short-lived per-developer stacks helped Zephyr’s application development team meet the demands of a fast-growing company. This post is a complement to the earlier post on structuring Pulumi projects, discussing how Zephyr uses Stack References to link their projects together and sharing some IaC best practices around the use of Stack References.

As you may have read in earlier Zephyr posts, the ultimate goal of the Zephyr series is to share best practices on the use of Pulumi to manage your infrastructure and application resources (using a fictional company and a somewhat complex containerized application as the use case). The series exposes those practices over time—not all right away, and not without also discussing the context for the recommendations. This is deliberate, demonstrating how “point-in-time” recommendations change based on the needs of the company and its requirements.

Reviewing Zephyr’s Current Status

This post continues the previous post in the series. Zephyr’s infrastructure footprint and Pulumi codebase is the same; that is, Zephyr now has three Pulumi projects (zephyr-infra, zephyr-k8s, and zephyr-app). Each of these projects handles a different aspect of Zephyr’s online store:

The “zephyr-infra” project handles base infrastructure (VPC and related constructs).
The “zephyr-k8s” project handles the Kubernetes platform layer, building atop the infrastructure components provided by the “zephyr-infra” project.
The “zephyr-app” project handles the deployment of the online store services onto Kubernetes, layering on top of the platform provided by the “zephyr-k8s” project.

The reasons for this structure are explained in structuring Pulumi projects post. For Zephyr, it came down to supporting multiple teams and allowing the application to evolve independently of the infrastructure that supports it.

You can continue to use the blog/multi-project branch in the GitHub repositories to see the state of Zephyr’s code and projects for this blog post.

It’s clear that there are dependencies across projects: The zephyr-k8s project needs to know the VPC ID and the subnet IDs from the zephyr-infra project. Similarly, the zephyr-app project needs to have access to the Kubernetes cluster details from the zephyr-k8s project. As the Zephyr team went about building the code that’s now in use in their multi-project architecture, they needed to decide how best to handle these cross-project dependencies. The Zephyr team knew that hard-coding output values from one project as configuration values in another stack wasn’t the ideal way (and in fact Pulumi strongly recommends against hard-coding output values), but what was the best solution to use? The answer to that question is stack references.

Linking Stacks with Stack References

Stack references allow you to access output values from one stack in another stack. In other words, stack references offer a way to programmatically “link” stacks in different projects for the purpose of sharing information. You can find more details in the documentation on stack references.

Using stack references, the Zephyr team can make the necessary connections they need:

The code handling the Kubernetes cluster can reference the VPC ID and subnet IDs from the base infrastructure layer.
Similarly, the code that is responsible for deploying the online store services can reference the necessary Kubernetes connection details (the Kubeconfig) for the Kubernetes cluster.

You can see how the Zephyr team did this for their projects, including code samples, toward the end of this post (in the section titled “Examining Zephyr’s use of stack references”). This graphical representation also illustrates Zephyr’s current repository and project structure as well as the stack references:

It’s worthwhile to note that any information that needs to be accessible from another stack via a stack reference must be exported as a stack output in the source stack. If you don’t mark it as a stack output, then it can’t be used in a stack reference. Adding stack outputs after the fact requires little effort and has no impact on existing infrastructure.

Also, any value retrieved via a stack reference is treated as a Pulumi Output, and therefore may require some extra work to transform values (such as the use of Output.apply). The recent addition of OutputDetails support in Pulumi—you can read more about OutputDetails in this blog post announcing the functionality—helps considerably in this situation. Some SDKs also have language-specific mechanisms that can help; for example, using Go’s .AsStringArrayOutput() method on a StackReference.GetOutput statement makes referencing subnet IDs from other project much easier.

While stack references are conceptually straightforward and not difficult to implement, there are some best practices to be mindful of regarding the use of stack outputs and stack references:

When it comes to stack outputs, export what is needed. If it needs to be accessed from outside the stack, export it; otherwise, don’t. Why? Stack outputs form the application programming interface (API) by which other Pulumi programs interact with this Pulumi program. Like in any API implementation, expose what the user cares about or is likely to need, and leave the rest as an implementation detail. You can add more stack outputs after the fact with only minimal code changes, and as has been mentioned already this is done with a quick pulumi up that has no affect on existing infrastructure (you’re modifying the stack object itself).
Be judicious in your use of stack references. This is sort of the “flip side” of the previous bullet. If you need to reference an output from another stack, then do so; otherwise, don’t. If you find yourself needing lots of stack references, then the way in which you’ve structured your projects may need to be re-examined so as to minimize the coupling between projects. Additionally, there is a potential “speed of light” concern here; if where you are executing your Pulumi code is across a high-latency link from where you store your Pulumi state, stack references will be affected by that latency.
If you need to expose lots of outputs on a stack, consider exporting a structured data object. It’s possible to construct a JSON object (or dict or struct, depending on your language) to hold all the stack outputs, and then export that object. Then a stack reference can read that object, resulting in a single call to the backend. Be aware, though, that you’ll need to write the necessary code to understand/import/unmarshall that JSON object in the referring stack, so there is a small amount of extra work required in this situation.
In line with parameterizing as much of your code as possible, also be sure to parameterize your stack references. A stack reference is built using an organization name, a project name, and a stack name. Don’t hardcode these values; instead, use configuration values to allow the users to specify from which source stack(s) the values will be referenced. You’ll see an example of that in the next section.
Plan for handling cascading stack updates. If a stack has outputs being consumed via stack references, what happens when those stack outputs change? The Pulumi Service supports webhooks that can help, and Pulumi is currently exploring options for having Pulumi Deployments help with this (see Pulumi Cloud Requests GitHub issue). In practice this should be uncommon, and frequent occurrences may be a good indicator that your resource partitioning (how resources are split among multiple projects) needs to be adjusted.

One thing not listed above is security—and that’s because a later post in the series focuses on this topic. You’ll want to keep the security of your stacks in mind when planning for the use of stack references. The teams managing the stacks where the stack references exist will need at least read-only access to the stacks where the outputs exist. (For more on this front, refer to the IaC Best Practices post on RBAC and security.)

Examining Use of Stack References

With these best practices in mind, you can examine the Zephyr team’s implementation to see how they put these recommendations into action.

In the code for the base infrastructure stack, you can see that Zephyr exported the essential values needed by the Kubernetes platform stack:

// Export some values for use elsewhere
export const vpcId = eksVpc.vpcId;
export const privSubnetIds = eksVpc.privateSubnetIds;
export const pubSubnetIds = eksVpc.publicSubnetIds;

In the Kubernetes platform stack, the Zephyr team parameterized the values needed for the stack reference. This particular approach, by the way, is key to preserving the per-developer stacks to which the Zephyr team has grown accustomed (each developer needs to specify the correct organization, project, and stack name):

// Grab some configuration values
// Some code omitted here
const infraOrgName = config.require("infraOrgName");
const infraProjName = config.require("infraProjName");
const infraStackName = config.require("infraStackName");

// Create a StackReference to get information from base stack
const infraSr = new pulumi.StackReference(`${infraOrgName}/${infraProjName}/${infraStackName}`);

As with the base infrastructure stack, only the key value needed by other stacks is exported (in this case, the Kubeconfig required to access this cluster):
```
// Export some values for use elsewhere
export const kubeconfig = eksCluster.kubeconfig;
```

Finally, in the application stack, the stack reference values are again parameterized, and the Kubeconfig—necessary for the application stack to deploy onto the provisioned Kubernetes cluster—is referenced via a stack reference:

// Grab some configuration values
const config = new pulumi.Config();
const k8sOrgName = config.require("k8sOrgName");
const k8sProjName = config.require("k8sProjName");
const k8sStackName = config.require("k8sStackName");

// Create a StackReference to get Kubeconfig from base stack
const kubeSr = new pulumi.StackReference(`${k8sOrgName}/${k8sProjName}/${k8sStackName}`);

All of the GitHub links in the paragraphs above reference the blog/multi-project branch in each repository, which has the code as of this blog post (and the earlier blog posts).

Summarizing Best Practices: Stack References

This post covered the following guidelines for working with Pulumi:

Export the minimum of what’s needed from your stacks. It’s not difficult to add stack outputs later, and you can do it with no impact to existing infrastructure.
Be judicious in the use of stack references. Reducing stack references where possible helps keep projects from becoming overly coupled together, much in the same way as minimizing stack outputs.
Use a structured data object for large numbers of values that need to be referenced. This enables Pulumi to retrieve all the data with a single call, but be aware you’ll need to write extra code to understand the data structure in the referring stack.
Parameterize your stack references. Don’t hardcode organization, project, or stack name values. Instead, pass these in as configuration values (with default values applied, where applicable).
Plan to build mechanisms to handle cascading stack updates. Using the Pulumi Service’s webhook functionality can be useful here. Frequent instances of cascading updates might be an indicator that your project structure needs to be revisited for optimal resource partitioning.

The next post in the IaC best practices series focuses on security and role-based access control (RBAC)—a topic you won’t want to miss! Before then, the next visit to the “Zephyr universe” will be to take a look at an oft-overlooked use case for Pulumi: local testing. Come see how to use Pulumi to automate local testing for development!

Pulumi Deployments: API-Driven Infrastructure at Scale

Evan Boyle — Wed, 29 Mar 2023 00:00:00 +0000

Delivering software has been conventionally driven by CI/CD workflows. A single commit is merged into a codebase, and a small, and static set of workflow runs are triggered by the CI system to update the appropriate environments (Dev -> Staging -> Production). This may have been fine when the only product a company had to offer was a single stateless web service, but increasingly companies are called upon to deliver cloud infrastructure as a product.

There are obvious forms of this, such as IaaS companies building database-as-a-service offerings and SaaS companies deploying self-hosted installations into customer environments. But there are less apparent forms like internal platform teams that are chartered with enabling self-service of Kubernetes clusters configured with best practices to thousands of application engineers within an enterprise. In any of these cases, engineers are delivering infrastructure as the product but only have conventional tools like gitops and CI/CD to reach for.

The assumptions that CI/CD systems make are limiting for infrastructure. We are not pushing a single application commit across three environments. We are templating infrastructure hundreds and thousands of times. Once that infrastructure is templated, we’re responsible for ensuring it is secure, up-to-date, and can be customized by self-service users. CI/CD is a poor fit due to the design choices that are locked in around source control and source events. CI/CD is typically configured via templated YAML that is checked into source control, with no way to create or customize behavior dynamically from a REST API. And beyond configuration, there aren’t REST APIs for triggering updates and workflow runs. The best you can expect is the ability to add more YAML to configure a cron job. What do you do if you need to create a self-service platform, or configure CI/CD across thousands of projects across your organization?

Today we’re excited to announce the release of a new version of Pulumi Deployments, a fully managed platform for shipping infrastructure at scale. It provides:

Compute - run infrastructure deployments and workloads (i.e. pulumi up) on fully managed, secure, and elastic compute.
Configuration - A simple, centralized format for specifying everything you need to deploy your infrastructure at the stack level. Source code, cloud credentials, OpenID Connect (OIDC), environment variables, build prerequisites, and dependencies are bundled in an atomic unit.
Composition - Trigger deployments via our REST API, click to deploy in the Pulumi Service UI, Git Push to Deploy, Remote Automation API, and other programmatic building blocks that you can compose to build custom cloud automation.

Pulumi Deployments is built on top of a powerful set of public REST APIs. Unlike any other platform, Deployments lets you configure CI/CD settings programmatically. Service-catalog scenarios and guard rails that enforce CI/CD best practices across thousands of projects are far easier to achieve when you have REST APIs that you can build software on top of. No more generating and templating YAML to configure infrastructure deployments at scale.

Deployment Settings and Triggers

Deployments can be configured at a stack level via Deployment Settings in three ways:

The Pulumi Service UI enables setting up OIDC and git push to deploy in just a few clicks
The Pulumi Service Provider lets you define deployment configuration as Pulumi code in you language of choice, reviewed and managed in source control.
The Deployment Settings REST API provides programmatic access to configure Deployments and CI/CD dynamically, without needing to generate or check in YAML.

import * as pulumi from "@pulumi/pulumi";
import * as service from "@pulumi/pulumiservice";

const config = new pulumi.Config();

// deployment settings configured via a pulumi program
const settings = new service.DeploymentSettings("deployment_settings", {
 organization: "service-provider-test-org",
 project: "test-deployment-settings-proj",
 stack: "dev",
 operationContext: {
 environmentVariables: {
 TEST_VAR: "foo",
 SECRET_VAR: config.requireSecret("my_secret"),
 }
 },
 sourceContext: {
 git: {
 repoUrl: "https://github.com/pulumi/deploy-demos.git",
 branch: "refs/heads/main",
 repoDir: "pulumi-programs/simple-resource"
 }
 }
});

Each stack configured with Pulumi Deployments gets access to a number of triggers that can be used to start a Pulumi operation (i.e. pulumi up):

Deployments REST API - an endpoint available at api.pulumi.com that can trigger a variety of pulumi (up/destroy/refresh/preview) operations. Calls to this trigger can override individual settings to customize deployment behavior (i.e. - deploy a different branch, or override a particular environment variable).
git push to Deploy - Deep integration with the Pulumi GitHub App that enables automatic previews on every pull request, and an automated update whenever code is pushed to the target branch.
Click to Deploy - trigger a deployment from the Pulumi Service Console with the push of a button. Very useful for day-to-day operational activities such as debugging a stuck stack, or rectifying drift via a refresh without having to pull a stack and its accompanying source code onto your machine.
Remote Automation API - Your favorite strongly typed Pulumi orchestration tool, Automation API, lets you offload update workloads into the cloud. Write deployment scripts that update thousands of environments simultaneously without worrying about compute.

You can mix, match, and compose Deployment Settings and Triggers to create a golden path to production that is perfect for your enterprise’s needs.

Example: Building RESTful Infrastructure APIs

Increasingly, infrastructure updates happen in response to user action and not source control events like a git commit. For instance, a SaaS database company might need to spin up dedicated compute and storage when a customer puts in a credit card and self-serves onto a specific SKU. This signup workflow might happen hundreds or even thousands of times an hour. Traditional deployment systems are optimized for infrastructure that spans, at most, a handful of environments (dev/staging/production).

Leading SaaS and infrastructure companies want the best of both worlds:

Custom, domain-specific REST APIs to create infrastructure on demand.
Desired state configuration that keeps track of what is deployed, and where.

You can build on top of the Deployments REST API to build your own infrastructure APIs. Here we have a simple Go web server that offers a RESTful interface over top of a static website. Users can create, update, and delete static websites, specifying the content for the site in the POST payload. Here’s our web server:

 router := httprouter.New()
 router.POST("/sites", server.create)
 router.GET("/sites/:id", server.get)
 router.POST("/sites/:id", server.update)
 router.DELETE("/sites/:id", server.delete)

 http.ListenAndServe(*addr, router)

Within the create handler, we first dynamically configure Deployment Settings for the stack, including the branch of the Pulumi program to be deployed, the AWS OIDC provider used for cloud credentials, and the configuration of git push Deployment Triggers`:

 err := s.client.patchDeploymentSettings(r.Context(), s.org, s.project, stack, DeploymentSettings{
 SourceContext: &sourceContext{
 Git: gitContext{
 Branch: s.branch,
 RepoDir: s.dir,
 },
 },
 OperationContext: &operationContext{
 Environment: map[string]string{
 "AWS_REGION": s.region,
 },
 OIDC: &oidcContext{
 AWS: &awsOIDCContext{
 RoleARN: s.roleARN,
 SessionName: s.sessionName,
 },
 },
 },
 GitHub: &gitHubContext{
 Repository: s.repository,
 Paths: paths,
 DeployCommits: true,
 PreviewPullRequests: false,
 },
 })

These settings configure the stack to trigger an update when new commits are pushed to the branch. In addition, our create handler also kicks off the initial update to create the infrastructure:

err = s.client.createDeployment(ctx, s.org, s.project, stack, createDeploymentRequest{
 DeploymentSettings: DeploymentSettings{
 OperationContext: &operationContext{
 Environment: map[string]string{
 "SITE_CONTENT": content,
 },
 },
 },
 InheritSettings: true,
 Operation: "update",
 })

Notice how this REST API call specifies InheritSettings: true so that Deployment Settings are read from the Pulumi Service and merged with the incoming request payload to create deployment configuration for this run. This stack also has to Click to Deploy via the Pulumi Service UI, so operational tasks like refreshing the stack can be done on demand without pulling down source code onto your local machine.

The Deployments Platform does all of the heavy lifting of managing deployment compute, providing asynchronous workflow orchestration, queueing, status and logging API access, and more. At the end, you get a Pulumi Stack and state file containing a manifest of all cloud resources and their properties managed by the Pulumi Service.

Deployment Settings and Triggers can be combined to ship infrastructure in novel ways with just a few lines of code. See the full RESTful infrastructure API example to try it out yourself.

Pave your Golden Path Today

Pulumi Deployments is the platform for shipping infrastructure at scale. It gives you the programmatic building blocks you need to deliver infrastructure on demand in a large organization. Get started with Deployments today. Happy automating!

Read Every Single Error

Evan Boyle — Tue, 21 Mar 2023 00:00:00 +0000

At Pulumi we read every single error message that our API produces. This is the primary mechanism that led to a 17x YoY reduction in our error rate. You’re probably wondering how reading error messages make them go away.

Doesn’t common wisdom tell us that we need a fancy observability toolchain, or to follow the Google SRE model? I can confidently say that you don’t. I’ll go a step further and state that throughout my career, every system I’ve worked on that relied on aggregate views of errors was a complete dumpster fire. In every team where we instead chose to read all the errors, reliability naturally improved over time.

I offer a concrete process that will drive your error rates down over time with math to back it up.

Read Every Error Message That Your System Produces

You should read every error message that your system produces. Simple but effective. Our team pumps every 5XX into a Slack channel and reviewing each of these is a top priority for the current on-call engineer. There’s a little more to it, but that’s the gist! Commit to this process and your error rates are guaranteed to drop. And I can prove it!

Reliability from First Principles

Why does reading error messages imply improving error rates? It isn’t magic. You still have to dedicate time to fix the bugs you’re shipping and make proactive investments. But you can model this process with a simple inequality:

(API Call Volume) * (Error Rate) * (Time to Triage an Error) < On-Call Attention

This comes with a few important constraints:

On-call attention is a fixed commodity
API call volume scales with your business
This process requires reading every error message

The end result is simple. Your error rate must improve over time to keep the scales balanced. If it doesn’t, the process becomes untenable.

Let’s assume you have one engineer on call, and they spend at most an hour a day triaging errors (they have other responsibilities too!) on each of the ~250 business days per year. That’s a cap of 250 hours per year that can be spent triaging errors. This is effectively fixed. Sure you can hire and split systems out into separate on-call rotations to increase capacity, but our goal is to scale exponentially with respect to humans, not linearly!

Let’s say that triaging an error is a 5-minute process that might involve any of the following:

Checking the issue tracker for a pre-existing ticket to add additional context, a +1, etc.
Reaching out to a teammate who just shipped some buggy code so they can get a fix out
Filing a well-documented bug
Starting a Slack thread with the team to raise a known issue that seems to be cropping up more regularly
Just opening a PR if it’s a simple fix

Remember we only have 250 hours or 15,000 minutes per year to triage with a single on-call engineer. At five minutes a pop we can triage ~3000 errors per year until things start dropping on the floor.

Now imagine you work on a new product and you just launched your MVP. It is early days, but there is some traction and you’re seeing API traffic at a rate of 1,000,000 requests per year. Your annual budget for triaging as we previously determined is 3000 errors, which yields a maximum permissible error rate of 0.3%, or a 99.7% success rate.

Over the next six months, the team iterates, listens to customers, and delivers a ton of value. As a result, traffic levels grow to a rate of 10,000,000 requests per year. But still, our team hasn’t grown much and we still have just one engineer on call at any given point in time meaning our error triage capacity remains fixed at 3000 errors per year. In order to keep up with triaging the error stream at the increased levels of traffic, the team must improve the error rate from 0.3% to 0.03%. And hopefully, this team continues to be successful, increasing API traffic superlinearly in years to come.

If you want to be able to read every error message, then the error rate has to come down as API traffic increases.

Why Should You Care?

Pulumi aspires to be the most reliable infrastructure that our customers interact with, and the benefits towards that end are reason enough for us. But this process is by no means free, and there is always an opportunity cost.

However, we noticed a powerful second-order effect emerge over time. The team began obsessing over the user experience. Following this process builds a visceral understanding of how your system behaves. You know when a new feature has a bug before the first support tickets get opened. You know which customer workloads will require scaling investments in the next 90 days. You know what features see heavy usage and which ones customers ignore. You’re forced to confront every wart in your application. Slowly, your team builds a better understanding of your customers and this trickles down into every aspect of product development. You begin to hold yourselves to higher standards.

A friend and engineer at a large-cap software company read a draft of this post and told me:

“I can’t imagine this process being set on any of the systems I work on at [redacted]. They are so prone to 500s and everybody shrugs their shoulders as if it is just commonplace.”

Apathy in place of customer obsession is not an option if you are a startup that wants to disrupt anything.

The SRE’s Folly

Error budgets and the SRE model are haute couture. Some preach that we should never look at errors at this level of granularity and instead use expensive tools that aggregate, categorize, and collect statistics on errors flowing through your system. But all of this automation can actually make things worse when you reach for it prematurely. Aggregating errors is a great way to gloss over important details early on. Collecting fancy metrics does not matter if your users are not happy. Cutting your teeth with the tools and processes that make sense for your level of scale is the only way to build a high-performance culture. Skipping straight to step 100 does not always help.

Admittedly, this process does not work for Google-level scale. But it works a lot longer than you might imagine. Pulumi manages a meaningful percentage of the resources deployed across all clouds. I asked large-cap software engineer about traffic levels and believe it or not, it is in the same order of magnitude as what we see at Pulumi.

We’re still reading every error.

IaC Best Practices: Structuring Pulumi Projects

Christian Nunciato — Fri, 17 Mar 2023 00:00:00 +0000

This is the third post in a series of blog posts focused on Zephyr Archaeotech Emporium—our fictional company—and their use of Pulumi to manage their online retail store. In the first post, you saw how Zephyr initially decided to go with a single Pulumi project for managing deployments of their online retail store application. In this post, you’ll see how Zephyr’s use of Pulumi changes as their company grows and evolves.

The first post about organizing code and stacks specifically mentions that the series will show “how Zephyr’s use of Pulumi evolves as the company grows and their online retail store application changes to accommodate their growth.” While the ultimate goal of this series is to discuss best practices for using Pulumi to manage a fairly complex containerized application, it’s important to note that many best practices are “point in time” recommendations: best practices are context-dependent and the recommended practices for a given organization will change as the company and its infrastructure needs also change. This blog post is an example of that evolution in action: the practices that worked well for Zephyr while they were smaller and had smaller infrastructure and less teams working on that infrastructure are no longer ideal as the company has grown.

Growing Development Teams

As a result of launching its online presence, Zephyr saw amazing success. The market for arcane artifacts and novel curiosities is booming, and Zephyr is becoming a leader in this market. Funded by this success, Zephyr has grown rapidly. Along with this growth, Zephyr’s software engineering and IT teams also grew and expanded. Fortunately, Zephyr’s use of per-developer stacks with short-lived ephemeral Git branches positioned the development teams well for this growth.

For a look at the state of Zephyr’s Pulumi code as it stood as of the end of the second post in the series, see the blog/dev-stacks branch of the pulumi/zephyr-app repository on GitHub.

Naturally this growth has resulted in some separation of duties. The IT team now has responsibility for the infrastructure that supports the online retail store. Within that team, some team members are focused on “core” infrastructure, while other team members are focused only on Kubernetes (a natural decision given Zephyr’s use of Kubernetes in their architecture). The application team continues to remain responsible for the online store application itself.

This left Zephyr again asking some questions about the best path forward:

Should they use a single project, or multiple projects? Zephyr had used a single Pulumi project up until now, but growth and organizational change meant they wanted to make sure they were using Pulumi in a way that best reflected their needs. What are the advantages and disadvantages of each approach?
If multiple projects, how many projects? What’s the best way to determine how many projects are needed? What factors play a role in this decision?

Structuring Pulumi Projects

One project, or multiple projects? And if multiple projects, how many projects? These are the core questions the Zephyr needed to resolve. In the end, there are a number of key factors that determine the answers to these questions:

Use case: Zephyr has only a single application (so far!), but in situations where there are multiple applications then keeping those applications (and sometimes their associated infrastructure, depending on other factors) in separate Pulumi projects is generally recommended. In some ways, “use case” is an aggregation of other factors—a new application with a distinct set of resources that do not share a lifecycle with existing resources and are potentially managed by a different team.
Company/department/team structure: How a company/department/team is organized should certainly be taken into account when structuring your Pulumi projects. In Zephyr’s early days with Pulumi, a single team managed everything so a single project was fine. Now with multiple teams, Zephyr will likely find using multiple projects to be more effective.
Security: Closely related to the previous bullet, security is a factor that affects how projects and stacks should be structured. In Zephyr’s case, the ability for one team to affect another team’s resources was present when everything was a single project. With multiple projects, Zephyr can use stack permissions and role-based access control (RBAC) in Pulumi Cloud to appropriately control who can affect what. (The IaC Best Practices blog post on RBAC and security discusses this topic in more detail.)
The relationship between resources: The relationships between resources—such as which resources are dependent upon other resources—is another factor that feeds into your Pulumi project structure. A “shared infrastructure layer” is often best handled in a separate project of its own, but may pull in resources that might otherwise have been kept with an application in its own project. (In some respects, this is a “counter-balance” to the first bullet point.) On the other hand, resources that belong exclusively to a specific application or function might be best bundled in the same Pulumi project.
Resource lifecycle: Do resources have a shared lifecycle, meaning they are always created, updated, or deleted together? If so, then those resources should be placed in the same project. If the resources do have a shared lifecycle, then placing them in the same project simplifies lifecycle management. A great example of resources with a shared lifecycle would be a Kubernetes Deployment and the associated Kubernetes Service.
Change rate of resources: Some resources may change very infrequently. Consider the example of an AWS VPC and associated objects. Once it is created, it may not change for extended periods of time, if ever. On the other hand, some resources may change frequently (in other words, these resources have a high change rate). A fast-moving software development team may revise the Kubernetes Deployment regularly to pull in a new container image with updated code. In a single project, combining resources that have a high change rate with resources that have a low change rate may be suboptimal.
Blast radius: Finally, there is the consideration of what happens in the event of human error, and limiting the “blast radius.” With a single project, all the resources in that project are within the blast radius—all of them could potentially be affected in the event of human error. Using separate Pulumi projects can limit the blast radius. It’s not something that should drive the decision all by itself, but it is a factor to be considered.

Now that you have an idea of some of the factors that affect how users structure their Pulumi projects, what are the advantages and disadvantages of each approach?

For single projects, the key advantage is simplicity. There’s no denying that a single project is simpler and less complex than working with multiple projects. This advantage, though, is also the key disadvantage. A single project does not afford the flexibility to let different teams manage different aspects of the infrastructure, for example, nor does it offer the ability to split resources that don’t have a shared lifecycle. And, as mentioned earlier, blast radius is a concern.

For multiple projects, the key advantage is flexibility. It’s true that the added flexibility comes at the cost of some added complexity (a future post will explore one aspect of that added complexity in the form of StackReferences). However, you gain the ability to separate infrastructure resources according to team ownership/responsibility, relationship to other resources (i.e., grouping resources in a “shared infrastructure layer”), or any of the other dimensions mentioned above.

In some ways, the decision points for single vs. multiple projects are analogous to the decision points for monolith vs. microservice: independent scaling and flexibility versus simplicity.

In general, most customers end up using multiple Pulumi projects per use case. As you’ve seen, though, there are a number of factors that influence that decision and will affect how many projects end up in use.

Moving Forward with Projects

Zephyr had a number of factors in play, not the least of which was the organizational change resulting from their growth and success. In the end, Zephyr decided to restructure their use of Pulumi for the online retail store into three separate projects:

An infrastructure project: This project handled the AWS VPC and related infrastructure resources.
A platform project: This project managed the Kubernetes cluster(s).
An application project: This project handled the actual online retail store itself.

You can use a combination of code refactoring and pulumi import to assist in the process of restructuring Pulumi projects, but it some cases it may be easier to create new infrastructure and migrate over to it.

In the code organization and stacks post, you read about some of the options for where to store Pulumi code relative to the applications it supports. At that time, Zephyr had decided to go with what’s called a monorepo approach, meaning that the Pulumi code was stored in the same repository as the application code. In conjunction with the project reorganization, Zephyr also decided to change how their code was organized:

Code for the infrastructure project moved to a separate “zephyr-infra” repository
A new “zephyr-k8s” repository housed the code for the platform project
The code for the online retail store application itself remained in the same “zephyr-app” repository

Note that relocating code (to a different GitHub repository or different filesystem location within a repository) generally has no impact on Pulumi. As long as the project name and stack name do not change, then changing the filesystem location won’t affect anything—keeping in mind, of course, that filesystem paths and such may need to be adjusted in code to account for any such changes.

After restructuring their projects, Zephyr’s arrangement of Pulumi projects and stacks, along with their associated GitHub repositories, looks something like this:

These repositories are available for you to review on GitHub (zephyr-infra, zephyr-k8s, and zephyr-app); use the branch/tag selector to find the blog/multi-project branch in each repository to see the state of the code as of this blog post. Be aware that the process for standing up the Zephyr online store is now a bit more complex; check the instructions in each repository for more details if you’d like to give these projects a spin.

Summarizing Best Practices: Pulumi Project Structures

This post covered the following recommended practices for working with Pulumi:

Use multiple Pulumi projects when you need the added flexibility that multiple Pulumi projects offer.
Align your Pulumi projects according to external factors like company/department/team structure, resource dependencies, lifecycle, and use case/application.
Change the filesystem location for Pulumi code as needed without impacting the resources managed by that code, as long as the Pulumi project name and stack name don’t change.

The next post will focus on some specific technical details related to Zephyr’s project reorganization; specifically, how they used StackReferences to share information between their Pulumi projects. Stay tuned!

IaC Best Practices: Enabling Developer Stacks & Git Branches

Christian Nunciato — Fri, 10 Mar 2023 00:00:00 +0000

In the first post about code organization and stacks, we introduced Zephyr, a fictional company that uses Pulumi to manage its online retail store. Following on from that post, which discusses code organization and stacks, this post explores two more questions users frequently ask when working with Pulumi in teams — namely, How can I best enable multiple developers to collaborate on a Pulumi project? And how can I use Git and Git branching to support this kind of collaboration? In this post, we’ll provide some guidance and best practices around these topics, using Zephyr and its online store as the use case.

The ultimate goal of this series is to discuss best practices for using Pulumi to manage a fairly complex containerized application. However, it’s important to note that these practices will emerge over the course of the series — not all immediately, and not all in the beginning. This is a deliberate decision to allow you to see how Zephyr’s use of Pulumi evolves as the company grows and its retail application changes to accommodate its growth.

Existing IaC Workflow

When we last met up with the Zephyr team, they were off and running, managing their newly refactored online store, Zephyr Archaeotech Emporium, with a single Pulumi project and two Pulumi stacks — one for development (dev) and another for production (prod). The team had chosen to use one Git repository (a monorepo) to manage the code for the online store and its infrastructure after refactoring the store into a set of containerized microservices deployed with Kubernetes on Amazon EKS.

For a snapshot of the code as it was at the end of the first post in the series, see the blog/original branch of the pulumi/zepyhr-app repository on GitHub.

Up to now, the two-stack approach has worked well for the team. It’s allowed them to focus on building new features, integrating those features into the shared dev environment by developing and testing locally (which we’ll cover in a future post) and merging their feature branches regularly into the base branch, main, using standard GitHub pull-request workflows. Around once a day, as the team’s changes are merged into main, the team lead (and original author of the Pulumi program) deploys them into the dev environment with Pulumi with the following commands:

git pull origin main
cd infra
pulumi stack select zephyr/dev
pulumi up

This builds new container images for any services whose code has changed since the last stack update, pushes those images to Amazon ECR, and updates the EKS cluster to deploy the services automatically.

Then, around once a week, as new features are ready for customers, the team lead deploys the main branch of the repository into production similarly:

git pull origin main
cd infra
pulumi stack select zephyr/prod
pulumi up

The deployment process is still a bit more manual than the team would like — but since updates typically take only a few minutes, the team lead is generally satisfied with the status quo.

A few more details worth noting out about Team Zephyr’s current setup:

Both of the project’s stack-configuration files, Pulumi.dev.yaml and Pulumi.prod.yaml, are checked into version control. This is safe and considered good practice (even with secrets) because it allows the team to review or change the configuration of a given stack easily by viewing or editing the code in each file.
The main branch is the only long-lived branch in the repository; all others (e.g., developers’ feature branches) are deleted once they’re merged into main.
Thanks to this workflow, the application deployed into the dev environment tracks closely with the latest code in the main branch. As such, the dev stack serves as a kind of pre-prod or integration environment that not only allows the team to dogfood new features as they’re added, but also to catch bugs or other regressions before they escape into production. When things do go wrong, the team can review the list of recent stack updates to identify the source of the defect and get it fixed.
Zephyr is using the Pulumi Service backend with a single organization (that everyone on the team belongs to) and separate AWS accounts for the dev and prod environments. Everyone on the team has read/write access to the AWS account used for development, but only the team lead has access to the AWS account used for production.

Growing Pains

Life for Team Zephyr was good — for a while.

Then one day during a particularly active sprint, a developer who’d started working on an improvement to the checkout process realized it might benefit from an updated version of Redis, which appeared to be presently pinned at 6.0 in the Pulumi program:

// ...
const checkoutRedisDeployment = new k8s.apps.v1.Deployment("checkout-redis-deployment", {
 spec: {
 template: {
 spec: {
 containers: [{
 image: "redis:6.0-alpine",
 // ...

A quick glance at the dev and prod stacks in the Pulumi Service confirmed this was the case.

After a quick spike to validate the enhancement locally, the developer felt confident that a Redis upgrade was the way to go — but since the deployed version of Redis would ultimately dictate the code that’d have to be written to finish the feature for real, they couldn’t really be sure without testing the theory with a full deployment. What they really wanted was to be able to deploy a Redis upgrade, along with the associated code changes, into an AWS environment as much like production as possible — the dev environment, say — without having to go through the process of making a pull request, getting it merged, and waiting for the team lead to deploy the dev stack only to realize later that the change didn’t actually work as expected, and be forced to revert.

In situations like these, it can be tempting to suggest that the developer simply pick up and use the dev stack — especially on small, fast-moving teams like Zephyr’s where everyone already has access to the Pulumi stack and the shared cloud environment. Indeed, Pulumi makes this sort of thing easy: to deploy their local changes into the dev environment, our developer need only check out their feature branch, select the dev stack, run pulumi up, and then wait a few moments for the changes to be deployed:

cd infra
pulumi stack select zephyr/dev
pulumi up

This approach definitely works — but it presents at least a few problems:

The team would lose all of the practical benefits of having the dev stack track closely with the latest content in main.
The team lead’s deployment of main into dev later that day would overwrite the developer’s own feature-branch deployment. There’s a good chance the lead would notice the diff of the deployment preview and deduce that someone else on the team had possibly deployed into dev out of band:
```
 kubernetes:apps/v1:Deployment (create):
 spec: {
 template: {
 spec: {
 containers: [
 [0]: {
- image: "redis:6.0-alpine"
+ image: "redis:7.0-alpine"
 ...
 }
 ]
 }
 }
 }
 ...
```
But if they didn’t, the developer’s ad-hoc deployment would be lost. At best, it’d force both team members to drop what they were doing and get in touch to figure out what to do next. At worst, it could lead to confusion and the potential for shipping untested code.
In addition to raising the risk of developers overwriting each other’s work, active collaboration the same stack increases the likelihood of deployment conflicts. On the one hand, this is a good thing in that Pulumi prevents same-stack updates from running concurrently:
```
$ pulumi up
error: [409] Conflict: Another update is currently in progress.
```
But on the other, it’s a not-so-good thing in that it’s a reduction in development velocity: The update that can’t run because another update is in progress is an update that will have to wait until sometime later. The more often this happens, the longer it ultimately takes for the team to get the validation it needs to keep moving forward.

So facing these issues, what should the Zephyr team do? The dev stack had originally been created for rapid development — and that worked well, when there was only one developer working on infrastructure. With several, however, things clearly get complicated. Should the team be using a dev stack at all? If so, how? And if not, what should they be doing instead?

Stepping Back to Reassess

With what we’ve learned so far, we can return to the questions we raised at the beginning of this post and begin to sketch out a few recommendations for Team Zephyr. Those questions, again, were:

How can I best enable multiple developers to collaborate on a Pulumi project? For Zephyr, an ideal answer to this question would give every member of the team the ability to iterate on both application and infrastructure code concurrently, deploying as often as needed without having to worry about disrupting the work or the velocity of anyone else on the team.
How can I use Git and Git branching to support this collaboration? So far, the feature-branch/pull-request workflow has been serving the Zephyr team well. Should they continue on the same path, merging frequently to main and deploying to dev and prod from there? Or is there some alternative workflow that might serve the team better, either now or at some time in the future?

Let’s answer these questions now by taking a look at how the Zephyr team handled them.

Using Developer Stacks

We’ve seen a few ways that having multiple developers share a single stack for ad-hoc deployment can lead to various problems. But we’ve also seen at least one situation in which being able to run ad-hoc deployments — without the risk of colliding with existing environments — can be quite valuable. In fact there are many reasons you might want to do this — and fortunately Pulumi makes it quite easy to do so.

The best practice is to allow developers to create and manage their own Pulumi stacks. In this workflow, developers working on feature branches are able to iterate freely on both application and infrastructure, deploying as often as necessary into a shared cloud environment (typically a sandbox environment, isolated from production) and destroying cloud infrastructure when it’s no longer needed.

For the developer, the typical workflow looks something like this:

Check out a new feature branch
Make changes to application and/or infrastructure code
Select your personal developer stack with pulumi stack select
Deploy your stack into your team’s shared cloud environment with pulumi up
Commit your code and submit a pull request, optionally including a link either to the Pulumi update (e.g., when using the Pulumi Service) or to certain key resources belonging to the stack — for example, an application’s primary HTTP endpoint

The benefits of a workflow like this one are many:

No more resource-naming collisions. Because Pulumi names cloud resources uniquely by default, no two stacks will ever produce cloud resources with the same names.
No more deployment conflicts. When every developer has a stack of their own, they can deploy to the sandboxed environment as often as necessary, without having to wait for another deployment to finish first.
Everyone on the team has their own production-like environment to use for end-to-end testing.

For Zephyr, the quickest path to adopting this workflow was to use the dev stack as a foundation for new developer stacks. To do that, team members simply ran the following Pulumi commands:

pulumi stack init zephyr/developer-name --copy-config-from zephyr/dev
pulumi up

Developer stacks are considered ephemeral, so they’re torn down with pulumi destroy when not in use in order to cut down on cost. Stack configuration files (e.g., Pulumi.developer-name.yaml) are checked into version control and stored alongside other stacks like Pulumi.dev.yaml and Pulumi.prod.yaml.

Keeping Feature-Branch Workflows

We’ve also seen that the Zephyr team’s existing Git workflow (short-lived feature branches merged frequently into a long-lived base branch) has worked well for them — and it turns out this workflow also works well when combined with the use of developer stacks.

For example, after creating and deploying a new stack to validate and test the proposed Redis upgrade, our Zephyr developer simply included a link to the application (deployed with their dev stack) in the associated GitHub pull request. Reviewers were then able to follow the link to the online store to see the effects of the shopping-cart change firsthand, which meant they were able to approve and get the change merged quickly and confidently knowing it’d work when deployed into production.

Naming Stacks Clearly

Lastly, having adopted the use of developer stacks, the Zephyr team realized the name “dev” probably didn’t work quite as well as it once had for the dev stack. Since the stack and its long-running deployment had essentially been serving (and continued to serve) as an environment for testing changes before releasing them into production, the team decided to rename the dev stack test instead. To do that, they simply ran:

pulumi stack select zephyr/dev
pulumi stack rename zephyr/test

Viewing the Next Iteration of Code

You can view the second iteration of Zephyr’s Pulumi and application code — the iteration that corresponds to the decisions described in this blog post — by navigating to the Zephyr App GitHub repository. From the branch/tag dropdown, switch from the main branch to the blog/dev-stacks branch.

From that GitHub repository, you can also choose to deploy the Pulumi code yourself. Full instructions for deploying the code are found in the repository.

Summarizing Best Practices: Developer Stacks and Git Branches

In this post, we covered several new guidelines for working collaboratively with Pulumi, all aimed at optimizing for high team and developer velocity:

Use short-lived feature branches and merge them frequently into a single base branch.
Deploy the base branch regularly into a long-running pre-prod environment to continuously test the integration of the components of the system.
Enable developers to create and use dev stacks of their own as sandboxes for development, testing, and collaboration.
Name your stacks clearly and in alignment with the environments they deploy into — test, staging, production, etc.

The next Zephyr blog post will dive into how the Zephyr team structures their Pulumi projects. Stay tuned!

IaC Best Practices: Understanding Code Organization & Stacks

Aaron Kao — Mon, 20 Feb 2023 00:00:00 +0000

This is the first in a series of blog posts that explores how a fictional company—Zephyr Archaeotech Emporium—uses Pulumi to manage their online retail store. This post explores a couple of common questions that users ask when working with Pulumi; specifically, where should I store my Pulumi code? And how do I support multiple environments with Pulumi? This post will provide some guidance and Infrastructure as Code best practices around these topics, using Zephyr and their online store as the use case.

The ultimate goal of this series is to discuss recommended practices for using Pulumi to manage a fairly complex containerized application. However, it’s important to note that these practices will emerge over the course of the series—not all immediately, and not all in the beginning. This is a deliberate decision to allow you to see how Zephyr’s use of Pulumi evolves as the company grows and their online retail store application changes to accommodate their growth.

Setting Up the Scenario

Zephyr is short for Zephyr Archaeotech Emporium, the fictional company in our scenario. Zephyr is an online retailer that specializes in the sale of “rare arcane artifacts and replicas.” Over the past few years, the company has experienced a significant increase in its online presence, making it easier for customers to purchase unique and mysterious items. Zephyr’s collection includes a variety of rare and unusual objects that are difficult to find elsewhere, making it a popular destination for collectors, enthusiasts, and adventurers.

Moving to a Containerized Architecture

Zephyr’s application is the second generation of their online store. Their first generation was a monolith that they manually deployed. It was sufficient in Zephyr’s early days, but as word about Zephyr spread and the volume of traffic to their online store increased, Zephyr found it increasingly difficult to deploy small, incremental changes to the store’s code. As a result, the velocity of their development efforts slowed. In preparation for the next phase of their growth—driven by robust sales of arcane artifacts—Zephyr’s team evaluated a number of different architectures that would enable them to increase development velocity and give them greater flexibility in scaling different aspects of their online store. In the end, they settled on a containerized architecture deployed to Kubernetes because some of the existing team was already familiar with these technologies.

Here’s a diagram of their second-generation application architecture:

As you can see in the diagram, the application has been broken down into microservices. The UI service communicates with other backend services via HTTP APIs; the various backend services communicate with various databases like MySQL and Redis for data persistence and/or caching. The backend services are responsible for implementing the functionality of a typical e-commerce site like orders, checking out, and managing the inventory catalog. While the choice of Kubernetes does afford Zephyr some much-needed functionality—like the ability to easily scale different services within the application—it also brings with it an added level of complexity that did not exist with their previous monolithic architecture. The Kubernetes manifest to deploy their application tops out at just over 1,000 lines of YAML!

Astute readers may note that Zephyr’s application looks very similar to this application; in truth, this series uses a fork of the AWS Containers retail store example as the application in this fictional scenario. (Many thanks to the AWS Containers DA team for their hard work here!) This is an admittedly over-engineered application, but it does afford the opportunity to address a range of customer scenarios in this blog series.

Managing Infrastructure and Application Deployments

As part of the switch to their new architecture, Zephyr decided they wanted to use Pulumi to manage both the infrastructure and the application deployment. Why Pulumi? Zephyr’s team recognized that moving to microservices on Kubernetes was going to make it more difficult to create reproducible, consistent implementations of their application. Zephyr’s team also knew that adopting infrastructure as code would help them with fast and repeatable deployments with little additional complexity. Another bonus was being able to use programming languages they already knew.

As they prepared to embark on using Pulumi, a couple of questions came up for the Zephyr team:

Where should they store their Pulumi code? The ability for Zephyr’s developers to use a full programming language to manage the infrastructure and deployment of their online store is a huge plus, but should this code go in the same repository as their application code? Or a different repository?
How do they address the need for multiple environments? It’s clear to the Zephyr development team that they’ll need multiple instances of the online store (for things like testing, QA, or development). What’s the best way to handle this when using Pulumi?

Let’s examine each of these issues in a bit more detail, and see how Zephyr chose to proceed.

Storing Pulumi Code

When it comes to answering the question of where to store Pulumi code relative to the application(s) it supports, there are two basic options:

In the same repository as your application code: This approach has the advantage of being simpler, but may not offer the necessary flexibility when dealing with multiple teams or varying levels of access control.
In a separate repository from your application code: Using a separate repository adds some complexity, but does provide greater flexibility to more easily address a variety of organizational requirements and varying levels of access control.

So which approach is best? That will depend on a number of different factors, many of which are outlined in the organizing projects & stacks documentation. Some other factors that users need to take into consideration include:

Who is responsible for maintaining the code? If the application code and the Pulumi code are managed by different teams, then using separate repositories may be the best approach.
What sort of access controls are needed? If the organization needs different access controls for the application code and the Pulumi code, then separate repositories are generally needed.
What is the relationship between the infrastructure resources managed by Pulumi and the applications? If the infrastructure is “shared,” meaning it is used by multiple applications, then keeping the Pulumi code in a separate repository may be preferred. If the infrastructure is dedicated to a specific application, then keeping the application code and the Pulumi code together may be preferred.
What about CI/CD? If an organization will use a CI/CD pipeline, that may affect how you organize your code. (This is something we’ll delve into more deeply later in this series.)

It’s worth stressing that the answers to these questions may change over time as organizations grow and their applications evolve. Addressing that change and its effects on an organization’s Pulumi projects and stacks is something we’ll be discussing later in this series.

The discussion above is working from the assumption that Pulumi users will store their Pulumi code in a version control system, like Git. Strictly speaking, using a version control system isn’t required for Pulumi to function (there is one exception we’ll touch on later in this series when we discuss Pulumi Deployments), but we very strongly recommend that all users use Pulumi in conjunction with version control.

In the case of Zephyr, their team felt like a monorepo approach (storing Pulumi code and application code together) was most appropriate for their specific requirements. Zephyr is a relatively small organization with a single team of developers who are responsible for managing both the cloud infrastructure as well as the development and deployment of the online store application. Zephyr’s team also decided to use a single Pulumi project—for now. (Be sure to follow this blog series to see how that evolves, and why!)

Addressing the Need for Multiple Environments

The second question Zephyr encountered is how to handle the need for multiple instances of their application. In addition to a production instance—which is the instance behind Zephyr’s public-facing online store—Zephyr also felt they needed an environment for the developers to use in testing changes to the online store.

This use case—needing to have multiple, separate instances of the infrastructure and applications created by a single Pulumi program—is exactly what Pulumi stacks were designed to address. Each stack is a separate instance of the resources created by a Pulumi program within a project. Further, each stack has its own independent state, and each stack has its own configuration values. Stacks can be short-lived (meaning the associated resources are also short-lived), or stacks can be long-lived. Aside from the cloud resources created by the Pulumi program, stacks are lightweight and simple to create or delete.

Zephyr decided to initially start with two stacks: a production stack (named “prod”) and a development stack (named “dev”). As you’ll observe throughout this series, this is a decision that is easily adjusted over time as your organization’s requirements change.

Organize your infrastructure with stacks

Model separate development, staging, and production environments as Pulumi stacks in a single project, each with its own state and configuration.

Get started

Viewing the First Iteration of Code

You can view the first iteration of Zephyr’s Pulumi and application code—the iteration that corresponds to the decisions described in this blog post—by navigating to the Zephyr App GitHub repository. From the branch/tag dropdown, switch from the main branch to viewing the blog/original branch.

From that GitHub repository, you can also choose to deploy the Pulumi code yourself. Full instructions for deploying the code are found in the repository.

Summarizing Best Practices: Organizing Code and Stacks

This article introduced you to Zephyr, the fictional company for this blog series, and reviewed the application architecture for Zephyr’s online store (a containerized application running on Kubernetes). Throughout the blog series, this application serves as the use case around which various recommended practices are discussed.

In this post the following guidelines were covered:

Use version control for your Pulumi code. Although it isn’t strictly required in many cases, a number of the benefits of adopting infrastructure as code are lost without the use of version control.
Place your Pulumi code into version control repositories in a way that aligns with organizational requirements. There is no “right” or “wrong” approach; choose the approach that enables you to satisfy your organization’s requirements around access control, code/repo ownership, and other organizational requirements.
Leverage Pulumi stacks to create multiple instances of cloud resources with a single Pulumi program. Pulumi stacks were created to address the need for organizations to create multiple instances of cloud resources, each with its own state and its own set of configuration data, from a single Pulumi program.

For now, Zephyr has decided to go with a monorepo approach—a single repository that contains both their application code and the Pulumi code to manage the infrastructure resources. All of the resources are defined in a single Pulumi project, with multiple stacks that correspond to development and production environments. Over the course of this series, we’ll see how Zephyr’s use of Pulumi changes as Zephyr grows and their application evolves.

The next Zephyr blog post will examine how the Zephyr team adjusts their use of stacks and Git branches to support developer productivity. Stay tuned!

FinOps With Pulumi

Matt Small — Tue, 14 Feb 2023 00:00:00 +0000

What is FinOps?

The FinOps Foundation eloquently defines FinOps as “an evolving cloud financial management discipline and cultural practice that enables organizations to get maximum business value by helping engineering, finance, technology and business teams to collaborate on data-driven spending decisions.” Simply put, FinOps is the continuous effort to control cloud spend.

Just as organizations have adopted operations-focused best practices into software development cycles and have considered how to best insert security best practices along the way, financial best practices may also be codified by developers writing cloud programs.

Adopting a FinOps practice brings real world OpEx(Operating Expenses) considerations to the decisions that cloud engineers regularly make. When you add FinOps to your Cloud Center of Excellence, technical teams bring financial context into service level and recovery objectives, scaling and placement decisions, and business continuity planning. Developers are typically more conscious of the resources they are using and consider the impact of their application design decisions at both development, test and production scale. Therefore organizations usually realize substantial OpEx savings while better forecasting their growth.

Who is Responsible for Cloud Financial Decisions?

In most smaller organizations, developers themselves are often responsible for the decisions that impact cost. In larger organizations, this responsibility typically falls to a central IT or Cloud Platform team, often seen as the cost center for infrastructure, and the best team to optimize that spend continuously. Some organizations are evolving toward FinOps as a practice, with dedicated teams responsible for enabling the organization to make smarter financial decisions and negotiating with cloud vendors for discounts.

Ultimately, the cloud account owner is responsible for the cloud bill, and failure to pay that bill will result in the suspension of cloud services. Given that infrastructure is often code, developers or IT teams are often made responsible for the task of optimization. Optimization recommendations may range from simply deleting orphaned cloud resources to re-architecting the entire application to use a modern serverless framework that would consume less resources overall.

Establishing FinOps Practices

The first step of establishing many FinOps practices is determining the allocation of costs. It is easy to allocate costs to individual users launching cloud services, but it becomes compoundingly hard once pipeline automation, team structure, and business unit reporting come into play. As an organization scales, it grows increasingly difficult to allocate costs across different business units and teams, across environments such as development versus production, or across application component services such as the data layer. Further, how does an organization account for supporting systems and services running alongside applications in the cloud, including third-party commercial software?

When starting your FinOps Journey is essential to understand cost allocation, typically done with consistent and enforced tagging of cloud resources. Creating a Tag Policy is covered below.

FinOps: Reactive vs. Proactive Optimization

Most FinOps processes today are reactive; once you have received your bill, you will observe (and pay) the costs and resources accounted for, review your architecture and make necessary adjustments such as reserving, reallocating or removing resources. Next month, you will do it again, also factoring in your company’s cloud growth, application design changes, team shifts and other dynamics that occurred in the meantime. Many platforms, tools, and job descriptions are oriented toward this reactive optimization process.

Proactive optimization and enforcement means providing real time feedback on how cloud engineering architectural changes may impact cost and creating preventative policies which do not allow for budget-busting cloud services to be provisioned. This is an evolving practice in many modern cloud-centric organizations. As more cloud operations are defined as code, this preventative, proactive optimization model is becoming more popular.

Financial optimization can also come in the form of strategic sourcing; most cloud providers offer multiple discount mechanisms, including large enterprise discount plans for committed consumption that may extend to third party SaaS software purchased through the cloud marketplaces, as Pulumi is. This level of optimization will provide dramatic savings on your top-line spend.

FinOps with Pulumi

Using cloud engineering practices with Pulumi, you have the opportunity to implement both proactive and reactive models to control your cloud spend.

As an infrastructure as code platform, Pulumi is in the unique position to be proactive about managing your costs.

You can design guardrails on your provisioning to limit expensive resource types or quantities,
Direct resources to reserved allocations,
Ensure resources are tagged and traceable to the correct cost center,
Connect resources to your observability and dedicated FinOps platforms,
Provide preventative visibility and control.

Consider uses for CrossGuard Policy as Code, multi-language components for defining resource abstractions, and Automation API to package up orchestration into web service endpoints, as described in Pulumi Features below.

If you are proactively managing your infrastructure, then reactively, the largest contributor to financial overrun is drift; the misalignment between the desired state and the actual state. Pulumi is used to define your desired state; however, many things may cause the actual state to begin to deviate from the desired state in your code, such as manual actions from the cloud console or ad-hoc deployment pipelines.

Drift detection, which is the name given to continuously scanning your environment to compare the actual state with the desired state and alerting and/or automatically remediating the difference in state, is a primary method of managing these creeping cloud costs. Leverage an automation system, such as a CI tool, to implement continuous scanning against your Pulumi code as described below.

Remediation of any infrastructure mismanagement requires infrastructure reconfiguration action to be taken; often resources need to be replaced with better suited, more cost effective resources, and resources that were provisioned outside of acceptable policies may have to be destroyed. However, any remediation should carefully consider the underlying reasoning for the misalignment in the first place. That should include both the reason for the initial deployment specifications as well as any decisions for why that’s no longer acceptable. Be sure to consider all architectural decisions, such as disaster recovery and business continuity planning, regional latency, and any other impact to service level objectives (SLOs). Understanding the context will help you code preventative measures into your Pulumi programs going forward and the self-documenting nature of infrastructure-as-code will ensure your decisions are memorialized for future developers and operators, for example, in a Stack README as described.

Automation of remediation is a goal for many cloud engineering organizations; if the perfect state is defined in code, then any drift may indicate the need to reconcile back to the desired perfect state. Given that some cloud resources may necessarily drift due to limitations in how they are managed, a combination of alerting and remediation of known patterns is implemented. For example, you might choose to detect and deprovision any newly instantiated resource not provisioned by Pulumi, but allow manual resizing of resources if needed. Just be sure to capture any infrastructure and improvements you want to make default and/or repeatable as infrastructure as code.

Best First Action: Implement a Tag Policy

Getting started with a FinOps usually falls to the resource owners or Cloud Engineering teams themselves. The more resources you have, the more complex and noisy things are going to get and the harder it will be to both allocate costs and resources where they are needed, as well as identify and eliminate the unneeded ones.

Implementing Tag Policy is the best first action to take. Tags are the set of required cloud metadata tags for any resources provisioned by your organization. While not every resource in every cloud may be tagged, most are. Getting into the practice early will help you maintain accountability for what you are running, demystify your cloud bills as they grow larger, and help update the application architecture or make cost-effective product decisions long-term.

A Basic Tag Policy for FinOps will include:

Owner - the person responsible for this resource that you would contact if you had to (e.g., johndoe@pulumi.com)
Origin - the pipeline, process, and/or system through which this came to be (e.g., ArgoCD, Jenkins, Pulumi)
Business Unit - the team responsible typically maps to the cost center associated (e.g., Sales, Engineering)
Environment - the teams/customers this deployment services (e.g., Dev, Staging, Production)
Application Component - the part of the application that this resource serves (e.g., data pipeline, authentication, front end)*

You will now be able to report on and manage resources in those tag groups. It’s advisable that you do not create too many required tags, or it will quickly get difficult to manage and it increases the likelihood of multiple tags applying, which reduces the insightfulness of the data.

Read more about Tag Policy implementation with Pulumi below, and for extended reading, refer to this post, Automatically Enforcing AWS Resource Tagging Policies from CEO, Joe Duffy.

Pulumi Features in Context

Multi-Language Components

Multi-language components allow you to create repeatable infrastructure patterns, modular abstractions of resources and/or groups of resources that meet your desired configuration standards. Packaging up resources within and across clouds, implementing standard deployment best practices, and making those components available to developers reduces the cognitive load that they have when building new applications from those pre-assembled, ready-to-run components.

Use these guardrails to enforce Tag Policies, for example that every tag for “Production” is actually “Production” and not “Prod” or “production” (yes, case sensitivity matters). Or you may ensure teams only deploy resources such as VMs to regions where you have allocated Reserved Instances or other financial benefits in your application architecture. With MLCs end users have access to your component and not to the underlying infrastructure resources, guaranteeing repeatabality, however, in order to realize the benefits of MLCs, your developers must be empowered to leverage them, but limited from using other methods.

CrossGuard

CrossGuard Policy as Code allows an Organization to define global rules for how resources may be provisioned and prevent resources from being provisioned by Pulumi in any other configuration. Pulumi Stacks that do not meet the Organization’s financial, security, data protection, or other pre-defined standards are prevented from provisioning any resources, and the developer is instructed to correct their desired configuration or program in accordance with the Organization-wide policies.

Policy as code ensures that any Pulumi code written and run meets an organization’s stated policies. It is the ultimate preventative weapon to prevent financial, security, or other placement and provisioning decisions that may lead to exposure or overrun.

Go here for examples of FinOps policies to get you started!

Automation API

Of the many things Automation API allows you to do, one is you can create an HTTP endpoint for a Pulumi program. Organizations have used this to control easy on/off switches for applications such as provisioning/deprovisioning single tenant SaaS architectures, short lived environments, and complex testing and blue/green deployments. Because of the encapsulation of a Pulumi program and the simplification this provides, this endpoint may be directly exposed to those responsible for making the financial decisions in the moment, such as a Sales Engineer who must turn on a new POC environment for testing who then must be sure to shut that off when completed. Oftentimes, these decisions are automated in processes and pipelines.

Pulumi FinOps in Practice: Sweeping up Dev and QA

A common automation in FinOps is to destroy things that don’t meet your Tag Policy. These “Sweeper Bots” will continuously query AWS for resources with specific tags and delete them. Read more about how Pulumi has implemented this using Pulumi to set tags and Lambda to sweep through and remove resources that meet certain specifications in this post, Controlling AWS Costs with Lambda and Pulumi.

Conclusion

In the manner of “Crawl, Walk, Run”, it’s best to start by implementing policies as code in a manner that will cause the most benefit while causing the least amount of change and/or downtime. Make sure you have the appropriate billing tags applied to all supported resources, ensuring you are able to allocate costs to the appropriate business units in your company. Then add in additional policies to make sure you are using instances covered by any reserved instance or similar savings plans. Continue evolving your infrastructure and policy as code strategy to incorporate both preventative and reactive checks and balances.

Have questions about using Pulumi when implementing FinOps in your IaC pipelines? Contact us and let us know your questions or request a demo.

Pulumi's Declarative and Imperative Approach to IaC

Ringo De Smet — Fri, 23 Sep 2022 00:00:00 +0000

On a regular basis, articles and tweets pass by discussing whether some specific tool is imperative or declarative.

It’s no surprise that Pulumi is often the tool being debated. What if I tell you that Pulumi is imperative, declarative and imperative?

When we look at our Frequently Asked Questions, we read the following on the declarative versus imperative topic:

Pulumi is a declarative tool that uses imperative languages to define your end state. The language is used for authoring your program. It’s not used for talking to the cloud provider API.

It is good to know to refresh what declarative and imperative mean:

Both terms refer to how the user provides direction to the automation platform. With an imperative tool, you define the steps to execute in order to reach the desired solution. With a declarative tool, you define the desired state of the final solution, and the automation platform determines how to achieve that state. (Source)

The title of the article mentions imperative and declarative both. Pulumi leverages the best of both worlds into our product.

Pulumi tries to offer a solution where our customers are only limited by their imagination rather than the tool at hand. Let me use the Pulumi architecture to highlight why I mention imperative twice.

Pulumi Architecture

Here is the diagram from our How Pulumi Works page:

You code your infrastructure in your preferred programming language. When you are done coding, you run pulumi up and the Pulumi CLI starts the language host for your selected programming language, as well as the required providers. The interaction between these 3 parts of the architecture results in the actual creation or modification of your infrastructure.

An Imperative Part: The Language Host

Under the hood, the Pulumi CLI does a lot of things, but one of the first actions is starting the language runtime, which is configured in the Pulumi.yaml project file:

name: blog_code
description: Example for declarative blog post
runtime: python
main: blog_code/

Once the language runtime is up, it turns to the program file. Here is a small Python example:

from pulumi_aws import s3

bucket = s3.Bucket('bucket')

for i in range(10):
 s3.BucketObject(
 f'object-{i}',
 s3.BucketObjectArgs(
 bucket=bucket.id,
 key=str(i),
 )
 )

A Pulumi program models the to-be state of your infrastructure. If you read this program, you can find that we define 11 resources as our to-be infrastructure:

1 AWS S3 bucket
10 Objects in the bucket created in the previous step

The stack itself is also modelled as a resource, and it is the parent to all other resources.

$ pulumi up

Updating (<masked>/dev)

View Live: https://app.pulumi.com/<masked>/blog_code/dev/updates/1

 Type Name Status
 + pulumi:pulumi:Stack blog_code-dev created
 + ├─ aws:s3:Bucket bucket created
 + ├─ aws:s3:BucketObject object-0 created
 + ├─ aws:s3:BucketObject object-1 created
 + ├─ aws:s3:BucketObject object-3 created
 + ├─ aws:s3:BucketObject object-2 created
 + ├─ aws:s3:BucketObject object-6 created
 + ├─ aws:s3:BucketObject object-4 created
 + ├─ aws:s3:BucketObject object-5 created
 + ├─ aws:s3:BucketObject object-7 created
 + ├─ aws:s3:BucketObject object-9 created
 + └─ aws:s3:BucketObject object-8 created

Resources:
 + 12 created

Duration: 13s

While this is definitely an imperative program, there is one important thing to understand: Instantiating an s3.Bucket, s3.BucketObject, or any other Pulumi resource should not be interpreted as an imperative creation of the resource in the language host. Behind the scenes, any resource instantiation in the language host triggers a Register Resource request to the Pulumi deployment engine. All these Register Resource requests together form the resource model you as an infrastructure developer want to get in the end.

Running your program always sends the full resource model to the Pulumi deployment engine regardless of what state your current infrastructure is in.

Our previous example was shown in Python, but recently we also delivered support for YAML. The creation of our S3 bucket could be converted to this snippet:

resources:
 bucket:
 type: aws:s3:Bucket

The separation of language support from the engine allows us to offer both imperative & declarative solutions. But remember that none of the supported languages do any of the provisioning directly in the language host.

If the resource provisioning is not taking place in the language host, where is the magic happening then?

The Declarative Part: The CLI and Engine

In the previous step, you found out that the language host sends requests to the engine to fulfill your to-be infrastructure.

It is now that the Pulumi deployment engine gets to work. The engine combines the intended model of the infrastructure received from the language host, the current state recorded in the state backend, and the actual resource state to compute which actions need to be executed to bring the actual state in line with the intended model.

Our little example contains dependencies: every s3.BucketObject uses the bucket.id as a way to define in which bucket these objects should be stored. The property id from the s3.Bucket is an Output. Outputs are Pulumi’s way of tracking which property of one resource is required by another, hereby creating a dependency between the resources. The engine uses all these outputs passed from one resource to another as the vertices in a directed acyclic graph (DAG). The engine determines the order of actions based on this graph.

On a first run of pulumi up of our example program, the engine will first create the bucket and wait for the provisioning to be complete, after which the actual bucket ID is passed to the creation of all the bucket objects. Since none of the bucket objects depend on other resources, these bucket objects can all be provisioned concurrently.

On a second run, assuming no modifications to our example program, the Pulumi deployment engine will compare the to-be model with the actual state and conclude that nothing needs to be done.

Let’s crank up the number of bucket objects to 11.

from pulumi_aws import s3

bucket = s3.Bucket('bucket')

for i in range(11): # <- number of objects increased by 1
 s3.BucketObject(
 f'object-{i}',
 s3.BucketObjectArgs(
 bucket=bucket.id,
 key=str(i),
 )
 )

The third Pulumi run will find the bucket and the first 10 objects in the state. The only action taken now is the creation of the 11th bucket object. This brings the actual state back in sync with your intended model.

$ pulumi up

Updating (<masked>/dev)

View Live: https://app.pulumi.com/<masked>/blog_code/dev/updates/2

 Type Name Status
 pulumi:pulumi:Stack blog_code-dev
 + └─ aws:s3:BucketObject object-10 created

Resources:
 + 1 created
 12 unchanged

Duration: 5s

Although you create the intended model of your infrastructure with an imperative language, the engine definitely processes this in a declarative way.

Another Imperative Part: The Providers

These examples hopefully made clear that the Pulumi deployment engine calculates a set of actions to declaratively bring the actual state in sync with your intended model. But the engine is not the component which knows how to talk to all the different APIs of cloud and tool vendors. That’s the role of the provider.

The engine and the provider processes are connected with a gRPC connection. This is similar to the connection between the language host and the engine, with the only difference of which requests are sent. The API that providers expose are of a CRUD nature:

Create: create a new resource
Read: read information of an existing resource
Update: update an existing resource with modified information
Delete: delete an existing resource when no longer needed

The providers receive requests from the engine dependent on the set of actions the engine calculated. A provider doesn’t know anything about the state, the correlation between resource, and so forth.

The nature of the provider API is clearly imperative.

Summary

If the question ever pops up again whether Pulumi is declarative or imperative, the answer is clearly we are both. It is only based on which component of our architecture you are talking about:

Language host: imperative (JS/TS, Go, Python, C#/F#/.NET, Java) and declarative (YAML)
Pulumi engine: declarative
Providers: imperative

To us, it mainly matters if we can solve your infrastructure automation problem. If you still miss something in our offering, we want to hear from you!

How Elkjøp Nordic enables self-service infrastructure for developers

George Huang — Fri, 01 Jul 2022 00:00:00 +0000

At PulumiUP 2022, Tomas Jansson, software architect at Elkjøp Nordic, gave a presentation on how to enable developers to self-service infrastructure by using Pulumi’s Automation API.

Elkjøp Nordic is the leading consumer electronics retailer in the Nordics. The company sells consumer electronics, mobile phones, computers, white goods, domestic appliances, and services linked to these products both directly to consumers and to businesses. It is an omnichannel retailer and serves customers both online and through more than 400 stores. Elkjøp has retail outlets in Norway, Sweden, Finland and Denmark, and franchise operations in Greenland and Faroe Islands.

To support the company’s modernization strategy, Tomas developed a self-service infrastructure platform that would enable any developer to easily provision cloud infrastructure that uses approved resources and security practices. The goal was to empower teams at Elkjøp to increase their development velocity by using the cloud, while maintaining guardrails that uphold security and compliance.

Watch the full presentation below or read on to learn the highlights of his presentation:

The Problem: Balancing Agility and Governance

To achieve his goals, Tomas knew he had to solve the problem of balancing freedom with control. On the one hand, he wanted to foster the creativity of his developers and encourage innovation. He wanted his team to feel they were trusted and had the room to create the best solutions they could. On the other hand, he knew he had to maintain some control. Elkjøp needed to keep its applications secure. It also needed good governance so that teams can keep track of who is making changes. Tomas also wanted to limit the impact of deploying untested code. Code that was still being developed needed to be confined to the developer’s own environment. Finding the right balance between independence and control was paramount. Tomas believed this would result in great software and in developers who would enjoy their work.

The Solution: A Self-Service Infrastructure Platform

Tomas’ solution was to build a self-service infrastructure platform. Self-service infrastructure makes it possible for developers to easily access the infrastructure they need while limiting access to resources they don’t need. In effect, self-service infrastructure is a walled garden. Within the garden, developers have all the freedom they need. However, nothing they do affects what’s outside the wall. Code that causes unintended changes only affects them and no one else.

Tomas built an application that provides a “walled garden” of self-service infrastructure. Developers can create the environments they need, including an Azure resource group and a Kubernetes namespace, all through a graphical user interface. Developers can define whether the environment is for development, test or production. They can also assign values, such as which team members should have admin privileges to the associated GitHub repositories or who can contribute. From there, the platform handles provisioning and configuring all of these resources.

Here is an example that shows how a developer can create a repository simply by selecting a few parameters from a drop-down list.

Tomas determined that the best way to implement self-service infrastructure was with Pulumi and its Automation API. By using Pulumi, he could use standard programming languages to define infrastructure while using familiar software tools like IDEs and test frameworks. Pulumi also natively supports Azure and Kubernetes with 100% coverage of each provider’s APIs and same-day access to new features and updates. Finally, the Pulumi Automation API would enable him to program infrastructure automation directly into his application.

The Pulumi Automation API is a programmatic interface for running Pulumi programs without the Pulumi CLI. Automation API encapsulates the functionality of the CLI (pulumi up, pulumi preview, pulumi destroy, pulumi stack init, etc.) but with more flexibility since infrastructure operations can be controlled programmatically.

How the self-service infrastructure platform was built

Here is the workflow for the Elkjøp self-service infrastructure platform.

The self-service platform is a web application that runs the Pulumi Automation API on the server side. The application uses Pulumi to provision and configure every environment resource such as Azure resource groups, Kubernetes namespaces, service principals, and Kubernetes service accounts. Each environment comes with a Git repository that’s used by developers to manage their deployments. The Git repositories are the locus of each environment because they store everything needed to deploy changes such as infrastructure code and environment credentials.

Using the Pulumi Automation API, the application creates an Azure resource group, a team in both Azure AD and GitHub, and a new GitHub repository. The application then grants administrative access to the GitHub repo to members of the AD group. In addition to creating a Git repository for each environment, the application uses stack references for each Azure environment to inject the associated credentials into the GitHub Secret Store. These credentials are scoped to each GitHub environment so that developers can only work in their own environment. If a team has multiple environments, each environment has its own credentials so developers can’t, for example, deploy from a test environment to a production environment. Secrets are managed through the Pulumi Service.

Every new repository comes with Pulumi programs that developers can use to deploy their application. GitHub Actions initiate the deployment. When an action triggers, Pulumi reads the credentials that were generated by the self-service application, allowing Pulumi to deploy new Kubernetes or Azure resources to the environment.

Watch the demo

To illustrate how Elkjøp’s self-service platform works, Tomas presented a simplified demo application that’s based on Elkjøp’s production platform. The code for the application is available at https://github.com/mastoj/pulumi-automation-demo.

The demo is a TypeScript application that includes an API that provisions Azure and GitHub resources. Pulumi Automation API code is used to deploy new stacks by calling handler code, which is a Pulumi program that defines resources and outputs. The handler code can create Azure resource groups or GitHub repositories, and stack references are used so that outputs generated by the Azure resources can be stored in GitHub (e.g., resource group credentials).

Ready to dive into how this works in detail? Watch the demo in the video below starting around 10:20.

Next steps

After watching the demo of a self-service infrastructure platform, you can try it out for yourself by checking out the demo code. You can also learn more about Pulumi Automation API in our documentation.

If you’re new to Pulumi, then follow our getting started guide.

Infrastructure as Code with Java and Pulumi

Mikhail Shilkov — Wed, 04 May 2022 06:59:00 -0700

Infrastructure has become a core part of application development as modern cloud capabilities such as microservices, containers, serverless, and data stores define your application’s architecture. The term “infrastructure” covers all of the cloud resources your application needs to run. Modern architectures require thinking deeply about infrastructure while building your application, instead of treating it as an afterthought. Pulumi’s approach helps developers, infrastructure engineers, and platform teams work together to leverage everything the modern cloud has to offer.

Pulumi has worked with hundreds of companies to get cloud applications into production, and Java has quickly risen to become one of the most frequently requested features by the community. Pulumi is an open source product, and we are grateful to our awesome community members who bootstrapped Pulumi for Java last year and were instrumental in helping us with this public preview. Thank you to Paweł Prażak and his VirtusLab colleagues!

Pulumi supports Java for all of your modern infrastructure as code needs, this means you can build, deploy, and manage your infrastructure on any cloud—including all of AWS, Azure, Google Cloud, Kubernetes, Oracle Cloud, and more—using Java and other JVM languages. With Pulumi, you will have the entire cloud at your fingertips without ever having to leave your code editor, while using production-ready infrastructure as code techniques.

What is Pulumi?

Pulumi lets you build, deploy, and manage infrastructure on any cloud using general-purpose programming languages (TypeScript, Python, Go, .NET, Java) and markup languages (YAML, CUE) to express your application’s infrastructure needs, using a powerful technique called “infrastructure as code.” You declare desired infrastructure, and an engine provisions it for you, so that it’s automated, easy to replicate, and robust enough for demanding production requirements. Pulumi takes this approach a step further by leveraging programming languages and software engineering tools to make modern cloud infrastructure patterns, such as containers and serverless programs, easy and first-class citizens.

With Pulumi for Java you can:

Declare infrastructure using programs, classes, and libraries written in Java or other JVM languages (Kotlin, Scala, Clojure, Groovy, etc.).
Automatically create, update, or delete cloud resources using Pulumi’s infrastructure as code engine, removing manual point-and-clicking in web consoles and ad-hoc scripts.
Use your favorite IDEs and tools, including IntelliJ IDEA and Visual Studio Code, taking advantage of features like auto-completion, refactoring, and interactive documentation.
Catch mistakes early on with standard compiler errors, analyzers, and an infrastructure-specific policy engine for enforcing security, compliance, and best practices.
Reuse any existing Java package, or distribute your own, whether that’s for infrastructure best practices, productivity, or just general programming patterns.
Deploy continuously, predictably, and reliably using GitHub Actions, or one of over a dozen CI integrations.
Build scalable cloud applications using cloud native technologies like Kubernetes, Docker containers, serverless functions, and PaaS services into your core development experience, bringing them closer to your application code.

Pulumi’s free open source SDK, which includes a CLI and an assortment of libraries, enables these capabilities.

Example: Provision a GKE cluster with a Kubernetes namespace

The following Java snippet demonstrates the power of Pulumi for Java (full source code). The program defines a Google Kubernetes Engine cluster, calculates its kubeconfig and exports it for user’s needs, and deploys a Kubernetes namespace into the newly provisioned cluster.

package gke_sample;

import java.util.*;
import java.io.*;
import java.nio.*;
import com.pulumi.*;
import com.pulumi.gcp.container.*;
import com.pulumi.kubernetes.core_v1.*;

public class Program {
 private static void stack(Context ctx) {
 // Create a GKE cluster
 var cluster = new Cluster("mygke",
 ClusterArgs.builder()
 .initialNodeCount(1)
 .minMasterVersion("1.20.7")
 .build()
 );

 // Build and export a Kubeconfig for the newly created cluster.
 var kubeconfig = Utils.buildKubeconfig(cluster);
 ctx.export("kubeconfig", kubeconfig);

 // Create a Kubernetes provider instance that uses our cluster from above.
 var clusterProvider = new Provider("gke-provider",
 ProviderArgs.builder().kubeconfig(kubeconfig).build());

 // Create a Kubernetes Namespace
 var ns = new Namespace("test",
 NamespaceArgs.Empty,
 CustomResourceOptions.builder().provider(clusterProvider).build()
 );
 }

 public static void main(String[] args) {
 Pulumi.run(App::stack);
 }
}

Resources are defined declaratively using class constructors and argument builders. Dependencies between resources are managed automatically by the Pulumi engine based on the way you use variables in the program. You are free to use any libraries, helper functions—for instance, to build the kubeconfig string above, classes, if statements, for loops, and all the other tools available to Java developers.

Why is Java great for infrastructure too?

Many of us love using Java to author our applications, so why not use it for infrastructure as code too? By using Java, you get many helpful features for your infrastructure code:

Familiarity: No need to learn DSLs or markup templating languages.
Expressiveness: Use loops, conditionals, pattern matching, async code, and more, to dynamically create infrastructure that meets the target environment’s needs.
Abstraction: Encapsulate common patterns into classes and functions to hide complexity and avoid copy-and-pasting the same boilerplate repeatedly.
Sharing and reuse: Tap into a community of cloud applications and infrastructure experts by sharing and reusing Maven or Gradle packages with your team or the global community.
Productivity: Use your favorite IDE and get statement completion, go to definition, live error checking, refactoring, static analysis, and interactive documentation.
Project organization: Use common code structuring techniques to manage your infrastructure across one or more projects.
Application lifecycle: Use existing ALM systems and techniques to manage and deploy your infrastructure projects, including source control, code review, testing, and continuous integration (CI) and delivery (CD).

Pulumi unlocks access to the entire JVM ecosystem—something that’s easy to take for granted but is missing from other solutions based on DSLs or CLI scripts. This approach also helps developers and operators work better together using a shared foundation. Add all of the above together, and you get things done faster and more reliably.

Join the community and get started

The first preview of Pulumi for Java includes support for the entire breadth of services in AWS, Azure, Google Cloud, and more. Give Pulumi a try, visit the Pulumi for Java docs.

There you will find several instructions on installing and getting started with Pulumi for Java. The following resources provide additional useful information:

Although Pulumi for Java is listed in “preview” status, it supports all of the most essential Pulumi programming model features (and the rest is on its way). Our goal is to gather feedback over the next few weeks, and we will be working hard to improve the Java experience across the board, including more examples and better documentation.

Pulumi is open source on GitHub and you can find the Java plugin at pulumi/pulumi-java. And you’re welcome to join the community in Slack to discuss your scenarios, ideas, and to get any needed assistance from the team and other end users.

We look forward to seeing the new and amazing cloud applications you build with Pulumi for Java!

Pulumi and RedMonk on developer-first infrastructure and why it matters

George Huang — Tue, 26 Apr 2022 00:00:00 +0000

What do assembly languages and the cloud have in common? Are abstractions the future of cloud computing? What does “infrastructure” really mean? And why do these questions matter to the platform engineers, infrastructure engineers, and developers who are building modern cloud applications today?

Joe Duffy (Founder & CEO, Pulumi) and James Governor (Co-founder, RedMonk) recently answered these questions and more in a conversation about developer-first infrastructure. Developer-first infrastructure means empowering developers to build and deploy modern cloud applications and infrastructure through the use of software engineering practices that tame modern cloud complexity.

If you’re interested in learning how the worlds of software engineering and cloud infrastructure are intersecting in addition to principles and best practices for building in the cloud, then watch the 28-minute video below.

Highlights of Pulumi and RedMonk’s discussion on developer-first infrastructure

In the first half of the discussion, Joe and James discuss the complexities of building on the modern cloud as application architectures have evolved from simple three-tier apps to the modern, distributed applications of today. Joe traces the evolution of modern computing in which complexity has been tamed over time through abstractions: from low-level assembly languages to higher-level languages and runtimes that make software more accessible, and finally to operating systems. They then discuss how an analogous transformation has yet to occur in the cloud on a mainstream basis.

“The level of abstraction [that most people are doing] with cloud infrastructure reminds me of assembly languages. You look at writing these YAML templates that over-specify every detail of the underlying architecture. I think [the industry] can do better. I think we can be inspired by what we’ve done in programming languages.” — Joe Duffy

In the second half of the discussion, the two discuss what developer-first infrastructure is, and how companies can use this philosophy to tame modern cloud complexity and accelerate development velocity. Companies that execute this well have a “developer-first multiple” — that is, the most valuable companies in the world today are the ones who’ve thought hard about the developer experience for their own employees or the communities that interact with them.

Joe describes five principles of developer-first infrastructure that make developers’ lives easier when building on the cloud.

Cloud Engineering
Harmonized Applications and Infrastructure
Beyond “Lift and Shift”
Secure and Observable By-Construction
An API for Everything

Finally, Joe dives into the best practices of developer-first infrastructure:

Unified Engineering: Software engineering tools and practices for applications and infrastructure.
APIs: Making everything programmable and composable with APIs.
Everything as Code: Use “as code” techniques like infrastructure as code and policy as code.
Polyglot: Embrace the best language ecosystems for the job at hand.
Modularity: Tackle complexity with real sharing and reuse, not copy & paste.
Continuous Delivery: Deliver and scale applications and infrastructure with automated techniques.
Continuous Verification: Verify and enforce guardrails at all times.
Environment Parity: Share as much as possible between dev, test, and prod environments.
Securely Configured: Loosely couple and securely configure environments.
Distributed Architectures: Go beyond “lift and shift” to building truly cloud-native applications and infrastructure.
Observable: Instrument applications and infrastructure to log high cardinality events, early and often.

Next steps

Developer-first infrastructure means empowering developers to build and deploy modern cloud applications and infrastructure through the use of software engineering practices that tame modern cloud complexity. If you feel inspired after viewing the Pulumi/RedMonk discussion video, then consider applying developer-first principles and practices for your next greenfield project or application using an infrastructure as code tool like Pulumi. You can also watch Joe Duffy’s presentation on developer-first infrastructure from the Cloud Engineering Summit to learn more about developer-first.

Six Things You Might Not Know About the Pulumi Service

Chris Smith — Mon, 24 Jan 2022 00:00:00 +0000

As a reader of this blog, you’ve probably heard of the Pulumi Service, the default state-management backend of the Pulumi CLI. If that’s the case, there’s also a good chance you’ve heard of a number of the Service’s key features, like helping you organize your projects and stacks, collaborate with others with the help of organizations, or handle sensitive data securely with built-in support for encrypted secrets.

What you might not know, though, is that we’re adding new features to the Pulumi Service all the time, and that some of these features can be fairly easy to miss. So in this post, we’ll highlight a handful of the features you might not be aware of, and that we think make it even easier to manage your infrastructure with Pulumi.

Let’s get started!

Feature 1: Stack Tags

The first thing we’ll cover is the Pulumi stack tags feature. Stack tags are key/value pairs that are associated with your Pulumi stack. You can use them for any number of purposes, from keeping things organized to even flagging whether or not it’s okay for a script to reclaim the stack’s resources.

While you can create, edit, and remove stack tags using the Pulumi CLI (see pulumi stack tag), you can also do so in the Pulumi Console:

Once set, you can quickly filter stacks on your organization’s Projects tab as well.

Feature 2: Link Updates to your CI/CD pipeline

Another helpful thing the Pulumi Service can do is link the Pulumi stack update to the CI/CD job or run that was used to perform it, as well as to the specific source commit:

When Pulumi updates a stack, it will store some information about the local machine state, such the current git SHA if the stack resides in a git repository. It is with this data that the Pulumi Service links to relevant services where possible.

If you’re curious to know all of the information Pulumi has for a stack update, you can navigate to the Environment tab for that update:

Feature 3: Custom Update Messages

One of the most useful pieces of additional data stored with each update is the message property. By default, Pulumi will use whatever the latest git commit message is. But did you know you can customize that message from the command line?

By passing the --message (or -m) flag to pulumi up, you can provide a custom update message, which can help explain the context for why you made a particular infrastructure change:

$ pulumi up --message "Release the hounds\!"

Feature 4: Viewing Resource Changes

Another thing you might have missed is that the Pulumi Service supports multiple views for a stack update’s logs.

The standard view mimics the Pulumi CLI’s output. However, you can switch to the Diff or Diagnostic views, too.

The Diff view displays all of the properties of modified resources, along with their previous/updated values. This can be helpful when trying to understand exactly what happened for a stack update.

The Diagnostic view just outputs so-called “diagnostic” messages from the Pulumi update. Most commonly, this is where you will see the output of things like calls to JavaScript’s console.log(...) or Python’s print(...) functions. It’ll also include the output of dependent tools, like the output when building Docker containers.

Feature 5: Stack Resource Visualization

Another cool feature of the Pulumi Service is being able to visualize a stack’s resources and their relationships to one another. This is a really important feature!

One of the advantages of using an infrastructure as code tool like Pulumi is that you aren’t just creating a flat list of resources. Instead, by the virtue of how the code and resource graph is structured, you are building a graph of resources.

This means that Pulumi has a semantic understanding of not only what resources are in your stack but also how they relate to one another.

If you navigate to a stack’s Resources tab, by default you see them listed in a table. You can filter this list by resource type, or by providing a custom filter:

However, in order to see the expanded resource graph, click the Graph View button. This will switch to rendering the resources as an interactive graph.

Notice here that the resources that are part of a component are grouped together. Also, you can double click on those nodes in the graph to expand or collapse them!

Feature 6: Drilling into Resource Properties

But you don’t need to just settle for a “top-level view” of your resources. You can click on any specific resource on the Resources tab, and see all of the properties of that resource.

This allows you to see all of the properties that Pulumi is tracking for that resource. It will also show you any relationships to other resources in the same stack.

The Pulumi Service also links you to the resource’s cloud provider’s console where possible, so you can see even more details.

Is your resource missing a link to the cloud provider console? With new services being launched from the myriad providers Pulumi supports every day, it’s hard to keep up. So if you would like us to add cloud provider links, please file an issue over in the pulumi/console-requests GitHub repository so we can update our database.

There are plenty of other features to check out in the Pulumi Service, and we have exciting improvements on the way! As always, feel free to stop by the Pulumi Community Slack to learn more, ask questions, or share anything cool you’re up to!

Cloud Systems: Static Sites and AWS S3

Kat Cosgrove — Wed, 15 Dec 2021 00:00:00 +0000

Cloud engineering is taking over software development. In a lot of ways, this is great; it allows us to build and deploy more complicated applications with less difficulty, and maintaining those applications becomes less troublesome too. We can release smaller updates more quickly than ever, ensuring that we can stay on top of feature requests and security issues. That said, the rise of cloud engineering has also introduced a lot of complexity in the form of dozens of services even within just one cloud provider. Figuring out where to start can be tough, so let’s take a practical tour! In this series, I’ll walk you through building a personal website and deploying it using modern cloud engineering practices.

The simplest website you can build is a static site. That means no web server, just HTML, CSS, and maybe some front-end JavaScript. Static sites can still be beautiful and interactive though, and for many people, they’re more than enough.

Amazon S3 (Simple Storage Service) is an AWS service that provides cloud storage for files, stored as objects in buckets. You can think of it like an external harddrive: plug it into your computer, and you have more storage space for your files, photos, and movies that you can unplug at will and move to another computer. An S3 bucket behaves very similarly in that you can throw just about anything in it, from data lakes to images to logs, and connect it to an application. This also means that you can use it to host a static website! Let’s look at how.

Prerequisites:

An AWS account
Pulumi account
Pulumi installed and configured for AWS
Python3
NodeJS (optional if you want to preview the site locally)

There are a lot of ways to provision an S3 bucket and set its permissions, but for this I’m going to use Pulumi, an Infrastructure as Code tool that allows you to provision, configure, and deploy a variety of cloud services and tools programmatically, without learning a new language. That way, we can stay in our code, and the syntax will be more familiar for developers. It also saves us the headache of going through the AWS Console. All of the code for this, both the website and the infrastructure code, can be found in the GitHub repository for the series, although I’ll be walking you through everything here too.

Building a Static Site

Create a new directory called s3-tutorial and navigate into it. From there, run pulumi new python -y to create a new Pulumi project using Python. You’ll see Pulumi create a few files for you, but the one we’ll be touching is __main__.py.

Before we move on, we need to build out our website. Create a subdirectory and call it website. In that directory, add index.html, style.css, and normalize.css.

This one is built using only HTML and CSS, plus one background image file. Replace the name and bio with your own, as well as the social links in the header.

index.html

<!DOCTYPE html>
<html>
<head>
 <meta charset="utf-8">
 <title>Hello world!</title>
 <link rel="stylesheet" href="style.css">
 <link rel="stylesheet" href="normalize.css">
</head>
<body>
 <header>
 <div class="logo"><i class="fas fa-cat"></i></div>
 <ul class="social">
 <li><a href="http://github.com/katcosgrove" target="_blank"><i class="fab fa-github-alt"></i></a></li>
 <li><a href="http://twitter.com/Dixie3Flatline" target="_blank"><i class="fab fa-twitter"></i></a></li>
 <li><a href="http://linkedin.com/in/katcosgrove" target="_blank"><i class="fab fa-linkedin-in"></i></a></li>
 </ul>
 </header>
<div class="banner">
 <h1>Your Name</h1>
 <h3>Job Title at Company</h3>
 <p>Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nam blandit tristique arcu, sed commodo ligula. Ut ullamcorper velit non luctus mollis. Sed placerat quam sed tellus aliquam, ac eleifend quam lacinia. Praesent bibendum velit at metus auctor, vitae feugiat quam luctus. Morbi eu imperdiet metus, et tincidunt tellus. Aliquam non ullamcorper justo. Cras ornare nulla vel pellentesque vulputate. Mauris in elit neque. Aliquam tempor aliquam libero, elementum luctus nisl imperdiet sit amet. Praesent urna ante, scelerisque non tellus mollis, laoreet sodales felis.</p>
</div>
</body>
<script src="https://kit.fontawesome.com/b4747495ea.js" crossorigin="anonymous"></script>
</html>

style.css

@import url('https://fonts.googleapis.com/css?family=News+Cycle|Teko&display=swap');

ul {
 list-style-type: none;
}

ul li {
 display: inline-block;
}

a {
 color: white;
 -webkit-transition: color .5s ease-out;
 transition: color .5s ease-out;
 text-decoration: none;
}

a:hover, a:active {
 color: rgb(255, 157, 112);
}

header {
 background-color: rgba(0, 0, 0, .8);
 height: 80px;
 position: absolute;
 top: 0;
 width: 100%;
}

header li {
 color: white;
}

.active a {
 color: rgb(255, 157, 112);
}

.social {
 position: absolute;
 right: 50px;
 top: -5px;
 font-size: 30px;
}

.social li {
 margin: 0 5px 0 5px;
}

.logo {
 position: absolute;
 left: 10px;
 margin-left: 20px;
 font-size: 60px;
 color: white;
}

body {
 background: url('./background.jpg');
 background-size: cover;
 background-repeat: no-repeat;
}

.banner {
 width: 60vw;
 font-family: Teko;
 font-size: 2vw;
 text-align: center;
 margin-top: 15vw;
 margin-left: 20vw;
}

.banner p {
 font-family: News Cycle;
}

Your normalize.css can be copied from this repository, and the expected background.jpg can be downloaded from this repository.

To see what this looks like before we deploy it, we’re going to use a tool called live-server. This isn’t a real, functional webserver, but it’s a useful tool during development for static websites. It will reload automatically when a change is detected, so you don’t have to wait for deploys.

Install it with this:

npm install -g live-server

And run it from the same directory as your index.html file:

live-server

Your browser should automatically open to localhost:8080, but if it doesn’t, navigate there yourself and you’ll see your website.

Configuring an S3 Bucket

It’s time to deploy this thing to S3. From your project’s root, run source venv/bin/activate to make sure we’re in a virtual environment, and run pip3 install pulumi-aws. From now on, we’ll be working in __main__.py

For imports, we need the following:

import json
import mimetypes
import os

from pulumi import export, FileAsset
from pulumi_aws import s3

We’re going to be reading files from the operating system and working with JSON, so we need the first three packages from the standard Python library. We’re also going to be using Pulumi. You will already have the pulumi package installed, but we also need the AWS provider, which we just installed.

web_bucket = s3.Bucket('s3-website-bucket',
 website=s3.BucketWebsiteArgs(
 index_document="index.html",
 ))

Here, we are instantiating an s3 Bucket object and assigning it to the web_bucket variable. S3 buckets are capable of hosting websites, but we have to tell the bucket that’s what it’s going to be doing and what it should expect the index document to be called. Ours is just called index.html.

content_dir = "website"
for file in os.listdir(content_dir):
 filepath = os.path.join(content_dir, file)
 mime_type, _ = mimetypes.guess_type(filepath)
 obj = s3.BucketObject(file,
 bucket=web_bucket.id,
 source=FileAsset(filepath),
 content_type=mime_type)

This codeblock defines our content directory (the website folder containing your site files, from earlier) and reads each of them in, before creating using the Pulumi AWS module to create an S3 bucket object for that file. We are also including a little bit of metadata for each file.

def public_read_policy_for_bucket(bucket_name):
 return json.dumps({
 "Version": "2012-10-17",
 "Statement": [{
 "Effect": "Allow",
 "Principal": "*",
 "Action": [
 "s3:GetObject"
 ],
 "Resource": [
 f"arn:aws:s3:::{bucket_name}/*",
 ]
 }]
 })

This function sets some permissions for our bucket. Since this should be publicly-accessible, we need to allow anyone the ability to read the files.

bucket_name = web_bucket.id
bucket_policy = s3.BucketPolicy("bucket-policy",
 bucket=bucket_name,
 policy=bucket_name.apply(public_read_policy_for_bucket))

Now we assign the ID of our bucket as the bucket’s name, then create a policy for it that includes its name and the policy we defined in the previous step. We’re almost done!

export('bucket_name', web_bucket.id)
export('website_url', web_bucket.website_endpoint)

These last two lines are using Pulumi to export some data for us to interact with. The first is the ID of the bucket so we know which one we’re working with, and the second is the website endpoint created by S3 once the bucket is online, so we know where to go to see our site!

Deploying

All that’s left to do is deploy! From your project’s root directory, define the AWS region you want to work in by running pulumi config set aws:region us-west-2, then run pulumi up!

Pulumi will first give you an overview of the expected changes and ask if you want to perform the update. In my case, it looks like this:

 Type Name Plan
 + pulumi:pulumi:Stack static-site-s3-dev create
 + ├─ aws:s3:Bucket s3-website-bucket create
 + ├─ aws:s3:BucketObject index.html create
 + ├─ aws:s3:BucketObject style.css create
 + ├─ aws:s3:BucketObject normalize.css create
 + ├─ aws:s3:BucketObject background.jpg create
 + └─ aws:s3:BucketPolicy bucket-policy create

Resources:
 + 7 to create

Do you want to perform this update? [Use arrows to move, enter to select, type to filter]
> yes
 no
 details

If everything goes smoothly, you’ll see more feedback from Pulumi showing each resource that was created:

 Type Name Status
 + pulumi:pulumi:Stack static-site-s3-dev created
 + ├─ aws:s3:Bucket s3-website-bucket created
 + ├─ aws:s3:BucketObject index.html created
 + ├─ aws:s3:BucketObject style.css created
 + ├─ aws:s3:BucketObject normalize.css created
 + ├─ aws:s3:BucketObject background.jpg created
 + └─ aws:s3:BucketPolicy bucket-policy created

Outputs:
 bucket_name: "s3-website-bucket-89f0341"
 website_url: "s3-website-bucket-89f0341.s3-website-us-west-2.amazonaws.com"

Resources:
 + 7 created

Duration: 6s

Notice the Outputs section there. Those are the two things we exported at the very end of our __main__.py: the ID of the bucket, and the website URL. Click on the website URL, and you’ll see your personal site, deployed to AWS S3 and publicly accessible by anyone! If you don’t want to keep this up, run pulumi destroy to tear down all of these resources, and you’re done!

We’re not quite done yet, though—this site is a little slow, and it’s not very interactive. Maybe we want some kind of database associated with it, or the convenience of containers, or the scalability of Kubernetes. Stay tuned for the next tutorial in this series, where we’ll keep building on this website to introduce more modern cloud engineering tools until it looks like something you’d happily call production!

Pulumi Recommended Patterns: The Basics

Aurélien Requiem — Thu, 04 Nov 2021 23:20:26 +0000

As a customer engineer, one of the most rewarding aspects of my work is to listen to customers and our diverse community to learn and share how they succeed in their day-to-day projects.

In this 3-article mini-series, we’re going to explore some of the recommended patterns used in the Pulumi community and the benefits of using those patterns. We’ll kick off with all the basic patterns to get you started. Next, as you progress into cloud engineering, we’ll go deeper into evolved patterns. We’ll then conclude this mini-series with cloud engineering practices to augment and accelerate how you approach cloud software.

Let’s dive in!

It all starts with the cloud

Pulumi is an infrastructure-as-code tool, and it is at the center of many other services, platforms, and products. Pulumi can act as a unifying agent for many of these services, enabling users to combine multiple services seamlessly at once to create exciting and innovative platforms. In this way, Pulumi is the ‘hub’ connecting the diverse ‘spokes’ of your solution, allowing you to really get rolling.

Pulumi is predominantly used with cloud providers like Amazon Web Services (AWS), Microsoft Azure, Google Cloud Platform (GCP), and Kubernetes. But many more are available and supported in the Registry. At the time of this writing, Pulumi supports 52 providers.

With such an impressive ecosystem, it’s quite normal to follow your vendors’ best practices as they are key ingredients to your long-term success on these platforms.

Here are a few links to get you started:

While these recommendations are important, you don’t have to wait to adopt them all before you start using Pulumi. Rather, integrating Pulumi early in your cloud journey will empower you to implement better cloud practices with more integrated security and improved development velocity.

First coding steps with Pulumi

When you start your journey into cloud engineering and infrastructure-as-code, you have to decide which programming language you will use. To have a smooth start with Pulumi, pick the programming language you are the most comfortable and familiar with, whether it be TypeScript, Javascript, Python, .NET, or Golang.

Of course, while the programming language used to describe your infrastructure desired state is important, so is your integrated development environment (IDE). You’re going to spend most of your time in that application, so choose the one you are the most efficient with.

According to the Stack Overflow survey in 2021, the top 5 IDEs were:

Visual Studio Code (Free, Open-source)
Visual Studio (Free and paid)
Notepad++ (Free, open-source)
IntelliJ (Free and paid)
Vim (Free, open-source)

If you’re unsure which IDE you should be using, here is a list of features you want to consider:

Code completion for your preferred language
Code syntax check and linting
Integrated code documentation support
Integration with your source code control workflows (git, Mercurial, SVN…)
Plugins and extension support to augment the IDE capabilities (issues and pull request management, prettier code, themes and colors, pair programming…)

And just as with the cloud vendors, you should definitely consider the do’s, don’ts, and recommended patterns for the programming language you selected. Following these best practices gives you safe guardrails and streamlines your developer experience across teams sharing the same programming language. Here’s the main ones for a few languages:

TypeScript
Python: PEP 8, PEP 20, and an additional blog post
C#
Golang: Effective Go and a talk

Pulumi recommended patterns

Now we’re all set, so let’s dive into writing a Pulumi application.

Give your resources a name

The first thing you’ll notice when you write a Pulumi app is that every single resource must have its own name. This Pulumi resource name, or a logical resource name if you prefer, is used inside your stack state file to uniquely identify your resources. This name is also what you will find at first when exploring your stack resources in the Pulumi web console.

Depending on the resource type, a second name (a cloud resource name or a physical resource name if you like) may also be provided. This name will be what is used in your cloud vendor’s web console. If this cloud resource name is unspecified, Pulumi uses the Pulumi resource name and appends a random suffix to it to make it unique.

As a recommendation, you should use explicit names for all your Pulumi resource names and then let Pulumi determine what the cloud resource names should be. Let’s explore the benefits:

As vendors have different requirements, cloud resource names have different lengths and constraints. Pulumi knows this fact and will make sure the name matches the vendor requirements.
The random suffix ensures that two stacks for the same project can be deployed without their resources colliding.
Finally, the random suffix allows Pulumi to do zero-downtime resource updates. Due to the way some cloud providers work, certain updates require replacing resources rather than updating them in place. By default, Pulumi creates replacements first, then updates the existing references to them, and finally deletes the old resources.

In this example, the resource inputBucket only has a Pulumi resource name set whereas outputBucket has both the Pulumi resource name and the cloud resource name set. As a result of the lack of a cloud resource name, inputBucketName uses the Pulumi resource name as the base name with a random suffix. As for outputBucketName, the cloud resource name is set as specified by our code.

Along with this recommendation, using a unique prefix in the name of your resources is an elegant solution to identify resources across multiple projects and stacks quickly. A way to achieve this goal is to use a customer name or a project name. As suggested by my colleague Mikhail, the resource name prefix may be composed of multiple values, such as ${organization}-${tenant}-${environment}-${resourceName}.

Here are a few links to understand physical names and auto-naming a bit better:

Handling secrets securely

Deploying any infrastructure or services almost always means dealing with secrets of some form. Most often we encounter a database password, an API key, or other sensitive information.

If you’ve worked with other popular tools, you often have to resort to different means to securely your handle secrets. If you want to embed your secrets as part of your deployment, you may have to store your secrets in plain text in the deployment configuration files; even if you can avoid this, encrypted secrets may still leak as plain text values in the deployment state files.

Pulumi goes above and beyond to ensure your secrets are secret at all times. First, the Pulumi Service always transmits and stores entire state files securely. Pulumi also supports encrypting specific values as “secrets” for extra protection. By default, the encryption method uses automatic, per-stack data encryption keys (DEK) provided by the Pulumi Service. This ensures your secrets are secure both in-transit and at rest, including in your stack state files. Last, Pulumi makes it easy for developers to make any variables a secret, and it’s as simple as flipping the light switch.

Let’s explore how secrets work in Pulumi.

Secrets in the stack configuration file

First, using the Pulumi CLI, you can specific static values in your stack configuration file as shown. When using the option --secret, your secret is automatically encrypted before being stored in your stack configuration file, and the encryption key is safely stored in your Pulumi account. The encrypted value belongs to that stack only.

The benefit to this is you can safely store secret values and even commit them to your source code repository without risk.

In this example, we see the value stored in the stack configuration file Pulumi.dev.yaml. The result is a base64-encoded version of the encrypted value, which can only be decrypted in this stack.

Secrets, outputs, and stack state files

It’s great to have a way to protect our secrets in the Pulumi stack configuration file by using string encryption. But what about revealing a secret by mistake in the stack state file, or by printing the value directly to your terminal?

This example shows you how Pulumi protects your secrets so you never reveal their values.

In the code, we make 3 attempts:

Line 9: First, we try to reveal the value using the native NodeJS function console.log().
Line 10: Second, we try to reveal the value using the Pulumi native function pulumi.log.info().
Line 14: Finally, we try to leak the value in the stack state file.

As the animation demonstrates, our first two attempts did not reveal the value of api_key. Pulumi has protection mechanisms in place to prevent this from happening.

Similar to the prior example, exporting the stack state file to extract the secret value doesn’t work either. The only information we see is the ciphertext, which is not usable.

Secrets and generated passwords

It’s often acceptable to provide your passwords, API keys, or other sensitive information to your stack via the stack configuration file. But in other situations, it’s even better to directly generate a random password as part of your Pulumi app. However, the challenge is to ensure the generated password doesn’t change value when updating your stack.

In this code, we generate a password using the API RandomPassword. As you would expect because we’re using Pulumi’s API, Pulumi already knows that the property dbPassword.result is a secret and encrypts it by default.

From plain text to secret

Software systems often rely upon one another, and it’s very common to obtain sensitive data from one system to be processed in another. With infrastructure-as-code and the commoditization of software to describe your infrastructure state, it’s never been easier to connect your corporate systems with your infrastructure-as-code. However, it’s important to protect this data.

With Pulumi, developers are first-class citizens as we expose useful functions like pulumi.secret(). As you’ve probably guessed already, pulumi.secret() has the ability to transform and protect any arbitrary data.

The code above shows how a plain text string can be converted into a secret in a single line of code. And the resulting variable protectedSecret has the same properties as any other secrets in your Pulumi application.

As demonstrated, your sensitive information is encrypted, and the value doesn’t leak in your stack state file.

Using Pulumi and its secret management capabilities is an important step toward more secure deployments in general. But also, it enables developers to reach “zero-knowledge” of any sensitive information, effectively separating code and deployment from passwords and other sensitive information.

Put these patterns to work

Name your resources clearly, keep secrets encrypted, and build infrastructure in a language you already know with Pulumi.

Get started

Summary

For this first article, we laid down the important foundations of infrastructure-as-code. Here is a quick summary:

Follow your vendors’ best practices as early as possible.
Choose the programming language and the IDE you’re the most comfortable with.
Give your Pulumi resources a name in Pulumi and in your cloud environment.
Leverage Pulumi secrets to protect your sensitive information during deployment.

In the next article, we’ll discuss code reusability and code documentation to accelerate your developer experience. We show you how to have a multi-stack environment, protect critical resources, and more.

Acknowledgement

Thank you to Stephanie Swenson at Furō for reviewing and suggesting many great edits in this article.

Thank you to Laura Santamaria, David Flanagan, Joe Duffy, and Luke Hoban at Pulumi for sharing their knowledge, reviewing, and suggesting edits in this article.

Introducing Pulumi Registry: Deploy Cloud Infra Faster

Alex Mullans — Mon, 18 Oct 2021 00:00:00 -0700

Pulumi offers the most complete infrastructure as code platform for building, deploying, and managing modern cloud infrastructure and applications. When you use Pulumi, a vast library of cloud resources—from compute, storage, and other cloud infrastructure to databases to identity providers to monitoring systems—is at your fingertips. As the Pulumi community has grown, we’ve heard from many of you that we needed a central hub where you could find all of those resources. We’ve also heard from Pulumi partners that they wanted a great place to showcase their integrations with Pulumi so that their customers can more easily learn how to use Pulumi to deploy and manage their products.

Today, we’re excited to launch Pulumi Registry, the one place to discover and share everything you can achieve using Pulumi. Pulumi Registry is a searchable collection of Pulumi Packages published by Pulumi and our partners. With Pulumi Registry, you can easily find the package with the resources you need, install that package directly into your project, and start building. You can choose from Providers that give you full access to everything a cloud provider has to offer, or choose a Component that gets you started quickly with best practices and sensible defaults baked in. All Pulumi Packages are available in all Pulumi languages, so you can build your infrastructure using C#, Go, Python, and TypeScript/JavaScript. You’ll also find all of the documentation you need to succeed: from detailed API reference to how-to guides with source code for specific use cases.

Pulumi Registry is launching with support for 64 of the leading cloud providers and modern cloud SaaS offerings, including Amazon Web Services, Microsoft Azure, Google Cloud, Kubernetes, Auth0, CloudFlare, Confluent Cloud, Datadog, DigitalOcean, Docker, GitHub, Kong, MinIO, MongoDB Atlas, PagerDuty, Snowflake, Spot by NetApp, and SumoLogic. Pulumi Registry also features some of the most popular Helm charts wrapped as Pulumi Components so you can take advantage of the most popular cloud native technologies.

We’re excited to enable any developer to take advantage of the power of these offerings, and we can’t wait to see what you achieve with them and Pulumi.

Find & Deploy Cloud Infrastructure with Pulumi Registry

The modern cloud has put more power and more capability to delight customers in developers’ hands than ever before. In Azure, Amazon Web Services, and Google Cloud alone, there are more than 6,300 distinct services, each of which might be the service that gives your team, or your company, the competitive advantage you need to win in the marketplace. But, with all of this power comes cloud complexity: you need to pick the right service, configure it correctly and securely, and integrate it with all of your existing cloud infrastructure.

Pulumi Registry, an integral part of the Pulumi Cloud Engineering Platform, tames this complexity by bringing together everything you can do with the cloud into a single location. You can see all of our Pulumi Packages, including our Native Providers for the largest and most popular clouds; our broad range of Providers for various cloud, database, networking, monitoring, and other offerings; and all the Components that get you to production faster than ever before. You can easily filter packages by these categories or to find packages for your modern cloud use case, including Serverless, Containers, or Kubernetes.

Once you find a package of interest, everything you need to know to build, deploy, and manage cloud infrastructure with that package is at your fingertips. If you’re using a package for the first time or using a service you haven’t used before, Pulumi Registry has tutorials and scenario-specific how-to guides to help you get started. If you want to jump right in, you can install the package–whether you need the NuGet package, the npm package, the Python package, or the Go module–right from Pulumi Registry. If you’re already familiar with the package and need the API reference, it’s only a click away. We’ve also taken this opportunity to make some improvements to the API documentation (which was previously hosted on pulumi.com/docs), like adding a left navigation and type-ahead filter so it’s easier to quickly find the resource or method you’re looking for.

👉 Visit the Pulumi Registry

Pulumi Providers: The Most Complete Cloud IaC Solution

Pulumi Registry’s packages come in two categories: Providers and Components. Providers are direct representations of all of the resources available from your choice of cloud or SaaS provider: everything from Alicloud to Cloudflare to Okta to Yandex. They’re a great choice when you want to work directly with individual cloud resources and define your infrastructure from scratch. (For higher-level building blocks, skip to the next section.)

For the largest and most popular clouds, we’ve created a special type of Provider that’s unique to Pulumi: Native Providers. Native Providers are the most advanced infrastructure-as-code packages available in the market. Each Native Provider is built directly from the underlying API of the cloud provider, so you get complete access to the full resource model of all of the provider’s services: every resource, every property, every method. We update Native Providers within 24 hours of the underlying API being updated, so you never have to wait to try out the latest improvements and new services.

Just a few weeks ago, we launched the AWS Cloud Control Provider, which is built directly from the new AWS Cloud Control APIs and gives you same-day access to every new AWS service. AWS Cloud Control joins our existing Azure Native Provider, Google Native Provider, and Kubernetes Native Provider.

Use Any Terraform Provider with Pulumi Registry

If Pulumi Registry doesn’t yet have a native provider for your preferred cloud service, that’s OK! With Pulumi, you have full access to the Terraform ecosystem and can use any Terraform provider via our Terraform bridge. A number of bridged Terraform providers are already available in Pulumi Registry, including providers for popular services like Auth0, CloudFlare, Datadog, DigitalOcean, GitHub, and PagerDuty.

Create & Publish Custom Pulumi Providers

We also invite the entire Pulumi community to contribute new Providers to Pulumi Registry. You can enable your customers to manage your cloud service or cloud resources using Pulumi’s modern infrastructure-as-code by authoring a Native Provider, or by bridging an existing Terraform provider into the Pulumi ecosystem, and publishing it to the Pulumi Registry. We’ve also seen passionate customers and community members author Native Providers for cloud services with public, well-known APIs! Pulumi Registry will enable fully self-service publishing in 2022, but until then, we’re happy to work with you to publish your Provider.

👉 Build and publish a Pulumi Provider

Pulumi Components: Prebuilt IaC for Faster Cloud Deployments

When you need full control over the cloud resources you’re building, Providers are a great choice, because they give you access to all of the resources, all of the settings, and everything else you need to build and deploy infrastructure from scratch. But, we frequently hear from developers and infrastructure engineers that they need higher-level building blocks that “just work”: things like secure and production-ready Kubernetes clusters or complete serverless configurations for popular combos like AWS Lambda and API Gateway or Azure Functions and API Management. Pulumi Components are the answer to those requests.

Components are higher-level building blocks that come with best practices and sensible defaults built in. They frequently encode the opinions of the component author so that you don’t have to configure every setting and understand every detail of all the cloud resources you want to use. Let’s look at some examples to understand the power of Components.

Deploy a Production-Ready EKS Cluster with Pulumi (in 15 Lines)

Amazon’s Elastic Kubernetes Service (EKS) is a great example of a powerful but complex service. With Pulumi’s AWS Cloud Control Provider, you can define a production-ready EKS cluster, but you need to know a fair amount about EKS. If you’re a platform team building out a custom Kubernetes shared services platform for a big organization, that’s OK, as you’ll likely want (and expect) full control over every setting on your cluster. For many other scenarios, you just want a well-configured EKS cluster with minimal configuration. Pulumi’s EKS Component gives you just that, with just 15 lines of code:

import * as awsx from "@pulumi/awsx";
import * as eks from "@pulumi/eks";

// Create a VPC for our cluster.
const vpc = new awsx.ec2.Vpc("my-vpc", {});

// Create an EKS cluster inside of the VPC.
const cluster = new eks.Cluster("my-cluster", {
 vpcId: vpc.id,
 publicSubnetIds: vpc.publicSubnetIds,
 privateSubnetIds: vpc.privateSubnetIds,
 nodeAssociatePublicIpAddress: false,
});

// Export the cluster's kubeconfig.
export const kubeconfig = cluster.kubeconfig;

import pulumi
import pulumi_eks as eks

# Create or lookup a VPC for our cluster.
vpc = ...

# Create an EKS cluster inside of our VPC.
cluster = eks.Cluster('my-cluster',
 eks.ClusterArgs(
 vpc_id = vpc.id,
 public_subnet_ids = vpc.public_subnet_ids,
 private_subnet_ids = vpc.private_subnet_ids,
 node_associate_public_ip_address = False,
 )
)

# Export the cluster's kubeconfig.
pulumi.export('kubeconfig', cluster.kubeconfig)

package main

import (
 "github.com/pulumi/pulumi-eks/sdk/go/eks"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // Create or lookup a VPC for our cluster.
 vpc := ...

 // Create an EKS cluster with the default configuration.
 cluster, err := eks.NewCluster(ctx, "my-cluster", &eks.ClusterArgs{
 VpcId: vpc.ID,
 PublicSubnetIds: vpc.PublicSubnetIds,
 PrivateSubnetIds: vpc.PrivateSubnetIds,
 NodeAssociatePublicIpAddress: pulumi.Bool(false),
 })
 if err != nil {
 return err
 }

 // Export the cluster's kubeconfig.
 ctx.Export("kubeconfig", cluster.Kubeconfig)
 return nil
 })
}

using Pulumi;
using Pulumi.Eks;

class MyStack : Stack
{
 public MyStack()
 {
 // Create or lookup a VPC for our cluster.
 var vpc = ...;

 // Create an EKS cluster with the default configuration.
 var cluster = new Cluster("my-cluster", new ClusterArgs {
 VpcId = vpc.Id,
 PublicSubnetIds = vpc.PublicSubnetIds,
 PrivateSubnetIds = vpc.PrivateSubnetIds,
 NodeAssociatePublicIpAddress = false,
 });

 // Export the cluster's kubeconfig.
 this.Kubeconfig = cluster.Kubeconfig;
 }

 [Output]
 public Output<object> Kubeconfig { get; set; }
}

Build a Serverless App with AWS Lambda & API Gateway in 21 Lines

For event-driven and stateless workloads, serverless compute can be a great choice that enables you to pay only for the compute time you need instead of keeping VMs or containers idle. But, setting up serverless functions and the networking required for them to work together isn’t always straightforward. Pulumi’s AWS API Gateway package makes it easy to create a serverless function (or several!) and make them accessible via an API Gateway:

import * as apigateway from "@pulumi/aws-api-gateway";
import * as aws from "@pulumi/aws";

const f = new aws.lambda.CallbackFunction("f", {
 callback: async (ev, ctx) => {
 console.log(JSON.stringify(ev));
 return {
 statusCode: 200,
 body: "goodbye",
 };
 },
});

const api = new apigateway.RestAPI("api", {
 routes: [{
 path: "/",
 method: apigateway.Method.GET,
 eventHandler: f,
 }],
});

export const url = api.url;

import pulumi
import pulumi_aws_api_gateway as apigateway

api = apigateway.RestAPI('api', routes=[
 apigateway.RouteArgs(path="/", method=apigateway.Method.GET, event_handler=f),
 apigateway.RouteArgs(path="/www", method=apigateway.Method.GET, local_path="www", index=False),
 apigateway.RouteArgs(path="/integration",
 target=apigateway.TargetArgs(uri="https://www.google.com", type=apigateway.IntegrationType.Http_proxy))
])

pulumi.export('url', api.url)

package main

import (
 "github.com/pulumi/pulumi-aws-api-gateway/sdk/go/apigateway"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // Create cluster with default settings
 api, err := apigateway.NewRestAPI(ctx, "api", &apigateway.RestAPIArgs{
 Routes: []apigateway.RouteArgs{
 &apigateway.RouteArgs{
 Path: pulumi.String("/"),
 Method: apigateway.MethodGET,
 EventHandler: f,
 },
 &apigateway.RouteArgs{
 Path: pulumi.String("/"),
 Method: apigateway.MethodGET,
 LocalPath: pulumi.String("www"),
 Index: pulumi.Bool(false),
 },
 &apigateway.RouteArgs{
 Path: pulumi.String("/integration"),
 Method: apigateway.MethodGET,
 Target: &apigateway.TargetArgs{
 Uri: pulumi.String("https://www.google.com"),
 Type: apigateway.IntegrationType.Http_proxy,
 },
 },
 }
 })
 if err != nil {
 return err
 }

 ctx.Export("url", api.Url)
 return nil
 })
}

using Pulumi;
using ApiGateway = Pulumi.AwsApiGateway;

class MyStack : Stack
{

 [Output("url")] Output<string> Url { get; set; }

 public MyStack()
 {
 var api = new ApiGateway.RestAPI("api", new ApiGateway.RestAPIArgs
 {
 Routes =
 {
 new ApiGateway.RouteArgs{ Path = "/", Method = ApiGateway.Method.GET, EventHandler = f },
 new ApiGateway.RouteArgs{
 Path = "/www",
 Method = ApiGateway.Method.GET,
 LocalPath = "www",
 Index = false
 },
 new ApiGateway.RouteArgs{
 Path = "/integration",
 Target = new ApiGateway.TargetArgs{
 Uri = "https://www.google.com",
 Type = ApiGateway.IntegrationType.Http_proxy,
 }
 },
 },
 });

 this.Url = api.Url;
 }
}

Create & Publish Your Own Pulumi Components

Want to share your own best practices and cloud configurations with the Pulumi community? You can author a Pulumi Component and publish it to Pulumi Registry. We’ll have fully self-service publishing in 2022, but until then, we’re happy to work with you to publish your Component. You can get started using the link below.

👉 Build and publish a Pulumi Component

What’s Next for Pulumi Registry?

Today’s announcements are just the start of the journey for Pulumi Registry. In the coming months, we’ll continue to invest in making it even easier to author and publish a Pulumi Package. In 2022, we’ll also launch self-service publishing on Pulumi Registry, so you can share your packages directly with the Pulumi community. And, soon we’ll be bringing all of the goodness of Registry–the central index, the easy-to-use documentation, and more–to Pulumi Enterprise customers, so you can host Pulumi Packages private to your organization.

✅ Explore the Registry Now: Pulumi Registry

✅ Try Pulumi: Sign Up for Pulumi Cloud

Cloud engineering fuels the next chapter of startup innovation

George Huang — Thu, 07 Oct 2021 00:00:00 +0000

The story of how the cloud fuels startup innovation seems never ending. In the beginning, AWS birthed cloud computing with its first service, SQS, in 2004 and quickly released several additional services (like S3, EC2, and SimpleDB). From this innovation, startups flourished because they were able to build, experiment, and grow faster than before at much lower cost. Airbnb, Netflix, Zynga, and many more were born, and the rest is history.

Since this post was published, Pulumi has added first-class support for HashiCorp Configuration Language (HCL). You can now write Pulumi programs in HCL directly, alongside general-purpose languages like TypeScript, Python, Go, and C#. To see how it works, see Pulumi HCL.

Today, a new generation of startups is flourishing because of the cloud, but this time with modern cloud architectures that are distributed, API-driven, and more resilient and scalable than ever. Today’s startups have to get to market even faster and rapidly innovate in order to delight customers and carve out market share. Most startups understand the benefits of adopting the modern cloud to help them achieve this goal. However, their ability to reap these benefits for competitive advantage depends on how well they can harness the modern cloud.

Within the cloud engineering community, we see several common patterns for harnessing the modern cloud. Some startups have teams of full-stack developers who need to deploy cloud infrastructure and applications safely and at high velocity. Others might have a few infrastructure or platform engineers who need to enable other developers to use cloud infrastructure easily on a self-serve basis. Many of these teams started off using domain-specific languages (DSLs) to manage infrastructure as code and quickly found that these languages were the limiting factor in achieving faster velocity. DSLs are cumbersome to use and don’t support the logic and expressiveness needed to build and manage modern architectures that are more complex in nature. DSLs are also a barrier to entry to most developers.

In contrast, cloud engineering makes cloud infrastructure more accessible and easier to use for everyone with tried and true software engineering practices and tools. It embraces the reality that all software is now cloud software. A core practice is using a commonly used language like TypeScript, Python, Go, or C#. That’s one reason why more and more startups are choosing Pulumi to help them achieve competitive advantage through cloud infrastructure. It also doesn’t hurt that Pulumi was rated the “Top Paying Technology” by developers in 2021.

Cloud engineering is a central character in the next chapter of how the modern cloud fuels startup innovation. Read on for three mini-stories of cloud engineering in action with startups.

Panther Labs

Panther Labs helps modern security teams build world-class detection and response pipelines using code and automation, developer-friendly workflows, and big data primitives. Its Platform Team is responsible for a large, complex serverless architecture on AWS. Because of the limitations of its legacy Infrastructure-as-Code (IaC) tool, the team was unable to manage and scale its cloud infrastructure with the speed and automation that the company needed to support its fast-growing business. After comparing different alternatives, Panther Labs decided to migrate to the Pulumi Cloud Engineering Platform. Pulumi increased the company’s deployment speeds by up to 10x, reduced the size of its infrastructure codebase by >50%, and enabled its developers to adopt cloud engineering best practices to deliver its cloud applications faster and more reliably.

Results

Infrastructure teardowns were twice as fast, fresh deployments were four times as fast, and simple changes were more than ten times as fast.
The infrastructure codebase was reduced by more than 50%.
Developers could use a programming language and existing software development practices that they already knew.
The ability to create reusable components made it easy to replicate and share components that always followed cloud engineering best practices.
Central visibility across hundreds of single-tenant customer deployments, including a history of when changes were made and what changed for each account.

Read the full case study→

WhyLabs

WhyLabs helps organizations run their AI applications with certainty by monitoring them for problems such as data drift and model degradation. Founded by alumni of AWS, the company began building their platform on AWS and used AWS-provided tools to manage their infrastructure. However, the WhyLabs team needed a platform that could handle the complexity of their modern cloud architectures and would set them up for success when they’re ready to adopt multi-cloud deployments. The team also wanted a platform that integrated with the modern development workflows used by WhyLabs’ developers, who have a software engineering-driven culture of tight collaboration and shared responsibility for infrastructure. By migrating to the Pulumi Cloud Engineering Platform, WhyLabs engineers can continuously and reliably ship new features faster than before for improved time-to-market. Pulumi enabled WhyLabs to adopt cloud engineering best practices out-of-the-box, including building infrastructure as code with general-purpose languages like TypeScript, versioning and reviewing infrastructure code through a code review process, and delivering infrastructure through its existing GitLab CI/CD pipeline using Pulumi’s integrations.

Results

Pulumi enabled WhyLabs engineers to continuously and reliably ship new features faster than before for improved time-to-market.
Pulumi enabled WhyLabs to adopt standard programming languages (like TypeScript) to build infrastructure as code. It also allows them to use software engineering best practices like reusability and unit testing.
Pulumi lowered the barrier to entry for developers to start working with infrastructure. A standard programming language means that every developer can quickly become involved in creating and maintaining the WhyLabs infrastructure.
Pulumi enables WhyLabs to shift infrastructure left because testing and security controls are automated with Pulumi and GitLab integration. Peer reviews and accountability are built into the GitLab CI/CD pipeline.
Pulumi’s integration with GitLab enables WhyLabs to deliver infrastructure through CI/CD pipelines just like with application code, which increases iteration frequency and accelerates time to market.
Pulumi’s support for multiple cloud providers will help WhyLabs expand their platform in the future to meet customers’ needs.

Read the full case study→

Fauna

Fauna provides a flexible, developer-friendly, transactional database delivered as a secure and scalable cloud API with native GraphQL. It needed to provide a way for customers to use its databases while meeting data residency requirements, such as complying with GDPR or FIPS. This meant allowing customers to store data in specific geographies while maintaining low latency and high availability. To build this capability, Fauna’s engineers needed a repeatable, scalable, and efficient way to build, deploy, and manage databases on behalf of its customers. Every database is backed by a cluster of nodes across multiple regions and cloud providers. After evaluating CloudFormation and Terraform, Fauna found that these tools’ domain-specific languages and features would be insufficient for the project. Fauna chose Pulumi to build, deploy, and manage its multi-cloud infrastructure because Pulumi enabled cloud engineering best practices that met its requirements. This included the ability to use a familiar and powerful programming language like Python, integration with existing software development workflows and tools, and software engineering automation.

Results

Delivered its new multi-region, multi-cloud database feature by using familiar and existing software tools and practices that were enabled by Pulumi.
Built and managed its complex cloud architecture more efficiently by using familiar languages like Python.
Increased the reliability and quality of infrastructure using existing code review processes and running unit tests.
Controlled access to its infrastructure using Pulumi’s SSO integration with its identity provider service.

Read Fauna’s blog post→

Summary

Cloud engineering is enabling startups to build, deploy, and manage modern cloud infrastructure and applications faster and with more confidence. By applying standard software engineering languages, practices and tools to cloud infrastructure, startups can harness modern cloud architectures like containers or serverless to their full potential. To illustrate this, we shared three case studies of startups practicing cloud engineering. Pulumi’s Cloud Engineering Platform was built to enable any organization to adopt cloud engineering best practices out-of-the-box. If you’re starting your cloud engineering journey, try Pulumi for free today with this Getting Started guide.

Cloud Engineering Summit Build Track

Kat Cosgrove — Thu, 16 Sep 2021 13:47:25 -0500

The Cloud Engineering Summit 2021 is coming up fast, and the speakers are out! To get you ready to attend, let’s take a look at the sessions for the Build track.

The Cloud Engineering Summit’s three tracks are built around three concepts: Build, Manage, and Deploy. I’m Kat Cosgrove, and I was responsible for selecting your speakers for the Build track! For us, that means building cloud applications and infrastructure with Modern Infrastructure as Code using general purpose programming languages. We embrace the fact that modern cloud applications have blurred the lines between the application and the infrastructure, and that success requires at least some level of proficiency in both. Whether you’re full stack or lean more towards one area, all cloud engineers apply a software engineering mindset and practices to building and testing applications and the underlying cloud infrastructure. This includes using standard programming languages, applying software principles such as reusability and abstractions and testing, and leveraging the rich ecosystem of software development tools.

Without further ado, let’s take a look at each of the talks I’ve selected for you!

Starting us off is the inimitable Kris Nóva with Infrastructure as Software. She’s going to be talking about why Turing complete config management is an antipattern, and why we should be putting the Dev back in DevOps. Nóva is an incredible speaker and engineer. Any talk she gives promises to be hot fire, so I wouldn’t miss this one.

Don’t like repeating yourself? Yeah, me neither. Fortunately, we have Sam Cogan to present his talk, Reuse, Don’t Repeat - Creating an Infrastructure as Code Module Library. Personally, I’m a huge fan of reusable modules for any purpose, when they’re built in such a way that it reduces overall toil and makes your codebase easier to understand. This talk should be useful for everyone.

Helping us ask and answer the right questions to make organizational and technological decisions is Joep Piscaer, presenting Kubernetes isn’t the answer, but what was the question again? I love Kubernetes, but it can be complex, and sometimes that complexity means it’s not the right solution for you (yet.) If you struggle with finding the right questions to ask about your tech needs, this talk could help.

I’m kind of nosy. I like knowing as much as possible about what’s going on inside of my applications and infrastructure. It’s hard to present all of that data in a way that’s easy to consume rather than overwhelming, though. Fortunately, we have Liz Fong-Jones to help out with her talk, What is OpenTelemetry? Don’t miss this one if you’re curious about a better way to gain insights into your distributed systems.

You’d think that when a particular tool is widely accepted as the industry standard for accomplishing a task, there’d be a wealth of information out there on how to accomplish most important tasks in an accessible, actionable way. Unfortunately, that’s not really the case. Fortunately, we have Steve Giguere presenting Hands-on: Taking a K8s Deployment from Default to Secure, wherein he’ll take us through securing an NGINX deployment!

Curious about how a big product gets built? Rahul Sharma is with us to talk about How AWS architects APIs for hyper scale. He won’t just be talking technology; we’ll also hear about the cultural changes required internally to launch a product, and how engagement with customers and partners helps shape the course of development.

Last but certainly not least is Heidi Waterhouse presenting The Future Should Be Uneven, a talk about personalization and accessibility in web design. Accessible design benefits us all, whether we actually need the accommodation or not. The sooner we all understand and embrace that, the better, so Heidi will walk us through some common accessibility settings that are really just good design for everyone.

Explore all of the rest of the great sessions, and don’t forget to register for the cloud engineering summit!

Preview of the Deploy Track at Cloud Engineering Summit 2021

Laura Santamaria — Wed, 15 Sep 2021 09:00:00 -0500

Cloud Engineering Summit 2021 is almost here! We’ve got a great line up this year.

Our tracks are built around the three pillars of cloud engineering: Build, Deploy, and Manage. I’m your track chair for the Deploy track, the track focused on automating and managing infrastructure. Deploy is all about unifying systems so everything in a cloud-based system is shipped together from the same automated, auditable process, reducing human error and improving quality across the board. That could mean focusing on automated testing and linting of infrastructure as code, providing shared services platforms for others to use in their pipelines, or exploring how unified and automated infrastructure changes engineering culture in an organization.

In no particular order, let’s go explore the Deploy track lineup.

Trying to get a handle on way too many environments across your pipelines? Adora Nwodo, Software Engineer at Microsoft, will explore bringing standard software engineering practices to solve the problem of maintaining many environments and configurations in a repeatable fashion. Adora has a very accessible way of explaining complex concepts in her blog and videos (find her at AdoraHacks), and I’m really looking forward to getting her perspective.

How can we improve the security of these unified processes? Alyssa Miller, Business Information Security Officer (BISO) at S&P Global Ratings, will discuss how threat modeling can improve CI/CD pipelines and general security. I’m excited to have Alyssa join us to help us understand how security should be part of the Deploy pillar; her Pitmaster’s Guide to Security is really good and I recommend watching it!

When you’re trying to share what you’re doing, you could share your code, but how do you share your infrastructure? Peter McKee, Head of Developer Relations and Advocacy at Docker, will demonstrate how to build into pipelines the ability to share features and changes in applications with others as a continuous preview. Peter’s a fantastic teacher who always has a great hands-on presentation in his back pocket, so I’m really looking forward to finding out what he has in store for us.

Data builds up while you’re building out, deploying, and managing cloud-based systems. How can you manage data while deploying your cloud infrastructure? Andy Dang, Co-Founder and Engineering Lead at WhyLabs, will talk about data integrity, observability, and monitoring with an open-source project called whylogs. Always love having a preview of an open-source project in the program, and I think data’s going to continue to be a big question for cloud engineering, so Andy’s talk was very welcome!

What about ARM-based systems? Apple certainly got into the game with their M1 chip, so it’s a platform you have to consider when doing deployments. Angel Rivera, Senior Developer Advocate at CircleCI, will explore how to make pipelines and workloads ARM compatible and capable of supporting ARM on various products. ARM-based systems are some of my favorites (yay Arduino and Raspberry Pi!), and they’re also really important considerations if you’re in the mobile or embedded space, so I can’t wait to hear Angel’s take.

How about some focus on doing all of this at scale? Jenna Pederson, Developer Advocate at AWS, will explore how bringing all of the best practices from software development to pipelines will help make working with cloud-based resources consistent and secure. I’m so happy Jenna could join us! Her content is wonderfully accessible and really explains technical concepts and complex systems clearly for all levels (which is my jam!). You don’t want to miss her talk!

Finally, how does all of this stuff work in a real life team? Evan Bradley, Software Engineer, and Tom Carrio, Senior Software Engineer, from Dynatrace are going to take us through bringing it all together with a real-life team, discussing the experiences they’ve had over time as they’ve implemented everything. I’m really excited to have folks bring their stories of what it’s all like so we can share pitfalls and solutions to move forward together. I think you’ll enjoy Evan and Tom’s talk, too!

Explore all the sessions, and don’t forget to register for Cloud Engineering Summit. I can’t wait to chat with you virtually at Cloud Engineering Summit 2021!

Preview of the Manage Track at Cloud Engineering Summit 2021

Matty Stratton — Mon, 13 Sep 2021 08:00:00 -0500

The Cloud Engineering Summit 2021 is coming up fast, and the speakers are out! To get you ready to attend, let’s take a look at the sessions for the Manage track.

The Cloud Engineering Summit’s three tracks are built around three concepts: Build, Manage, and Deploy. I’m Matt Stratton, and I’m your charismatic track chair for Manage. For us, that means managing cloud applications and infrastructure with Policy as Code, visibility, and access controls. For example, managing infrastructure with policies that detect configuration drift, enforce best practices, and even prevent compliance violations before deployment. It means building visibility across your cloud infrastructure so that you always understand its current and past states, including detailed audit history. Finally, you ensure the right guardrails and controls are set in place so that distributed teams can securely develop.

Without further ado, let’s take a look at each of the talks I’ve selected for you!

Overwhelmed by permissions and authorization? Me, too. Fortunately, we have Joy Ebertz presenting Authorization: Ensuring Only Ada Can Access Her Files. Joy is going to cover the various authorization models available to us, with use-cases for each. Authorization can be overwhelming if you don’t have a good grasp of the options available to your organization and an understanding of what works when, so I’m excited for this one.

Taking a higher-level view, Quintessence Anx will be presenting DevSecOps and Secure Incident Response. The introduction of security to the DevOps equation means some shifts in the way we think about the software development lifecycle, and Quintessence is uniquely qualified to talk about why that’s true. A look inside PagerDuty’s 14 Step Secure Incident Response process promises to be interesting for anyone involved in either development or operations.

I don’t think we have the ability to predict the future, but J. Paul Reed wonders if we can develop the ability to forecast incidents just as we have learned to have staggeringly accurate weather short-term forecasts. With Forecasting the Future: Creating a Radar for Risk, Paul will share with us how Netflix uses Risk Radar to collect and make sense of risk in their sociotechnical systems.

When we talk about infrastructure as code, there’s more to it than just the code! Fariba Khan and Stephen Van Gordon will take us on a journey with Lifecycle of a Pulumi Program to share how they leverage a custom state backend with SSO, RBAC, and programmatically configurable pipelines powered by CI/CD tooling and the Pulumi Automation API to drive IaC at their organization.

Snowflake’s Charles Xu will be here to talk about Day-two Operation of Multi-cloud Kubernetes and Vault, a very cool overview of Snowflake’s Kubernetes infrastructure and how Pulumi helps, from blue-green updates, to managing Hashicorp Vault, to rotating TLS certificates. This is a pretty complex topic, so I’m very interested to see where it goes!

Gremlin’s Jason Yee comes to us with Trailblazers: exploration, discovery, & navigating failure - a set of topics that I personally find fascinating and essential! I think we are constantly in a state of discovery and understanding of our complex systems, and learning how to explore them - and work through the inevitable failures - helps us build and operate even more reliable systems.

Rounding things out is the always incredible Tim Banks, with Toxic Velocity: Speed Kills. In this talk, Tim is going to cover the various stressors that put pressure on both organizations and individuals, how they affect us, and what it looks like when things start to spiral out of control. I’m excited for this one because while crashes do happen, we’d all like them to happen less often (or not at all!), and being able to identify some of these things before they become a real problem is better for our businesses and our people.

Explore all of the rest of the great sessions, and don’t forget to register for the cloud engineering summit!

Top 5 Things an Azure Developer Needs to Know: Serverless

Sophia Parafina — Fri, 06 Aug 2021 00:00:00 +0000

The previous article was a deep dive into virtual machines. First, we used the Azure Portal to create and deploy a virtual machine; then, we repeated the process using infrastructure as code. We further demonstrated how to automate provisioning as part of cloud engineering’s build and deploy processes.

This article will explore the other end of the cloud infrastructure with serverless, which is an on-demand, fully-managed cloud architecture.

What is serverless, and when to use it?

Serverless is an event-driven architecture that executes code when triggered. Azure implements serverless with Azure Functions and Events. A function typically performs a single task, such as saving a log file. A function can perform multiple tasks, but in general, functions adhere to the principle of “you only had one job.” Functions can be written in any Azure supported language: C #, F#, JavaScript, PowerShell, Python, and Typescript.

An Event calls a function to execute. It can be a webhook, an HTTP request, or a scheduled trigger. Azure provides many function templates with a range of triggers; furthermore, they can be customized to application requirements.

Azure Functions are ideal for specific use cases that include:

when demand varies unpredictably, serverless can scale automatically
when a service is used infrequently, serverless can reduce costs by only charging per use
serverless can eliminate maintenance costs
when you have client-heavy applications, where most of the business logic is in the client
if you have high latency tasks like data processing

Serverless is not ideal if the application runs constantly, needs a quick response (otherwise known as the cold start problem, or is a long-running process that can exceed the timeout period for your execution plan.

With those caveats, let’s jump into creating a serverless function using the Azure Portal to understand how to write a function and how the pieces work together.

Our first serverless function

For this example, we’ll use Visual Studio Code and the Azure Functions extension.

Step 1: With the Azure Functions extension installed, click on the Activity Bar and click on Create New Project.

Select a folder for your project.
Select a language, use Typescript.
Select HTTP trigger.
Name your function, e.g., myHttpTrigger.
Set the Authorization level to Anonymous

When finished, you’ll have a template.

import { AzureFunction, Context, HttpRequest } from "@azure/functions"

const httpTrigger: AzureFunction = async function (context: Context, req: HttpRequest): Promise<void> {
 context.log('HTTP trigger function processed a request.');
 const name = (req.query.name || (req.body && req.body.name));
 const responseMessage = name
 ? "Hello, " + name + ". This HTTP triggered function executed successfully."
 : "This HTTP triggered function executed successfully. Pass a name in the query string or in the request body for a personalized response.";

 context.res = {
 // status: 200, /* Defaults to 200 */
 body: responseMessage
 };

};

export default httpTrigger;

Step 2: (Optional) If you have Azure Functions Core tools installed, you can run the function locally. Press F5 or click on Run > Start debugging.

The service will run locally and provide you with a URL:

Functions:

 myHttpTrigger: [GET,POST] http://localhost:7071/api/myHttpTrigger

Navigate to the URL in a browser or use curl.

$ curl http://localhost:7071/api/myHttpTrigger
This HTTP triggered function executed successfully. Pass a name in the query string or in the request body for a personalized response.

Press Control-C in the terminal to stop the debugger.

Step 3: Publish the function to Azure.

Make sure you are signed into Azure and click on Deploy function app…

Select Create a new Function App in Azure and enter a name, “exampleHttpTrigger”
Select a runtime stack, Node.js 14 LTS
Select a location for new resources, East US

In the terminal pane, click on the Output tab, and you’ll see the URL for the function. You can call the function with curl with the name parameter.

$ curl https://myazurehttptrigger.azurewebsites.net/api/myhttptrigger?name=Pulumipus
Hello, Pulumipus. This HTTP triggered function executed successfully.

Don’t forget to delete the function in the Azure Portal!

Deploying Azure Functions with code

Creating and deploying an Azure Function is a bit more involved than using the Azure Function extension with VS Code. What goes on behind the scenes after you write the function is a process that includes:

Creating a ResourceGroup

// Create a separate resource group for this example.
const resourceGroup = new resources.ResourceGroup("functions-rg");

Creating the storage account and code container that holds the function

// Storage account is required by Function App.
// Also, we will upload the function code to the same storage account.
const storageAccount = new storage.StorageAccount("sa", {
 resourceGroupName: resourceGroup.name,
 sku: {
 name: storage.SkuName.Standard_LRS,
 },
 kind: storage.Kind.StorageV2,
});

// Function code archives will be stored in this container.
const codeContainer = new storage.BlobContainer("zips", {
 resourceGroupName: resourceGroup.name,
 accountName: storageAccount.name,
});

Uploading the function code as a zip file to the storage account. Note that the source parameter points to the function’s directory.

// Upload Azure Function's code as a zip archive to the storage account.
const codeBlob = new storage.Blob("zip", {
 resourceGroupName: resourceGroup.name,
 accountName: storageAccount.name,
 containerName: codeContainer.name,
 source: new pulumi.asset.FileArchive("./javascript"),
});

Setting the Consumption Plan to determine how to bill when the function executes

// Define a Consumption Plan for the Function App.
// You can change the SKU to Premium or App Service Plan if needed.
const plan = new web.AppServicePlan("plan", {
 resourceGroupName: resourceGroup.name,
 sku: {
 name: "Y1",
 tier: "Dynamic",
 },
});

Set the connection string to the storage account and the zip archive’s Shared Access Signature (SAS) to grant rights to use the storage account. We will need these to create the function in Azure.

// Build the connection string and zip archive's SAS URL. They will go to Function App's settings.
const storageConnectionString = getConnectionString(resourceGroup.name, storageAccount.name);
const codeBlobUrl = signedBlobReadUrl(codeBlob, codeContainer, storageAccount, resourceGroup);

Now that we have all the parameters needed to deploy the function: the resource group, the billing plan, the connection string to Azure storage, the Functions version (3), the runtime (Node.js), the runtime Node version (14.0), and the SAS url for the function code, we can build and deploy our Azure Function.

const app = new web.WebApp("fa", {
 resourceGroupName: resourceGroup.name,
 serverFarmId: plan.id,
 kind: "functionapp",
 siteConfig: {
 appSettings: [
 { name: "AzureWebJobsStorage", value: storageConnectionString },
 { name: "FUNCTIONS_EXTENSION_VERSION", value: "~3" },
 { name: "FUNCTIONS_WORKER_RUNTIME", value: "node" },
 { name: "WEBSITE_NODE_DEFAULT_VERSION", value: "~14" },
 { name: "WEBSITE_RUN_FROM_PACKAGE", value: codeBlobUrl },
 ],
 http20Enabled: true,
 nodeVersion: "~14",
 },
});

export const endpoint = pulumi.interpolate`https://${app.defaultHostName}/api/HelloNode?name=Pulumi`;

At the very end of the script, we export the URL to the service for you to trigger.

Why is it so complicated?

Isn’t infrastructure as code supposed to simplify infrastructure deployments? This seems more effort than using the Visual Studio Code extension or even the Azure CLI on the surface. However, it’s important to note that this is boiler-plate code that you can reuse to deploy multiple functions with different runtimes and Consumption Plans. The key to using infrastructure code is that it is repeatable, customizable, and a CI/CD pipeline can deploy changes automatically.

Try out the complete example in either Typescript or C# to get familiar with Azure Functions. You can also learn how Functions are written in all supported languages by checking out the Azure Functions in All Supported Languages example.

In the following article, we’ll do a deep dive into Azure static websites, which is a popular option for Jamstacks.

Cloud Engineering on the Rise

George Huang — Tue, 03 Aug 2021 00:00:00 +0000

One of the most fulfilling aspects of working at Pulumi is learning how customers and the community practice cloud engineering in their teams. It’s exciting to see how they use cloud engineering and Pulumi to implement best practices that enable leveraging the cloud to accelerate innovation and enable better business outcomes.

Pulumi is born from the experiences and needs of teams practicing cloud engineering every day. When we announced the Pulumi Cloud Engineering Platform in April, CEO & Founder Joe Duffy talked about bringing cloud engineering to everyone. Over the past year we have seen significant growth in cloud engineering and usage of Pulumi across companies of all industries and sizes, and spanning a diverse spectrum of teams and engineering disciplines. We’re also seeing growing numbers of job postings with “cloud engineer” in the title or which have Pulumi as a requirement or desired skill set.

Recently, we published several case studies about how teams are applying cloud engineering best practices. Cloud engineers apply standard software engineering practices and tools uniformly across infrastructure management, application development, and security to tame the complexity of delivering and managing modern cloud applications. We’ve published on our website and in this blog some of the key cloud engineering best practices that we see broadly across the community, and encourage you to read further to see three stories of cloud engineering in action.

These three case studies highlight a small sampling of the many types of companies and teams representing cloud engineers. Some are platform teams at large organizations, like Atlassian, who are responsible for empowering the rest of their developer teams. We also see DevOps teams using cloud engineering, such as at Skai and SANS Institute. There are other types of teams, like full-stack engineers or SREs, which we will share in future posts. All of these teams share a software engineering mindset and make a leveraged impact on the business by increasing speed, automation, reliability, and agility through cloud infrastructure innovation.

Atlassian’s Bitbucket DevSpeed team is responsible for improving developer productivity through better workflows and tooling. It used Pulumi to make it easier and faster for its developers to use cloud infrastructure and reduced developers’ time spent on maintenance by 50%. DevSpeed originally needed a way to expand regional support for its deployment pipeline, which deploys instances on AWS for more than one hundred team members around the world. The team also wanted to build a self-service platform that provisions cloud infrastructure for Bitbucket’s developers, without having to learn a complex domain specific language (DSL). They moved from a legacy DSL-based infrastructure as code tool to the Pulumi Cloud Engineering Platform, which let them define and deploy infrastructure in general-purpose languages that Bitbucket developers already used, such as Python. Using a familiar language like Python also made it easy for them to add cross-regional support to their deployment pipeline. They also built a self-service dashboard that allows any developer to deploy and configure private Bitbucket instances for feature development.

Atlassian’s Key Practices

Builds infrastructure as code with Python.
Makes infrastructure code readable and accessible to all developers in the organization by using a common language like Python.
Created a self-service platform that deploys and configures private Bitbucket instances for any Bitbucket developer. Underneath, everything runs through a CI/CD pipeline with Pulumi, Bitbucket Pipelines, and Bamboo.
Manages and deploys updates to all developer instances in the organization using features like the Pulumi Automation API.

Atlassian’s Results

Allowed developers to use general-purpose programming languages like Python and familiar software tools to deliver and manage infrastructure.
Enabled Bitbucket developers to easily provision approved cloud infrastructure using a self-service dashboard built with Pulumi.
Reduced the time Bitbucket developers spent maintaining their instances from 8 hours per week to fewer than four.
Reduced the size and complexity of its infrastructure code compared to its previous infrastructure tool while increasing the code’s clarity.

Read the full case study→

SANS Institute is the global leader in cybersecurity training. Using Pulumi, the organization adopted cloud engineering practices that enabled it to reduce deployment times by 70%. Its DevOps team needed a more consistent and reliable way to build, deploy, and manage the SANS virtual training environments used in labs. It had previously used legacy infrastructure tools which require inflexible DSLs that aren’t suited to modern cloud applications. The limitations of the DSLs meant the team needed to use AWS Lambda functions and Bash scripts to fill in gaps. The team needed a cleaner solution built for the modern cloud that would be more scalable and easier to manage. By adopting Pulumi, SANS increased delivery speed, quality, and consistency using cloud engineering practices such as building infrastructure as code with TypeScript and software tools, and deploying cloud applications with Git and automated delivery pipelines. It plans to use Pulumi to help other SANS business units modernize their practices and tools.

SANS Institute’s Key Practices

Builds infrastructure as code with TypeScript.
Deploys infrastructure through a GitOps and CI/CD process while managing its Git branches using a GitFlow pattern.
Production deployments can be initiated through an API call through a self-service platform that deploys an application instance.
Manages infrastructure configuration through Policy as Code which checks for compliance.

SANS Institute’s Results

Reduced deployment times for game servers by up to 70%.
Simplified their deployment process by eliminating the need to “glue together” several infrastructure as code tools, scripts, and functions. This let them remove a manual ticketing step.
Enabled developers to use familiar programming languages and tools, which also helped make their code cleaner and - more uniform, resulting in streamlined pipelines.
Helped to identify opportunities in other SANS departments where Pulumi can streamline operations.

Read the full case study→

Skai is an independent, global marketing platform for strategy, measurement, and best-of-breed activation across all of the world’s most influential digital channels. Skai possesses a highly technical engineering organization with over 350 software engineers, data experts, and DevOps engineers. Skai’s microservices run on AWS, but its core monolith service was hosted in a data center running private cloud infrastructure. Skai’s DevOps group needed to migrate the service to the cloud because demand increased. Skai’s DevOps team decided to transition from using its legacy infrastructure tool because its domain-specific language (DSL) wouldn’t be suited to the demands of the newly migrated modern cloud architectures. Instead, Skai sought a solution that enabled it to use familiar languages and software engineering best practices to manage its infrastructure. That’s why Skai chose the Pulumi Cloud Engineering Platform to build, deploy, and manage its infrastructure with Python.

Skai’s Key Practices

Builds infrastructure as code in Python.
Leverages reusable infrastructure components.
Multiple teams collaborate on cloud infrastructure, including an architect, application development teams, and a core platform/DevOps team.
Teams use a “loosely-coupled, tightly-aligned” approach to create component classes for different infrastructure elements, with each team working on a different element.
Uses Python code to integrate infrastructure processes with internal metadata and IT systems.

Skai’s Results

Migrated a core monolith service with hundreds of terabytes of data from a data center to AWS.
Refactored the service’s infrastructure from being managed with a legacy infrastructure configuration tool to Pulumi, using Python as the language of choice.
Created modular infrastructure components that enable different teams to collaborate and work on different parts of the infrastructure in parallel.
Streamlined the registration and deregistration process for new application instances by integrating Pulumi with its internal IT systems.

Read the full case study→

Summary

Cloud engineers are a fast-growing community made of tens of thousands of members and thousands of companies. They apply standard software engineering practices and tools uniformly across infrastructure management, application development, and security to tame the complexity of delivering and managing modern cloud applications. To illustrate cloud engineering in action, we shared three case studies of teams using cloud engineering in their organizations. Pulumi’s Cloud Engineering Platform was built to enable any organization to adopt cloud engineering best practices out-of-the-box. If you’re starting your cloud engineering journey, try Pulumi for free today with this Getting Started guide.

What Exactly Is Cloud Engineering?

Matty Stratton — Fri, 23 Jul 2021 12:46:42 -0500

When we think about the idea of “cloud engineering,” we often think about the concept of taking standard software engineering practices and tools, and making them available and consistent across development, infrastructure, and compliance teams.

It sounds a lot like what DevOps was supposed to accomplish, right? Many great practices have come out of software engineering that we can apply to operations and infrastructure. Likewise, practices from operational disciplines are equally applicable to development teams.

In cloud engineering, we look at how all of these practices are available to multiple functions and teams. It’s a compelling concept, and the more that we refactor our thinking around this, the more effective we can be at delivering value to our customers and users.

Cloud engineering has three main components: Build, Deploy, and Manage. These are all different areas, but they have a lot in common. We will dig into each of these areas in detail in upcoming posts in this series, but here are the key concepts around each.

Build

This might sound like it’s just about writing code, but the build area of cloud engineering is about creating services and infrastructure that provide what our customers need. In today’s world, we use cloud resources to build applications, services, and infrastructure. These resources can make up a shared service platform, giving a single, consistent experience across multiple teams.

It’s also advantageous to create reusable infrastructure components so that application and service delivery focuses on differentiators rather than “reinventing the wheel.” Reusable components provide a consistent and standard implementation, using our organizations’ and teams’ existing practices.

Besides expressing our own best practices and common configurations and approaches, when we build our infrastructure using standard programming languages, we can take advantage of the entire ecosystem surrounding them! We can leverage existing IDEs, test frameworks and approaches, and other wonderful tools available for those languages. Using modern architectures also focuses on the value our services and platforms provide, rather than bespoke and custom implementations.

Deploy

It doesn’t count until it’s in production, right? Code and infrastructure don’t give any value until it’s in front of our customers and users. But doing this in a manner that is highly efficient and quality-consistent is key. Deployment processes that take too long or require too many manual steps can block us from getting new features to our customers or resolving service issues.

When we apply software engineering practices to our deployment processes, this can ensure that we ship the same way, every time. It’s become common practice to apply the principles of continuous integration and delivery to our application software, but we can use the same principles with our infrastructure. This means that new and changed infrastructure resources can meet our quality controls and be tracked and understood when we are investigating the dreaded “what changed?” problem.

The value of automating deployment isn’t just about providing the tests; it ensures all the steps we require are performed every time. Regardless of years of experience, every human is capable of missing a step or making an error. We want to have our skilled humans focus on the areas that benefit from human expertise and not waste their time on the things that don’t.

“But what about checklists?” Checklists are great! But they are even better when a human defines them and are run by software. The power of a checklist is in defining steps, and in many ways, automation expresses our checklists in code. Code that can be tested, reviewed, and managed.

But beyond simple automation, having a unified approach to deployment for both application code and infrastructure changes allows us to consider automation as the key part of the application. We think about infrastructure as a critical and essential component rather than something that happens “on the side.”

Manage

Getting our services into production is a key step, but it’s not the end. Our customers constantly use our services and applications, and we need to manage all of the resources in use. As it applies to our applications and services, visibility across all of our infrastructure allows everyone on our team, regardless of their role, to have a common understanding of what’s going on.

You may be familiar with the adage “security is everyone’s job.” What that means in the world of cloud engineering is that we consider security and compliance (whether regulatory policies or organizational policies) to be closely integrated into our work. Treating our policy as code, just as we treat our infrastructure as code, is a powerful idea! When we express these policies as code rather than prose in a document, we can apply these policy checks both before and after we deploy our services and infrastructure. Doing so extends the common “vocabulary” for collaboration across all teams, regardless of where they sit in the org chart.

Another key piece of the manage story is that we need controls in place to allow who can make changes and what they can change! We trust our team members to want to do the right thing, but we also need guardrails and controls to ensure that they can do so. This means that we need visibility into all changes that occur - treating our infrastructure just like we do our source code in git! Additionally, having the capability and intentionality around fine-grained access controls is critical to make all of our team members successful, provide reliability around our services, and provide confidence for our customers.

Summary

When we take the ideas of cloud engineering and apply them to our organizations, we can deliver value to our customers and users. We can do this by applying the practices and tools already available to our teams and applying the same principles to our infrastructure and application code. We get increased collaboration capability within our teams, a higher level of trust and confidence in our services and applications, and a better handle on the complexity of the modern cloud. We will explore some specific practices and tools around each of these areas in future posts and how they can be applied to your organization!

Do you have a great story around your experiences with cloud engineering? We would love to hear it at the upcoming Cloud Engineering Summit! Submit your proposal to the CFP by July 31st!

How to Test Infrastructure with Pulumi

Sophia Parafina — Fri, 23 Jul 2021 00:00:00 +0000

In the Testing Practices for Cloud Engineering article we discussed how to apply software testing methodologies to cloud engineering. We also examined testing regimes starting from the testing pyramid to the trophy and honeycomb models of testing better suited to distributed and cloud architectures. These testing regimes include three types of tests suited for cloud architectures:

unit tests for testing methods and functions within a service
property tests for validating specified service outputs
integration tests to ensure that resources interact as specified

In this article, we’ll do a deep dive into each of these testing methods.

Cloud engineering testing differs from application testing because multiple dependencies between components can have high latency. Let’s examine these dependencies using an example application composed of an application layer, a database layer, and a network layer. We can visualize these dependencies with Pulumi’s resource graph.The graph is constructed from the inputs needed to create a resource.

The resource graph for the application layer shows many dependencies. The application deploys containers in AWS Fargate, which are instantiated with an application service that requires a service definition with many inputs.

Unit tests are ideal for validating inputs and ensuring that cloud resources are created. Let’s look at a simple example with just two resources, an EC2 instance running a web server and a SecurityGroup policy.

We can use mocks to test if the webserver has a name tag. First, we’ll create the mocks and import the infrastructure that uses the mocks.

import unittest
import pulumi

class MyMocks(pulumi.runtime.Mocks):
 def new_resource(self, args: pulumi.runtime.MockResourceArgs):
 outputs = args.inputs
 if args.type == "aws:ec2/instance:Instance":
 outputs = {
 **args.inputs,
 "publicIp": "203.0.113.12",
 "publicDns": "ec2-203-0-113-12.compute-1.amazonaws.com",
 }
 return [args.name + '_id', outputs]
 def call(self, args: pulumi.runtime.MockCallArgs):
 if args.token == "aws:ec2/getAmi:getAmi":
 return {
 "architecture": "x86_64",
 "id": "ami-0eb1f3cdeeb8eed2a",
 }
 return {}

pulumi.runtime.set_mocks(MyMocks())

# Now actually import the code that creates resources, and then test it.
import infra

We can write a test that checks if the web server has tags and specifically a name tag. First, here’s the code that creates the web server instance:

server = ec2.Instance('web-server-www',
 instance_type="t2.micro",
 vpc_security_group_ids=[ group.id ], # reference the group object above
 # Comment out to fail a test:
 tags={'Name': 'webserver'}, # name tag
 # Uncomment to fail a test:
 #user_data=user_data) # start a simple web server
 ami=ami_id)

Using Python’s unittest framework, the test looks like this:

@pulumi.runtime.test
def test_server_tags():
 def check_tags(args):
 urn, tags = args
 assert tags, f'server {urn} must have tags'
 assert 'Name' in tags, 'server {urn} must have a name tag'

 return pulumi.Output.all(infra.server.urn, infra.server.tags).apply(check_tags)

The test has two assertions: one for tags on the server urn and another if the instance has a Name tag. When we call the test, we pass the server urn and tags. Note that we did not instantiate an EC2 instance and used a mock. This is why it can run unit tests frequently with quick results.

Validating cloud resource properties in tests

In addition to unit tests, we want to test the properties of a cloud resource. For example, we want to restrict access to a resource by creating a policy that limits ingress. For this use case, we need a property test to ensure the exact values required by a policy. The resource graph from our simple example shows that a SecurityGroupRule sets ingress to the web server. We can test if port 22 is open, which would enable logging into the server via ssh.

group = ec2.SecurityGroup('web-secgrp', ingress=[
 { "protocol": "tcp", "from_port": 22, "to_port": 22, "cidr_blocks": ["0.0.0.0/0"] },
 { "protocol": "tcp", "from_port": 80, "to_port": 80, "cidr_blocks": ["0.0.0.0/0"] },
])

The code that tests for an open port 22 for any rule that includes from_port and cidr_blocks in the resource definition.

# Test if port 22 for ssh is exposed.
@pulumi.runtime.test
def test_security_group_rules():
 def check_security_group_rules(args):
 urn, ingress = args
 ssh_open = any([rule['from_port'] == 22 and any([block == "0.0.0.0/0" for block in rule['cidr_blocks']]) for rule in ingress])
 assert ssh_open == False, f'security group {urn} exposes port 22 to the Internet (CIDR 0.0.0.0/0)'

 # Return the results of the unit tests.
 return pulumi.Output.all(infra.group.urn, infra.group.ingress).apply(check_security_group_rules)

Because web-secgrp has port 22 open to all IP addresses, we can expect it to fail this test. You can find the complete example for the unit and property tests on Pulumi’s example Github repository.

Integration testing

Integration testing requires deploying resources in either a test environment or an ephemeral environment. In this integration testing example, we’ll deploy a single resource using Pulumi’s Automation API. Spinning up ephemeral environments for integration testing is an excellent use for the Automation API, which provides a programmatic interface for building infrastructure without a CLI or web-based console.

In this simple example, we’ll use Automation API to create an S3 bucket and perform a set of tests that include:

verifying the bucket name and AWS region
upload a test file
delete the file
delete the stack with the bucket

The following code sets up the Pulumi stack and S3 bucket with Automation API. It’s interesting to note that the tests don’t import the Pulumi AWS SDK and uses a combination of Python’s built-in unittest framework and Amazon’s boto3 Python library. This illustrates the extensibility and flexibility of infrastructure as code by letting developers choose the tools and SDKs.

import os
import unittest

import boto3
from pulumi.x import automation as auto

from resource_s3 import BUCKET_NAME, OUTPUT_KEY_BUCKET_NAME, OUTPUT_KEY_REGION


class TestS3(unittest.TestCase):

 @classmethod
 def setUpClass(cls) -> None:
 cls.STACK_NAME = 'staging'
 cls.REGION_NAME = 'eu-north-1'
 cls.WORK_DIR = os.path.join(os.path.dirname(__file__))
 cls.FILE_NAME = 'bucket.txt'

 cls.stack = auto.create_or_select_stack(stack_name=cls.STACK_NAME, work_dir=cls.WORK_DIR)
 cls.stack.set_config("aws:region", auto.ConfigValue(value=cls.REGION_NAME))
 cls.stack.up(on_output=print)
 cls.outputs = cls.stack.outputs()
 cls.s3 = boto3.resource('s3')

The TestS3 class has a second @classmethod decorator for the tests; for example, the following code uploads a file and checks to see if the file is stored in the bucket.

 def test_s3_create_permission(self):
 output_bucket_name = self.outputs.get(OUTPUT_KEY_BUCKET_NAME).value
 created_response = self.s3.Bucket(output_bucket_name).put_object(Key=self.FILE_NAME, Body='Hi')
 self.assertEqual(created_response.key, self.FILE_NAME)

When we run the test, it creates the stack and resource, runs the tests, and tears down the infrastructure. This process can be part of a CI/CD pipeline to run an automated integration test as part of the build process. The complete example for integration testing is available on Github.

Infrastructure testing strategy: final thoughts

Although distributed infrastructure presents some unique challenges because of its asynchronous nature, we can perform efficient and automated testing. Whether it is unit and property testing that uses mocks and stubs or integration testing that requires deployed resources, infrastructure as code lets you use familiar testing frameworks and SDKs. Moreover, Pulumi’s Automation API makes it simple to create stacks and spin up ephemeral infrastructure for integration testing. Learn more about infrastructure testing with Pulumi’s Testing User Guide and get hands-on with our testing examples.

Testing Practices for Cloud Engineering

Sophia Parafina — Tue, 22 Jun 2021 00:00:00 +0000

Cloud engineering brings industry-standard software development practices to building, deploying, and managing cloud infrastructure. Testing is a common practice for evaluating software to ensure that it meets requirements. Similarly, infrastructure testing checks for missing requirements, bugs, and errors; it also ensures security, reliability, and performance. Testing uses manual or automated tools to identify bugs that can cause unexpected infrastructure behavior.

There are many benefits to infrastructure testing, including:

reduced costs to fix bugs when caught early in the development lifecycle,
discovering security risks and problems earlier,
delivering a quality product that creates customer satisfaction through a great user experience

Testing shifts left the risk inherent with distributed architectures composed of many resources. Ultimately, testing increases release velocity, reliability, and confidence in your application.

This article is the first in a two-part series about testing infrastructure. The terminology for testing can be confusing because of broad definitions that overlap. This article will narrow those definitions that originated from application testing and apply them to infrastructure and cloud engineering. Let’s take a look at the different types of testing used with infrastructure as code.

Functional testing vs. non-functional testing

With infrastructure as code we want to test if parts of the system work as required or how parts interact with each other; this is functional testing. We will also want to test how the system behaves concerning performance, security, reliability, usability, and compliance. In general, these tests are called non-functional tests because they test the system’s behavior as a whole rather than by its parts.

In functional testing, unit tests typically test a single function or class. They are low-level tests that can be executed quickly and at low costs. Integration tests are functional tests that validate how components interact with each other, i.e.; it checks the interaction between software components. Non-functional tests are frequently performance tests, which evaluate the behavior of the system.

Mike Kohn proposed the testing pyramid that shows the different layers of testing and the amount of testing for each layer.

In their original form, unit tests comprise most tests, with integration or service tests as the middle layer of the pyramid. At the top of the pyramid are end-to-end or UI tests, which indicate performing non-functional tests sparingly.

However, pushback against this model was proposed in 1999 at the height of popularity for Xtreme Programming (XP). Alternative models include Kent Dodd’s Testing Trophy

and Spotify Lab’s Testing Honeycomb.

As you can see, both models suggest that the majority of tests should be integration tests. These models responded to the change from monolithic architectures prevalent when Dodd proposed the Testing Pyramid to the shift to microservices and distributed architectures that make up modern applications.

For deploying modern applications with cloud engineering, we propose an alternate model called the testing rocket. Static testing is inherent in cloud engineering because of the software toolchains, such as IDEs, that perform static checking. Both unit and integration testing are equally important to ensure that infrastructure is deployed and managed reliably. End-to-end tests for performance and scalability should be performed less frequently because of complexity and costs.

Cloud engineering testing

Cloud engineering applies software testing principles to infrastructure where we focus on functional testing. We use three types of functional tests with infrastructure: unit tests, integration tests, and property tests (sometimes called functional tests). We’ll examine each type of test in detail.

Static tests

Infrastructure templating languages such as YAML or JSON have limited static validation capabilities; primarily, they are limited to linting: validating and formatting the code. Pulumi’s approach of deploying infrastructure with programming languages lets you use built-in tools in IDEs that perform static tests, highlight errors in your code, and offer other features such as code completion, enumerations, and syntax checking. Pulumi’s preview also performs static checking before deploying a resource.

Unit tests

Unit tests for applications test the smallest piece of code, which are often methods and functions of classes or modules. They are commonly written by the programmer and run quickly.

With infrastructure, the smallest unit is often the cloud resource. Because cloud resources haven’t been created by the time you run unit tests, you can’t write a test to evaluate infrastructure behavior. For example, you can’t make HTTP requests to endpoints because there’s no web server to serve them.

Pulumi’s cloud engineering approach enables unit testing by replacing all external dependencies with mock objects that replicate the behavior in a specified way. This enables you to write fast unit tests that run in memory without any out-of-process calls. They provide rapid feedback loops during development, making them suitable for Test-Driven Development (TDD).

As with applications, unit tests for infrastructure in Pulumi are authored with the same language used to declare infrastructure. This means you can use familiar test and mock frameworks such as PyTest for Python or Mocha for Node.js. Using a framework ensures configuration is correct before provisioning and that the resulting infrastructure has the specified properties. In addition, team standards and security guidelines are enforced.

Property tests

Another type of test is property tests, which are sometimes called “functional tests.” Property tests evaluate the business requirements of an application by verifying the output of an action without examining the code that’s executed. Property tests, like integration tests, require components to interact. However, property tests require that the component return a specified value, while integration tests verify that the function works as required. For example, a property test would require a value, such as the count of rows returned, and an integration test would only require that rows are returned.

Property tests in cloud engineering and Pulumi use Policy as Code to set guardrails and enforce compliance for cloud resources. In addition to authoring company-wide policies, It also enables another type of infrastructure testing where a policy becomes a property that a test can evaluate and assert. Property tests run before and after infrastructure provisioning and have access to all input and output values of all cloud resources instantiated in a Pulumi program. Unlike unit testing, property tests can evaluate real values from the cloud provider and not mocked values.

Finally, there are several ways to run property tests against any cloud environment. They can be configured as a persistent stack for acceptance testing, or as an ephemeral cloud environment created by a pull request, or a combination of both.

Integration tests

Integration tests validate whether services or modules in an application work as specified. Unlike unit tests, they use actual dependencies instead of mock objects, and they provide less precise feedback than unit or property tests. Because integration uses actual dependencies, they require that services be complete and functioning. Tests are run in a strict order to ensure that modules or services are instantiated before the test. Developers are less likely to write and run integration tests, leaving it to SRE experts in specialties such as chaos engineering and pipeline automation to write tests run in a CI/CD system.

In cloud engineering, an infrastructure integration test uses infrastructure deployed in an ephemeral environment. As an example, we can use the Pulumi CLI to deploy the ephemeral environment as a stack that builds a dependency graph based on inputs and outputs that ensures the required resources are instantiated and available before the integration test.

Once the resources are deployed, the integration test retrieves the stack outputs, which is often a public IP address or resource name. The test can be as complex as a suite of application-level tests between the various services or components or as simple as a health check that expects an HTTP 200 status return from an endpoint. The primary advantage of integration tests is that they use the same cloud infrastructure used in production to return actual values.

However, there are two areas of concern. First, deploying ephemeral infrastructure takes time to create the resources. The impact is that integration tests are not performed as frequently as unit tests raising the possibility of finding bugs later rather than sooner. Secondly, there is a cost to deploying ephemeral infrastructure, and you can easily overrun your testing budget if performed frequently.

Distributed architectures have become the norm for cloud-based applications. For example, microservices are composed of individual services interacting with each other. Because of the interaction between stand-alone services, we can see why both the trophy model and the honeycomb model of testing prioritize integration testing despite these concerns.

Summary

Testing is an integral part of cloud engineering. With its origins in software engineering, we can use the same methodologies to build reliable infrastructure. We’ve presented three types of tests for infrastructure deployed with programming languages. When we use a programming language for infrastructure deployment, we can take advantage of the software toolchain to perform static tests not possible with templating languages. We can also perform unit tests and property tests without deploying infrastructure by using mocks and policies. Finally, we can build ephemeral infrastructure to perform integration tests at the application level. We’ll look at specific examples for performing unit, property, and integration tests in the follow-up article. Stay tuned.

Refactoring Infrastructure as Code

Sophia Parafina — Tue, 01 Jun 2021 00:00:00 +0000

The central principle of cloud engineering is adopting software engineering practices. Refactoring is a technique for making changes to code that improve maintainability, enhance performance, scalability, and security without changing its external behavior. In devops, refactoring often occurs with modern applications; however, we can apply those same techniques to cloud infrastructure with infrastructure as code.

Refactoring results in many advantages. First and foremost, the code is more readable and easier to understand for other team members –this aids in maintainability and well-organized code, providing a solid foundation for future releases. Overall, if done well, refactoring reduces complexity which makes future changes more efficient.

The topic of refactoring is a broad topic covering many different techniques. Many references, such as Martin Fowler’s Refactoring, cover refactoring in depth. In this article, we examine standard practices used with infrastructure as code.

Infrastructure is different

A major difference between refactoring application and infrastructure code is that resources, such as databases persist. Typically, if you change the name, type, or parent path of a resource, it is deleted and a new resource is created. Other infrastructure as code solutions tie a file or module to a resource’s identity. An advantage of Pulumi is that moving a resource across file boundaries does not recreate the resource because a resource’s identity is based on its object ID, irrespective of its file location.

In cases where an object’s ID changes – due to a rename, reparenting to a new component, or otherwise – Pulumi provides aliases. An Alias accepts a list of identifiers that can include old Alias objects or old resource URNs. When a resource is annotated with an alias, Pulumi can connect the old to the new object state, and the resource can be updated in place.

A similar situation occurs with component resources, i.e., a logical grouping of related resources such as VPC with subnets pre-configured. When you change a parent resource of a component resource, the identity of the component resource also changes. Pulumi provides a parent option to tie a child resource to the parent resource. The combination of alias and parent arguments to resources lets you refactor infrastructure while preserving existing resources.

Let’s examine how to apply refactoring to infrastructure.

Test-Driven Development

Test-driven development (TDD) is a common technique used for code refactoring. TDD breaks down refactoring into three steps:

Red: consider the required behavior and what needs to be coded
Green: get the code to pass a test for that behavior
Refactor: review the code for improvements and implement as needed.

In a previous blog post, Ringo de Smeti used TDD to refactor Cumundi’s code for provisioning repositories by customer. Ringo created a ComponentResource subclass which moved individual resources into a customer resource class.

// Pulumi custom resource representing a customer project
export class Project extends pulumi.ComponentResource {
 constructor(name: string, args: ProjectArgs, opts: pulumi.CustomResourceOptions = {}) {
 super('customer:Project', name, {}, opts);

 var customerId = name.replace(/([a-zA-Z])(?=[A-Z])/g, '$1-').toLowerCase();
 var serviceId = name.replace(/([a-zA-Z])(?=[A-Z])/g, '$1').toLowerCase();

 const gitlabProject = new gitlab.Project(name,
 {
 name: customerId,
 description: pulumi.interpolate`${args.customer} code`,
 namespaceId: args.gitlabNamespace,
 visibilityLevel: "private",
 defaultBranch: "master",
 pipelinesEnabled: true,
 issuesEnabled: false,
 wikiEnabled: false,
 snippetsEnabled: false,
 containerRegistryEnabled: false,
 mergeRequestsEnabled: false,
 mergeMethod: "ff",
 onlyAllowMergeIfPipelineSucceeds: true,
 sharedRunnersEnabled: true
 }
 )

Calling the Project class creates a new customer.

const firstCustomer = new customer.Project("FirstCustomer",
 {
 customer: 'First Customer',
 needsGoogleInfra: false,
 gitlabNamespace: gitlabNamespace
 }
)

const secondCustomerProject = new customer.Project('SecondCustomer',
 {
 customer: 'Second Customer',
 needsGoogleInfra: true,
 gitlabNamespace: gitlabNamespace
 }
)

By using TDD, Ringo reduced code duplication, resulting in a clean and efficient codebase.

Refactoring by Abstraction: Pull-Up/Push-Down

The Pull-Up/Push-Down method is used frequently to reduce duplicate code. It does this by abstracting methods into classes to implement inheritance and extraction. The Pull-Up method uses a superclass to consolidate similar methods, whereas Push-Down moves methods from a superclass into subclasses.

The Pull-Up method in this example deploys the Kubernetes Guest Book. The simple version is a port of the original YAML and the component version, shown below, creates a service class that creates both the service and deployment.

export class ServiceDeployment extends pulumi.ComponentResource {
 public readonly deployment: k8s.apps.v1.Deployment;
 public readonly service: k8s.core.v1.Service;
 public readonly ipAddress?: pulumi.Output<string>;

 constructor(name: string, args: ServiceDeploymentArgs, opts?: pulumi.ComponentResourceOptions) {
 super("k8sjs:service:ServiceDeployment", name, {}, opts);

 const labels = { app: name };
 const container: k8stypes.core.v1.Container = {
 name,
 image: args.image,
 resources: args.resources || { requests: { cpu: "100m", memory: "100Mi" } },
 env: [{ name: "GET_HOSTS_FROM", value: "dns" }],
 ports: args.ports && args.ports.map(p => ({ containerPort: p })),
 };
 this.deployment = new k8s.apps.v1.Deployment(name, {
 spec: {
 selector: { matchLabels: labels },
 replicas: args.replicas || 1,
 template: {
 metadata: { labels: labels },
 spec: { containers: [ container ] },
 },
 },
 }, { parent: this });

 this.service = new k8s.core.v1.Service(name, {
 metadata: {
 name: name,
 labels: this.deployment.metadata.labels,
 },
 spec: {
 ports: args.ports && args.ports.map(p => ({ port: p, targetPort: p })),
 selector: this.deployment.spec.template.metadata.labels,
 // Minikube does not implement services of type `LoadBalancer`; require the user to specify if we're
 // running on minikube, and if so, create only services of type ClusterIP.
 type: args.allocateIpAddress ? (args.isMinikube ? "ClusterIP" : "LoadBalancer") : undefined,
 },
 }, { parent: this });

 if (args.allocateIpAddress) {
 this.ipAddress = args.isMinikube ?
 this.service.spec.clusterIP :
 this.service.status.loadBalancer.ingress[0].ip;
 }
 }
}

With infrastructure as code, you can use your IDE to refactor.

Composing

Composing is another refactoring method for reducing duplicate code. The most common way to use composing is through extraction and inline methods.

Extraction consolidates code by creating a new method and replacing duplicate methods with a call to the new method. For example, when we create services, we often export information about the service, such as IP address or hostname, as shown in the example below.

export const publicIp = server.publicIp;
export const publicHostName = server.publicDns;

We can create a function that exports the information about a service and call it as needed.

function public_server_info(server) {
 export const publicIp = server.publicIp;
export const publicHostName = server.publicDns;
}

public_server_info(server);

Inline refactoring reduces unnecessary methods by finding all calls to the method and replacing them with the content of the method. In this policy as code example, we can refactor two methods into a single method.

export function requireEbsVolumesOnEc2Instances(name: string): ResourceValidationPolicy {
 // TODO: Check if EBS volumes are marked for deletion.
 return {
 name: name,
 description: "EBS volumes should be attached to all EC2 instances",
 enforcementLevel: "mandatory",
 validateResource: validateResourceOfType(aws.ec2.Instance, (instance, args, reportViolation) => {
 if (instance.ebsBlockDevices !== undefined) {
 reportViolation("EC2 Instance should have EBS volumes attached.");
 }
 if (instance.ebsBlockDevices.length === 0) {
 reportViolation("EC2 Instance should have EBS volumes attached.");
 }

 }),
 };
}

The function can be simplified to the following:

export function requireEbsVolumesOnEc2Instances(name: string): ResourceValidationPolicy {
 // TODO: Check if EBS volumes are marked for deletion.
 return {
 name: name,
 description: "EBS volumes should be attached to all EC2 instances",
 enforcementLevel: "mandatory",
 validateResource: validateResourceOfType(aws.ec2.Instance, (instance, args, reportViolation) => {
 if (instance.ebsBlockDevices !== undefined && instance.ebsBlockDevices.length === 0) {
 reportViolation("EC2 Instance should have EBS volumes attached.");
 }
 }),
 };
}

Simplifying Method Calls

Simplifying method calls makes them easier to understand by clarifying the logic for interaction between classes. One way to simplify code is to add, remove, or introduce new parameters with explicit methods and method calls.

Roles in AWS are typically stored as JSON documents. In the example below, we create the role by passing in a string.

lambda_role = iam.Role('lambdaRole',
 assume_role_policy="""{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Action": "sts:AssumeRole",
 "Principal": {
 "Service": "lambda.amazonaws.com"
 },
 "Effect": "Allow",
 "Sid": ""
 }
 ]
 }"""
)

sfn_role = iam.Role('sfnRole',
 assume_role_policy="""{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Effect": "Allow",
 "Principal": {
 "Service": "states.%s.amazonaws.com"
 },
 "Action": "sts:AssumeRole"
 }
 ]
 }""" % config.region
)

You can write a function called create_role and pass in the name and policy as JSON to create many roles.

const lambda_role_document = JSON.parse('{"Version": "2012-10-17","Statement": [{"Action": "sts:AssumeRole","Principal": {"Service": "lambda.amazonaws.com"},"Effect": "Allow","Sid": ""}]}')

function create_role(role :string, policy :JSON) {
 const newrole = new aws.iam.Role(role,{
 assumeRolePolicy: JSON.stringify(policy),
 })
 return role;
}

const lambdaRole = create_role("my_lambdarole",lambda_role_document);

Moving Features

Moving features creates new classes and moves functionality between old and new classes. A utility class may not have some methods you need, but you can’t add those methods to that class. The solution is to create a new class with the methods you need.

In this example, we extend a class and add new methods that configure an EC2 instance post-provisioning.

export class Provisioner<T, U> extends pulumi.dynamic.Resource {
 dep: pulumi.Output<T>;
 result: pulumi.Output<U>;
 constructor(name: string, props: ProvisionerProperties<T, U>, opts?: pulumi.CustomResourceOptions) {
 const provider: pulumi.dynamic.ResourceProvider = {
 diff: async (id: pulumi.ID, olds: State<T, U>, news: State<T, U>) => {
 let replace = false;
 let replacementProperties = [];
 if (JSON.stringify(olds.dep) !== JSON.stringify(news.dep)) {
 replace = true;
 replacementProperties.push("dep");
 }
 // Only trigger replacement due to the `changeToken` property, IFF
 // the changeToken still has a value in the new inputs, and it doesn't
 // match with the old value. If, say, the user decides to no longer specify
 // the changeToken in the new inputs, then we don't trigger a replacement.
 if (news.changeToken && olds.changeToken !== news.changeToken) {
 replace = true;
 replacementProperties.push("changeToken");
 }
 return {
 changes: replace,
 replaces: replace ? replacementProperties : undefined,
 deleteBeforeReplace: true,
 };
 },
 create: async (inputs: State<T, U>) => {
 const result = await props.onCreate(inputs.dep);
 if (result !== undefined) {
 inputs.result = result;
 }
 return { id: uuid.v4(), outs: inputs };
 },
 };
 super(provider, name, { dep: props.dep, changeToken: props.changeToken, result: null }, opts);
 }
}

Preparatory Refactoring

Preparatory refactoring occurs when adding a new feature to a release. It is typically part of a software update instead of a separate refactoring process. Preparatory refactoring reduces future technical debt.

Conclusion

Writing clean and efficient code is part of cloud engineering, and we can achieve this by using well-established refactoring methods used by application developers. Pulumi has many easy-to-understand examples in Node.js, Python, Go, and .Net. Because the examples are written explicitly to make them understandable, some could use refactoring. What code samples can you find in the Pulumi Github repository that could use some refactoring?

Cloud Engineering: The Future Is Now

Sophia Parafina — Thu, 29 Apr 2021 00:00:00 +0000

Thank you for joining the PulumiUP event. We had a stellar set of speakers and panelists discussing the future of DevOps and how Cloud Engineering is providing the tools and processes that enable faster delivery, the right mix of architecture, and foster collaboration among teams in an organization. Here are some of the highlights and takeaways from our speakers.

Putting Cloud Engineering in Practice

Our speakers all commented on how the cloud is speeding up development. Applications are dynamic, and your infrastructure should also be dynamic, according to Adrian Cockcroft, VP of Sustainability Architecture at Amazon. The future of infrastructure is a mixture of lightweight, scalable services combined with event-driven architecture. Furthermore, composing infrastructure should take advantage of well-architected frameworks such as those published by AWS and Liberty Mutual.

Underscoring this, Justin Fitzhugh, VP of Cloud Engineering at Snowflake, cited the need to disaggregate compute and storage to scale independently. Moreover, Snowflake needed to scale across three public clouds with self-managed infrastructure. In Snowflake’s experience, scaling with manual processes was not a viable option. They codified actions by following the software engineering process of code commits, tests, CI processes, and a CD pipeline for execution.

Keith Redmond, VP of SaaS Engineering at Fernergo share a perspective from the financial services industry. He reiterated the need for a deployment framework that bridged dev and ops teams and an automated build and release process to maintain integrity and reliability. He noted that this required a change in culture from teams handing off tasks to each other without input from the other to a completely integrated team where all actions with infrastructure begin with a code commit. The culture shift to an engineering-managed process also changed hiring patterns from operators and system administrators with some coding skills to software engineers interested in infrastructure.

In the current cloud engineering landscape, we will continue to see movement towards heterogeneous architectures, an increase in the use of software engineering methodologies to deploy infrastructure, and more automation for building and deploying in the cloud. This segues into the future of cloud engineering with a focus on tooling, process, and teams.

The Future of Cloud Engineering

The cloud engineering panel included Justin Fitzhugh, Dana Lawson, VP of Engineering at GitHub, Charity Majors, CTO at Honeycomb, and was moderated by Kat Cosgrove, Developer Advocate at JFrog. The session, titled “Your Peers,” discussed the future of cloud engineering.

When planning for the future of modern cloud applications, the panel agreed that the industry would continue to iterate and evolve away from domain specific languages, DSLs, and towards software engineering practices for infrastructure. They also agreed that while Go would be the most popular language for Infrastructure as Code, Ruby and JavaScript will also continue to be key languages.

Cloud engineering personas will change as new technologies such as the Internet of Things, availability of 5G, orchestration, edge computing, and future network APIs roll out. Cloud engineers should continue to focus on containers and what goes into applications. Knowing the entire lifecycle of their application from development to production is a must for cloud engineers. It underlies the concept of a single persona that can use services to compose development and services. Ultimately, the job of the cloud engineer is to reduce complexity and reduce friction in the pipeline by using patterns as needed and building platforms and tools consistently.

The panel discussed the reality of testing in production. Charity Majors highlighted that you have no choice, whether you admit it or not; there will always be known unknowns because devs or operators cannot test everything. Software will always have bugs, but the problem can be managed by having more ways to separate problems and test them in a small and controlled way. Testing in production begins with observability, starting with usual tools such as strace and other enhanced toolsets to inspect your code. Also, feature flags and deployment models help find bugs and limit their impact when applications go into production.

Finally, the panel looked backward at the impact and legacy of devops. The panel agreed that devops brought both sides to the table. Operations learned about code and applications while developers learned about infrastructure. Practitioners are software engineers, but each with a different focus. Implementing infrastructure is similar to creating a new feature. However, there is a concern for losing deep systems knowledge, and software engineers are unlikely to have a deep understanding of networking, system administration, or storage. The solution is to train people cross-functionally, i.e., software engineers on systems and operators on coding and application. Teamwork is the key to breaking down operational and knowledge silos in an organization.

Summary

Thank you for attending PulumiUP! Your response to the Pulumi 3.0 release was overwhelming. If you didn’t have the chance to catch PulumiUP on launch day, you can watch the entire event.

PulumiUP: the Event for Cloud Engineers

Wendy Smith — Wed, 14 Apr 2021 00:00:00 +0000

My professional background has included nearly ten years of managing field events and user conferences. I never thought I would say this, but I miss traveling. I even missed Vegas and AWS re:Invent this year. I miss connecting with customers and advocates in our communities. I wish we could all be looking forward to getting together in person in Seattle or Austin or insert any city here. As the year continued, it became clear we were not going back to in-person events anytime soon, and everyone in the industry pivoted to virtual programs while video conferencing became an all-day activity.

We spent a lot of time considering how to best engage and share announcements with our community and encourage the cloud engineering community’s growth. All of us here at Pulumi recognize how valuable your time is and how much time we’ve all spent on virtual event platforms in the last year. With that in mind, we made a deliberate decision to avoid any all-day or multi-day programming. Instead, we are hosting a short 2-hour keynote that we think you will find insightful and valuable, with best practices and real-life situations that you can learn from and apply to your approach to building, delivering, and managing modern cloud applications.

PulumiUP Agenda - Kicking off on 4/20 at 9 am - 11 am PST - Save your spot

Speakers	Title and Company	Subject
Joe Duffy	Co-Founder and CEO, Pulumi	The Modern Cloud Era: All Software is Cloud Software
Adrian Cockcroft & Aaron Kao	VP of Sustainability Architecture, Amazon VP of Marketing, Pulumi	Discussion: What’s different about building in a modern cloud application world
Justin Fitzhugh	VP of Cloud Engineering, Snowflake	Perspectives from Cloud Engineering Pioneers
Keith Redmond	VP of SaaS Engineering, Fenergo	Perspectives from Cloud Engineering Pioneers
Luke Hoban	CTO, Pulumi	Placing Cloud Engineering in the Hands of Every Developer
Sophia Parafina	Technical Marketing Manager, Pulumi
Komal Ali	Software Engineer

Panel: Hear from the Experts: Planning for the Future of Modern Cloud Applications	Title and Company
Moderator: Kat Cosgrove	Developer Advocate, JFrog
Justin Fitzhugh	VP of Cloud Engineering, Snowflake
Charity Majors	CTO/Co-Founder, Honeycomb
Dana Lawson	VP of Engineering, GitHub

Engineering leadership from Snowflake, the cloud data warehousing organization, and Fenergo, a client lifecycle management provider for the financial services industry, will share their perspectives and experiences practicing cloud engineering for the modern cloud era. Justin Fitzhugh, VP of Cloud Engineering at Snowflake, and Keith Redmond, VP of SaaS Engineering at Fenergo, will each share how they reduced costs while empowering every developer on their team.

Everyone at Pulumi is super excited to be sharing this material with our community. If you are interested in cloud engineering, cloud infrastructure, software development, modern applications, or Pulumi, you want to save your spot for this 2-hour event. Whether you’re a seasoned cloud engineer or just curious to learn what’s all the fuss about cloud engineering, you will learn something new about the future of building on the cloud at PulumiUP. You can also sign up for new workshops and other events occurring in April and May. We’ll miss seeing you in person but hope that changes soon! Pulumipus and I are packing our bags and hoping for the best.

Save your spot today!

Building a Development Environment for Cloud Engineering

Sophia Parafina — Wed, 03 Feb 2021 00:00:00 +0000

Starting can be daunting. Before you take your first step, there’s a lot to consider, but you can prepare your development environment ahead of time to make your first steps in cloud engineering smooth and productive. In this article, we’ll cover how to set up your development environment to work across cloud providers, multiple languages, and different operating systems.

Building your toolbox

You might have heard of the French culinary term “mise-en-place,” which means laying out all the ingredients and cookware before starting to cook. That’s what we’re going to do here; layout all the accounts, authorizations, and software that you need to be a successful cloud engineer. We are going to need the following:

A package manager: Every operating system has a package manager for installing software. Unlike binary installers, a package manager lets you manage all software packages, including updates. Package managers can help resolve dependencies, saving you from frustration.
Cloud provider accounts: You can choose to set up one or multiple accounts. The important thing is how to configure credentials in your development environment,
Programming languages: You can choose one or many, but they all have different versions and dependencies.
Code editor: This is personal preference but make sure that it can perform code completion, error checking, and use enums. These features can be the difference between hitting Tab and searching through online documentation for a function.
Pulumi: You can install Pulumi with a package manager and configure

Decisions

Before proceeding, you need to answer three questions:

Which operating system to use for building cloud resources: macOS, Windows, or Linux?
Which cloud provider are you using? This guide covers AWS, Azure, and Google Cloud.
Which programming language will you use? Pulumi supports Node.js (JavaScript and Typescript), Python 3.7 or higher, Golang, and .NET (C#, F#, and VB).

Once you’ve made these choices, you can follow this guide in a choose-your-own-adventure style.

Package manager

Let’s start with the package manager. We’ll use it to install and manage all the software we need, including cloud provider CLIs, programming languages, editors, and Pulumi. Choose your operating system below.

Homebrew is the most popular package manager for macOS. The Command Line Tools (CLT) for XCode is required to install and build Homebrew. Install the XCode Tools first, then install Homebrew from the command line.

$ xcode-select --install
$ /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

If you are uncomfortable with downloading and running an online shell script (with good reason), Homebrew provides an alternate installation method.

Chocolatey is a popular package manager for Windows. It should be installed using an administrative shell.

Make sure that Get-ExecutionPolicy is not restricted.

> Get-ExecutionPolicy

If the command returns Restricted, we can use Bypass to install Chocolatey with the following command:

> Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))

To check if it’s installed type:

> choco

Linux distributions include a package manager, apt for Debian based distributions or yum for Red Hat-based Linux systems. However, they are primarily used for application and system management.

Homebrew is a popular package manager for utilities, Software Development Kits, and programming tools. Install Homebrew from the command line.

$ /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

To add Homebrew to your PATH and your bash shell profile script (~/.profile on Debian or ~/.bash_profile on Red Hat), follow the Next steps instructions to install and configure brew. To check if it’s installed, install a package;

$ brew install hello

Setting up a cloud account

The first task is signing up for an account. Once you have that out of the way, the next steps are installing the CLI and configuring your credentials.

macOS

We’ll use brew to install AWS CLI version 2 and verify if it’s installed by checking the version.

$ brew update && brew install awscli
$ aws --version

The next step is to create and download your AWS access keys and configure your environment to make them available to both the AWS CLI. To create your access keys, follow these directions for programmatic access.

Pulumi uses the AWS SDK to authenticate requests. Your AWS access keys are never sent to pulumi.com.

To configure the AWS CLI with your credentials, use aws configure:

$ aws configure
AWS Access Key ID [None]: <YOUR_ACCESS_KEY_ID>
AWS Secret Access Key [None]: <YOUR_SECRET_ACCESS_KEY>
Default region name [None]:
Default output format [None]:

This creates a ~/.aws/credentials file used by the AWS CLI to authenticate requests.

We’ll use brew to install Azure CLI and check if it’s installed.

$ brew update && brew install azure-cli
$ az --version

To log into Azure, run the `login` command, which opens a browser to log into [Azure](https://login.microsoftonline.com/common/oauth2/authorize).

```bash
$ az login
The default web browser has been opened at https://login.microsoftonline.com/common/oauth2/authorize. Please continue the login in the web browser. If no web browser is available or if the web browser fails to open, use device code flow with `az login --use-device-code`.
You have logged in. Now let us find all the subscriptions to which you have access...

The Azure client is authenticated and ready to use.

Pulumi uses the Azure SDK to authenticate requests. Your AWS access keys are never sent to pulumi.com.

We’ll use brew to install the Google Cloud SDK and CLI.

$ brew update && brew install --cask google-cloud-sdk
Update done!
==> Source [/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/completion.bash.inc] in your profile to enable shell command completion for gcloud.
==> Source [/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/path.bash.inc] in your profile to add the Google Cloud SDK command line tools to your $PATH.

Add the gcloud SDK to $PATH in you ~/.bash_profile

source /usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/completion.bash.inc
source /usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/path.bash.inc

Open a new shell or source ~/.profile for Debian or ~/.bash_profile for Red Hat and check to see if it’s installed and pathed.

$ source ~/.bash_profile
$ gcloud version
Google Cloud SDK 325.0.0
bq 2.0.64
core 2021.01.22
gsutil 4.58

Authenticate using the gcloud CLI.

gcloud init

Pulumi uses the Google Cloud SDK to authenticate requests. Your AWS access keys are never sent to pulumi.com.

Windows

We’ll use chocolatey to install AWS CLI version 2 and check if it’s installed. Open a cmd Command Prompt as Administrator:

> choco install awscli
> aws --version

The install adds the AWS CLI client to the $PATH, so either open a new cmd window or use the refreshenv command to update the window’s environment variables and use the AWS CLI.

Pulumi uses the AWS SDK to authenticate requests. Your AWS access keys are never sent to pulumi.com.

To configure the AWS CLI with your credentials, use aws configure:

> aws configure
AWS Access Key ID [None]: <YOUR_ACCESS_KEY_ID>
AWS Secret Access Key [None]: <YOUR_SECRET_ACCESS_KEY>
Default region name [None]:
Default output format [None]:

This will create the ~/.aws/credentials file used by the AWS CLI to authenticate requests.

We’ll use chocolatey to install Azure CLI and check if it’s installed. Open a cmd Command Prompt as Administrator:

> choco install azure-cli

The Azure CLI client was added to the path, so either open a new cmd window or use the refreshenv command to update the window’s environment variables. To log into Azure, run the login command, which opens a browser to log into Azure.

> refreshenv
> az login

The Azure client will open a browser window and prompt you to sign-in to your account. Once signed-in, the Azure CLI is authenticated and ready to use.

Pulumi uses the Azure SDK to authenticate requests. Your AWS access keys are never sent to pulumi.com.

We’ll use chocolatey to install the Google Cloud SDK and CLI.

> choco install gcloudsdk --ignore-checksums

The gcloud CLI was added to the path, so either open a new cmd window or use the refreshenv command to update the window’s environment variables. Check to see if gcloud CLI is installed and pathed correctly.

> refreshenv
> gcloud version
Google Cloud SDK 325.0.0
bq 2.0.64
core 2021.01.22
gsutil 4.58

Authenticate using the gcloud CLI

> gcloud init
Welcome! This command will take you through the configuration of gcloud.

Your current configuration has been set to: [default]

You can skip diagnostics next time by using the following flag:
 gcloud init --skip-diagnostics

Network diagnostic detects and fixes local network connection issues.
Checking network connection...done.
Reachability Check passed.
Network diagnostic passed (1/1 checks passed).

You must log in to continue. Would you like to log in (Y/n)? Y

Your browser has been opened to visit:
 https://accounts.google.com/o/oauth2/...

A browser window will open to authenticate your client.

Pulumi uses the Google Cloud SDK to authenticate requests. Your AWS access keys are never sent to pulumi.com.

Linux

We’ll use brew to install AWS CLI version 2 and check if it’s installed.

$ brew update && brew install awscli
$ aws --version

Pulumi uses the AWS SDK to authenticate requests. Your AWS access keys are never sent to pulumi.com.

To configure the AWS CLI with your credentials, use aws configure:

$ aws configure
AWS Access Key ID [None]: <YOUR_ACCESS_KEY_ID>
AWS Secret Access Key [None]: <YOUR_SECRET_ACCESS_KEY>
Default region name [None]:
Default output format [None]:

This will create the ~/.aws/credentials file used by the AWS CLI to authenticate requests.

We’ll use brew to install Azure CLI and check if it’s installed.

$ brew update && brew install azure-cli
$ az --version

To log into Azure, run the `login` command, which opens a browser to log into [Azure](https://login.microsoftonline.com/common/oauth2/authorize).

```bash
$ az login
The default web browser has been opened at https://login.microsoftonline.com/common/oauth2/authorize. Please continue the login in the web browser. If no web browser is available or if the web browser fails to open, use device code flow with `az login --use-device-code`.
You have logged in. Now let us find all the subscriptions to which you have access...

The Azure client is authenticated and ready to use.

Pulumi uses the Azure SDK to authenticate requests. Your AWS access keys are never sent to pulumi.com.

We’ll use brew to install the Google Cloud SDK and CLI.

$ brew update && brew install --cask google-cloud-sdk

Update done!

==> Source [/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/completion.bash.inc] in your profile to enable shell command completion for gcloud.
==> Source [/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/path.bash.inc] in your profile to add the Google Cloud SDK command line tools to your $PATH.

Add the gcloud SDK to $PATH to your ~/.bash_profile or ~/.profile.

source /usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/completion.bash.inc
source /usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/path.bash.inc

Open a new shell or source ~/.profile for Debian or ~/.bash_profile for Red Hat and check to see if it’s installed and pathed.

$ source ~/.bash_profile
$ gcloud version
Google Cloud SDK 325.0.0
bq 2.0.64
core 2021.01.22
gsutil 4.58

Authenticate using the gcloud CLI

gcloud init

Pulumi uses the Google Cloud SDK to authenticate requests. Your AWS access keys are never sent to pulumi.com.

Choose a Programming Language

Pulumi supports Node.js (JavaScript and Typescript), Python 3, Golang, and .NET Core (C#, VB, and F#) languages.

Node.js

Use brew to install Node.js for JavaScript and Typescript.

$ brew install node

Use chocolatey to install Node.js for JavaScript and Typescript.

> choco install nodejs

Use brew to install Node.js for JavaScript and Typescript.

$ brew install node

Python

MacOS includes Python; however, versions shipped before December 2019 have Python 2.7 installed, which is deprecated. Pulumi requires Python 3.7 or higher. Use brew to install Python, which installs it at /usr/bin/local/python3.

$ brew install python

In macOS versions with Python 2 installed, calling python uses the 2.7 binary. To ensure that you will always use Python 3, you can add an alias to your .bash_profile.

alias python=/usr/local/bin/python3

It is a best practice to create a virtual environment and activate it for a Python project. Pulumi creates a new venv environment when you start a new Project with pulumi new.

You can install Python in several ways, including typing ‘python` in the command prompt, which brings up the Microsoft Store application. In this article, we’ll continue using chocolatey to install Python 3.

> choco install python --pre

It is a best practice to create a virtual environment and activate it for a Python project. Pulumi creates a new venv environment when you start a new Project with pulumi new.

Use brew to install Python3.

$ brew install python

Use python3 to call Python, but you can add an alias to your .bash_profile or .profile to use just python to run scripts.

alias python=/usr/local/bin/python3

It is a best practice to create a virtual environment and activate it for a Python project. Pulumi creates a new venv environment when you start a new Project with pulumi new.

Golang

Use brew to install golang.

$ brew install golang

It is a best practice to create a local programming environment and set GOPATH as an environment variable, although it is not required for golang versions after 1.8. Pulumi creates a new local environment and go.mod to work with modules each time you start a new Project with pulumi new.

Use chocolatey to install golang.

> choco install golang

It is a best practice to create a local programming environment and set GOPATH as an environment variable, although it is not required for golang versions after 1.16. Pulumi creates a new local environment and go.mod to work with modules each time you start a new Project with pulumi new.

Use brew to install golang.

$ brew install golang

.NET

Install .NET with brew, Pulumi requires .NET Core 3.1 or higher.

$ brew install dotnet

Install .NET with chocolatey. Pulumi requires .NET Core 3.1 or higher.

> choco install dotnet-sdk

Install .NET with brew, Pulumi requires .NET Core 3.1 or higher.

$ brew install dotnet

Get a code editor

While knowing how to use vim or similar text editors is a worthwhile skill, you should use a modern code editor. Here’s a non-exhaustive list why you should use a code editor.

Quickly navigating to a type
Autocompletion when you can’t remember the names of all members by heart
Automatic code generation
Refactoring
Organise imports
Warnings as you type.
Hovering over something to see the docs
Keeping a view of files, errors/warnings/console/unit-tests and source code on the screen
Running unit tests from the same window
Integrated debugging
Integrated source control
Navigating to where a compile-time error or run-time exception occurred directly from the error details.

You can use an IDE (Integrated Development Environment) such as Microsoft Visual Studio, Xcode, or any one of JetBrains’ language-specific IDEs. Alternatively, you can use a lightweight solution with many of the features of an IDE. Popular code editors include Visual Studio Code, Sublime Text, and Atom.

You can install some editors with brew.

$ brew install --cask visual-studio-code
$ brew install --cask sublime-text
$ brew install --cask atom
$ brew install --cask pycharm-ce
$ brew install --cask goland

Install Pulumi

macOS Ventura (13) or later is required.

Homebrew

You can install Pulumi through the Homebrew package manager:

$ brew install pulumi/tap/pulumi

This will install the pulumi CLI to the usual place (often /usr/local/bin/pulumi) and add it to your path.

Subsequent updates can be installed in the usual way:

$ brew upgrade pulumi

Installation Script

To install, run our installation script:

$ curl -fsSL https://get.pulumi.com | sh

This will install the pulumi CLI to ~/.pulumi/bin and add it to your path. When it can’t automatically add pulumi to your path, you will be prompted to add it manually. See How to permanently set $PATH on Unix for guidance.

Windows 8 and 10 are supported.

Chocolatey

You can install Pulumi using elevated permissions through the Chocolatey package manager:

> choco install pulumi

This will install the pulumi CLI to the usual place (often $($env:ChocolateyInstall)\lib\pulumi) and generate the shims (usually $($env:ChocolateyInstall)\bin) to add Pulumi your path.

Subsequent updates can be installed in the usual way:

> choco upgrade pulumi

Verifying your Installation

After installing Pulumi, verify everything is in working order by running the pulumi CLI:

$ pulumi version
v3.259.0

$ pulumi version
v3.259.0

> pulumi version
v3.259.0

Next steps

Congratulations! You have a fully configured environment, and you’re ready to jump into cloud engineering. You can maintain your development environment using the package manager to add or update your toolset. Your code editor provides a modern development platform that takes advantage of all the advances in software engineering. Your coding experience will be more productive and less frustrating.

What are the next steps? Begin with Pulumi’s Getting Started. You can skip the configuration sections and jump straight into your first project. Once you’re done with your first project, try out example projects on Github. You can start with simple projects using the pulumi CLI, such as deploying a web server on AWS with python.

$ pulumi new https://github.com/pulumi/examples/tree/master/aws-py-webserver

This command will download the project from Github, create a virtual environment and activate it, and download all the python package dependencies. You’re ready to go and deploy with pulumi up. Want more? How about deploying Kubernetes on Azure with python?

$ pulumi new https://github.com/pulumi/examples/tree/master/azure-py-aks

You can use the examples as a starting point for building your cloud infrastructure and add resources documented on Pulumi’s Registry and Guides.

The Past, Present, and Future of Cloud Engineering

Joe Duffy — Thu, 08 Oct 2020 00:00:00 +0000

All software is cloud software. All modern applications interact with the cloud in some way, whether it’s using cloud for storage, compute capabilities, or with rich cloud services for data, AI/ML, and so much more, to deliver amazing new experiences. As a result, all developers today are cloud developers, and infrastructure teams are key to enabling innovation across the entire organization. I had a great time telling this story at the Cloud Engineering Summit today and wanted to take a moment to put pen to paper.

Introduction

The cloud is the world’s biggest supercomputer. Although the cloud broke our architectures, workflows, and teams, we are now amidst a renaissance of rebuilding new ways to create truly cloud-native applications and experiences.

We conceived of the Cloud Engineering Summit to celebrate this renaissance. The talks throughout the day explore how we build architectures and applications. We also explore how to operate those cloud architectures in a team setting through workflows using automation and processes to build, deploy, and manage those applications and their infrastructure. Finally, teams are working together more closely and in new ways in order to collaborate to deliver applications and infrastructure across all cloud engineering disciplines.

This is what we are calling Cloud Engineering: the next quantum leap in how we build cloud software. Cloud engineering means bringing the cloud closer to application development and infusing its capabilities to create new cloud-enabled experiences. Cloud engineering also means bringing more engineering practices to infrastructure, enabling us to leverage infrastructure as not just code but software. And finally, cloud engineering enables us to unlock collaboration, enabling the entire team to move faster with confidence.

Cloud Engineering in Practice

When the cloud came along, our initial approach was to reuse the same application patterns from the Internet 1.0 era, starting with “lift and shift” and virtual machine-centric architectures. We’ve gone through at least two major sea changes since then, including the introduction of containers and serverless architectures, and if we were to start over with a blank slate today, we’d surely do things very differently.

In fact, the cloud effectively broke our existing approaches for architectures, workflows, and teams.

As the cloud incrementally broke these over the past decade, we invented best practices, tools, and techniques to cope with it. This led to DevOps and a fundamental change to how we worked together to build increasingly cloud-enabled software. DevOps started from a place of collaboration, bringing engineers and operations people closer together and was a very powerful precursor to what is now happening.

This is where cloud engineering now enters the picture. Instead of being reactionary to change, we can now proactively pursue approaches designed for the cloud. Instead of just coping, we can take a step back and reimagine new ways of doing things.

Cloud engineering takes the best practices from DevOps and extends them. There is no question that DevOps, despite its original intent, ended up more “opsy” than “devvy”. With cloud engineering, application developers are now embracing the cloud and infusing cloud capabilities such as microservices and serverless into applications in fundamentally new ways, creating new application experiences for end users.

And at the same time, infrastructure experts, capabilities, and teams are key to enabling innovation more than ever before. During the cloud era, infrastructure became programmable software, which allows us to apply all the software engineering practices we know and love to infrastructure, including testing, refactoring, security, continuous delivery, and agile.

More cloud infrastructure capabilities have moved into application land, while at the same time more application development practices have moved into infrastructure land. What this means is that the whole team is collaborating more closely than ever before. Everyone is empowered to bring their unique skillsets to the table to enable moving faster and deliver amazing software capabilities, regardless of title — developer, DevOps, systems engineer, infrastructure architect, security engineer, etc. This is the modern cloud engineering organization.

Cloud Engineering Architectures

Infrastructure used to be a tax, and IT a cost center, but cloud infrastructure is increasingly a competitive advantage that can transform entire business models. Just think of the many cloud era companies that are enabling new experiences thanks to the cloud — AirBnB, Amazon, Lyft, Spotify, and others. Zoom experienced a 20x increase in usage when COVID started and, because they architected their application for the cloud, they were able to scale and meet demand. Five to ten years ago, without cloud engineering, that wouldn’t have been possible.

Zoom massively scaling is a great example of the difference between just tossing a traditional application into a virtual machine, versus fully embracing cloud capabilities as a foundational piece of the application’s design. The ability to scale up elastically, and pay as you go, is an immense competitive advantage when designing applications to run on the cloud’s planet-scale supercomputer.

This means that cloud infrastructure is a superpower to be wielded and used for great things, not an afterthought. If only teams can figure out how to harness them, they can accomplish new business outcomes and reshape how applications work.

Application models have also evolved in ways that enable this new cloud engineering era. In the 2010s, you had XML web services and SOA, which was the precursor to microservices, and now with modern programming languages, the shift to distributed architectures is a lot easier. Asynchronous programming and concurrency models are pervasive on all of our modern languages and frameworks, and we can use those things to build truly distributed applications that are composed of many smaller pieces that communicate asynchronously and can scale on demand.

Cloud Engineering Workflows

It’s not just the application architectures that have changed; the way that we build, secure, test, ship, and manage those applications and their infrastructures have also fundamentally changed. This is what we call “workflows”: the way in which we operationalize, and often continuously deliver, our cloud applications and infrastructure into our many dev, test, and production environments.

Infrastructure as code lets us declare infrastructure using “code” (although, in many cases, code is really just markup or “text”). We are only just now beginning to apply all the engineering practices that are available to developers when building infrastructure, with technologies like Pulumi that enable infrastructure as software. This includes leveraging great IDEs and tools, enabling higher levels of productivity. Instead of copying and pasting text, we have real patterns and best practices thanks to sharing and reuse.

Among the many things the cloud broke is the inner loop development cycle. We’re often stepping into and out of our IDE, running command lines, pointing and clicking in the cloud console, and so on. The infrastructure as text world often means finding failures 10 minutes into a deployment, when it’s too late, rather than finding errors interactively in your IDE. By embracing infrastructure as software, and the fact that infrastructure is moving closer to application architectures, our inner loop and great productivity is starting to be restored.

Treating infrastructure as software enables us to test infrastructure before it has been deployed, spin up ephemeral environments to exhaustively validate changes before attempting to deploy them, or even apply advanced forms of testing, such as chaos testing, to ensure applications function properly during a service outage, or fuzz testing, to ensure the application works in the face of randomized changes.

Security is a also a key cross-cutting concern but is still often overlooked during the inner loop of developing cloud software. The security perimeter for cloud software is no longer the four walls of the data center but has instead moved to software managed networks and firewalls. This changes the threat model and requires that teams adopt zero trust and principle of least privilege approaches, where applications, infrastructure, and team members have the least permissions possible to get the job done, minimizing the blast zone of possible incidents. We must also adopt approaches that ensure software is secure before deploying, including applying policies that do things like preventing unintended Internet access to services, block S3 buckets and databases open to the Internet, and scan containers for known vulnerabilities.

Once you’ve shipped an application to production, the job has only just begun. Teams must adopt patterns and practices for how to monitor and react to situations that occur in their live environments. That often also includes testing and flighting changes in production. One approach is to adopt an “OODA loop” like approach, where OODA means Observe, Orient, Decide, Act. This is a decision-making framework that allows for rapid decision making with the context of a situation, and good analogy for how we maintain our cloud applications in the real world. How do we know something is wrong? How do we react? The old model of simply relying on performance counters, such as CPU or memory utilization, and highly structured logging, doesn’t really work in the modern cloud era. Instead, teams are adopting observability, and a “log everything” approach, so that unstructured and semi-structured high-cardinality data is available at all times, enabling us to observe, debug problems in realtime as they occur, and make decisions and act when when things go wrong.

Continuous delivery is not a new idea, however, teams are now moving beyond just delivering applications towards shipping entire environments, packaging applications and infrastructure together. This enables all of the above practices, including security and testing, to apply to entire cloud application environments. Continuous delivery is also how application and infrastructure teams work more closer together, enabling teams to go from shipping features shipping quarterly, to monthly, to every week, to daily, to shipping features on every Git commit. Delivery and workflow are intertwined, and our cloud engineering brings them even closer together across the entire team.

Cloud Engineering Teams

Cloud engineering teams are empowered: everybody brings their unique skills, as an expert in their domain, and collaborates to enable 1+1=3 outcomes. Cloud engineering is not about “NoOps” or operations being less important than it was before. It is about recognizing that all of these skills are essential and that a cloud engineering team is the composition of all of these skillsets.

Application developers can focus on what they’re great at: architecting, implementing, and delivering new functionality to their application’s end users. Developers probably aren’t going to want to suddenly go deep in what it takes to create and configure an AWS virtual private cloud, but they sure do want to tap into all of those amazing data and AI/ML services, and bring container and serverless architectures and continuous delivery into the inner loop of how they are building and shipping code. And there’s just no way to ignore the runtime environment when it comes to debugging, performance tuning, scaling the application, and observability, just to name a few important facets.

Infrastructure and DevOps engineers likewise can also focus on what they’re great at: building scalable, reliable, and cost-effective foundations that the entire organization runs on. Infrastructure teams can benefit from great productivity wins of great IDEs, tools, and true sharing and reuse of best practices. That’s not to say suddenly infrastructure engineers need or want to learn everything there is to know about the latest application frameworks or patterns. But many of the foundations of DevOps, where we apply more software engineering and agile practices to how we ship infrastructure, including test driven infrastructure, have reshaped how infrastructure experts operate.

Security engineers are also empowered to be part of this entire lifecycle. This ensures that security isn’t an afterthought, but instead, that security engineers have all the requisite insertion points into the application and infrastructure architectures and workflows, to ensure that solutions are secure on day one, and stay secure. Security engineering is a central part of how cloud engineering teams operate.

Finally, all of these modern cloud engineering practices have helped teams to cope in this suddenly all-remote COVID world. By embracing distributed architectures, automated workflows, and multi-faceted teams that collaborate closely, entire businesses have made the shift to all-remote and online collaboration much faster. Those teams practicing cloud engineering have had an easier time reacting to rapid change.

Towards a Cloud Operating System

Today’s cloud is comprised of 1,000s of building block services across a landscape of multiple cloud providers. AWS alone offers over 200 of these building blocks, and every SaaS infrastructure provider is part of the picture too. As we have seen above, this offers incredible capability, if teams can only figure out how to harness the power of those building blocks. It’s not always easy but the cloud engineering practices above are key to getting started.

It does beg the question, however: If the cloud is a giant supercomputer, where is its operating system? When will we go beyond building blocks? In fact, we are well on our way.

Kubernetes gives us some fundamental operating system-like components — a scheduler, resource model, and a security model, for example — but a complete operating system must go well beyond this, including programming and application models, and many key facets of the cloud, including all those AWS building blocks, still live outside of the Kubernetes cluster (and likely always will).

What’s the user interface to this giant supercomputer supposed to look like? What’s the human experience of using that computer? We’ve extended our previous experience from client-server desktop and Internet applications into the cloud, but it’s just the start of how we interact with the cloud.

What does the developer’s interface to that supercomputer, the programming model, look like? For now, YAML and tools like Terraform give us a way to configure the building blocks using low-level markup languages, but we need to go beyond this and leverage all the things we know and love about other operating system programming models. This includes using great languages and runtimes like TypeScript, Python, .NET, and Go. This is the approach we’ve taken with Pulumi to start creating richer developer experiences with respect to how we stitch together all those building blocks and build bigger things out of smaller things, but it is still early days here.

Finally, where is the application model? How do we reliably and repeatably install, version, secure, work with, and manage these cloud applications? Containers provide a building block for building and deploying applications in the cloud, Helm and ideas like CNAB provide an installation model, but it is a far cry from an “AppStore” experience that we get on other operating systems. The cloud supercomputer interface is very low-level and bottom-up compared to application models which approach the experience from a top-down perspective.

Despite the clear gaps, it is clear we’ve taken some major steps in the direction of a true cloud operating system.

The Future of Cloud Engineering

As we’ve seen, the future of cloud engineering is already here in a very real way. The cloud is being infused into application development, we are increasingly treating cloud infrastructure as software, and teams are embracing new ways of working together.

We hope you enjoyed learning more about cloud engineering architectures, workflows, and teams. Watch the Cloud Engineering Summit Keynote here or check out one of the many amazing Cloud Engineering talks now!

This event would not have been possible without the participation and support of our top sponsors: