Pulumi Blog: Mikhail Shilkov

Pulumi Updates, Explained: AI-Powered Features in Pulumi CLI

Artur Laksberg — Thu, 22 May 2025 00:00:00 +0000

Note: This post discusses Pulumi Copilot, which Pulumi Neo has replaced. Learn about Neo →

We’re excited to announce the new AI capabilities for Pulumi CLI powered by Pulumi Copilot that translate complex infrastructure changes into clear, human-readable explanations and help resolve deployment errors with actionable guidance. Enable these preview features with the --copilot flag.

At Pulumi, we’re committed to helping you deploy infrastructure efficiently and with minimal friction. Anyone who has worked with cloud infrastructure knows the frustration of sifting through large previews with numerous changes and the difficulty of interpreting cryptic error messages from cloud provider APIs when deployments fail.

There are two key challenges we’ve identified:

Understanding Preview Changes: Reviewing raw update previews can be overwhelming, especially for complex deployments with many resources. It’s difficult to quickly grasp the full scope and impact of pending infrastructure changes.
Interpreting Error Messages: When deployments fail, cloud providers often return error messages that, while comprehensive, bury the core issue within extensive diagnostics, requiring specialized expertise to interpret and address.

Our latest CLI update introduces AI capabilities that tackle both issues. Let’s explore how these new features improve your infrastructure management workflow.

Update Explanation

To address the first challenge above, we’ve introduced a new “explain” menu item in the CLI. When running pulumi preview or pulumi up with the --copilot flag, you’ll see this option in the interactive menu:

In this screenshot, you can see the standard Pulumi update preview showing an AWS VPC with multiple subnets, route tables, and gateways that need to be updated. The interactive menu at the bottom shows the new “explain” option with a sparkle emoji, indicating the AI-powered feature. When selected, this tool analyzes and provides a clear summary of all changes in your pending infrastructure update:

In this example, the AI explanation immediately identifies that this is a “VPC Renaming Operation - Tag Updates Only” and explains that the update is simply changing tags across all components. It clearly states this is a “safe, non-disruptive change that only affects resource labels.”

This explanation serves as a valuable sanity check before deployment. It presents a plain-language summary of exactly what changes will occur, helping you verify that the planned modifications align with your intentions. This visibility is particularly valuable when managing complex infrastructure with many resources, as it helps identify unexpected changes or potential drift between your code and deployed resources. The system also proactively flags potential issues such as security vulnerabilities or destructive updates that might impact your production environment.

Diagnostics Summarization

For the second challenge, when infrastructure deployments fail, cloud providers often return error messages that are technically accurate but difficult to decipher without specialized knowledge. Consider the following scenario where a Go-based Docker image build fails with vendoring issues:

In this example, the standard Go build process failed with a cryptic error about inconsistent vendoring. While the raw output does contain the necessary information, it’s buried within extensive build logs. Our AI-powered Copilot diagnostics immediately identifies the problem and provides a clear, actionable solution: “Docker image update failed due to inconsistent vendoring. Sync vendor with ‘go mod vendor’.”

This clear explanation identifies root causes and provides actionable guidance, transforming dense technical output into concise explanations. It reduces time to resolution by pinpointing both the problem and the exact command needed to fix it, eliminating the need to search through documentation or guess.

Getting Started

The AI-powered features are currently in preview. You can enable them by adding the --copilot flag to your Pulumi CLI commands:

pulumi preview --copilot
pulumi up --copilot

These features require Pulumi CLI version v3.171.0 or greater and the Pulumi Cloud backend. Organization administrators must enable Pulumi Copilot by navigating to Settings > Access Management > Pulumi Copilot in the Pulumi Cloud console.

During this preview period, we’re focused on:

Refining explanations based on user feedback
Adding new capabilities like suggesting fixes for common errors
Improving performance to ensure minimal impact on command execution time

We will continue to monitor usage patterns and incorporate your feedback before enabling these features by default.

We’re excited to see how these AI-powered insights enhance your infrastructure deployment workflow! Please share your experiences with us:

Connect with us in the Pulumi Community Slack
Open an issue on GitHub
Learn more about Pulumi Copilot

Try this today, Get Started with Pulumi for free

Your input will directly influence how we evolve these features from preview to general availability.

AI-Assisted Infrastructure as Code with Pulumi's Model Context Protocol Server

Mikhail Shilkov — Tue, 08 Apr 2025 00:00:00 +0000

Note: This post discusses Pulumi Copilot, which Pulumi Neo has replaced. Learn about Neo →

Infrastructure as Code (IaC) has revolutionized how we manage cloud resources, but navigating complex cloud provider APIs, writing boilerplate code, and iterating through deployment cycles can still be time-consuming. Pulumi offers a fantastic developer experience using familiar programming languages. But what if we could make it even faster and more intuitive by integrating powerful AI assistants directly into the development loop?

This is where the Pulumi Model Context Protocol (MCP) Server integration shines. MCP is a specification that allows language models (like the AI in your coding assistant) to interact with external tools and data sources in a structured way. By connecting AI-powered code assistants with Pulumi’s CLI and registry via MCP, we can bring real-time resource information and infrastructure management directly into the development environment, dramatically reducing friction and accelerating workflows.

Several AI coding assistants like GitHub Copilot, Anthropic’s Claude Code, Windsurf and others are rapidly evolving; this post will use Cursor (an AI-first code editor) to demonstrate a real-world example of this synergy in action.

Setting up the Pulumi MCP Integration in Cursor

Before diving in, you typically need to configure your AI assistant to communicate with the Pulumi MCP server. For Cursor, you can do so by clicking the following button:

The assistant can now leverage Pulumi tools seamlessly. These tools are specific actions enabled by the MCP server—like searching the Pulumi Registry or running a pulumi command—allowing the assistant to gather information or interact with your Pulumi project.

For detailed setup instructions for Cursor, Claude Code, Windsurf, Claude Desktop, and other AI assistants, see the Pulumi MCP Server documentation.

The Goal: Provisioning an AKS Cluster

Our objective for this walkthrough is to provision a temporary Azure Kubernetes Service (AKS) cluster for a short experiment using Pulumi and TypeScript. We need the cluster created with minimal fuss and its kubeconfig exported for access.

The Traditional Approach

The conventional method for this task involves significant context switching: searching Azure and Pulumi documentation in a browser, writing code in an editor, running commands (pulumi preview, pulumi up) in a terminal, and manually correlating information between these different environments. This process can be slow, requires deep knowledge recall (or constant lookups), and is prone to errors during the manual translation from documentation to code.

The AI Assistant + Pulumi MCP Approach

Let’s walk through how the same task unfolds much more efficiently using an AI assistant integrated with Pulumi MCP.

Understanding the AI Assistant + Tool Interaction

Before detailing the steps, it’s helpful to understand the user interface flow when the AI assistant uses integrated tools like those provided by the Pulumi MCP integration. Typically:

The developer provides a prompt or instruction in the chat interface within their editor.
The AI assistant analyzes the request and determines that it needs specific information (like resource properties) or needs to perform an action (like running pulumi preview).
The assistant indicates it will use a specific tool (e.g., pulumi_registry_listResources, pulumi_cli_preview). This often appears as a distinct UI element in the chat, showing the tool name and parameters being used.
The tool executes via the MCP server, interacting with the Pulumi CLI or registry as needed.
The output or result from the tool (e.g., registry listings, preview results, deployment errors) is displayed directly in the chat interface.
The AI assistant processes this output and uses it to continue the task – either by generating code, providing an answer, or deciding on the next step (like suggesting a fix for an error).

This tight loop keeps the developer focused within their editor environment, minimizing disruptions.

Step-by-Step Walkthrough

Here’s how the AKS provisioning task played out:

The Request

The developer starts with a natural language request to the AI assistant within the editor:

“I have an empty Pulumi project with TypeScript. Please edit the program to provision an AKS cluster for me. It’s a temporary AKS cluster that I need for a short experiment, so I don’t need any particular configuration of it. Just export its kubeconfig when you are done. Please use the tools to lookup resource information and to run pulumi preview to make sure your code works. When done, run pulumi up for me.”

AI-Powered Resource Discovery

Instead of the developer manually searching docs, the AI assistant leverages the MCP integration.

The AI assistant first queries the Pulumi Registry via MCP to list resources within the Azure Native provider’s containerservice module (using the pulumi_registry_listResources tool). This immediately identified ManagedCluster as the relevant resource.
The AI assistant then requests detailed information specifically for ManagedCluster (using pulumi_registry_getResource), directly retrieving its required properties, descriptions, and structure without leaving the editor.

AI Code Generation & Editing

Using the information retrieved from the Pulumi Registry, the assistant generated the necessary TypeScript code. It defined the ManagedCluster resource with a basic configuration and used its code editing capabilities to insert the code directly into the developer’s index.ts file.

Integrated Validation

Before attempting a potentially time-consuming deployment, the assistant ran pulumi preview using the integrated CLI tool (pulumi_cli_preview). The preview output appeared directly in the chat.

The preview succeeded, showing the resources that would be created. The assistant then proceeded to run pulumi up using the integrated CLI tool (pulumi_cli_up). However, the deployment flagged an error: an incorrect attempt to interpolate a Pulumi Output<string> directly into a resource property.

AI-Assisted Debugging & Iteration

The assistant analyzed the error message from the tool:

It identified the incorrect string interpolation and proposed a fix (using pulumi.interpolate), applying it via the editing tool.
Self-correction: The first fix inadvertently introduced a circular dependency (caught by linters integrated into the editor and surfaced to the assistant). The assistant recognized this and further refined the code by removing the problematic property, correctly relying on Azure’s default behavior.
Another attempt to deploy using pulumi up failed, this time surfacing an Azure API error: the provided SSH key format was invalid.

Collaborative Problem Solving & Tooling

The developer, seeing the SSH key error, suggested using the Pulumi TLS package to generate a valid key dynamically.

The assistant used the integrated terminal tool to execute npm install @pulumi/tls.
It then edited the index.ts file again, incorporating the tls.PrivateKey resource and correctly using its publicKeyOpenssh output for the ManagedCluster’s Linux profile.

Successful Deployment

The subsequent pulumi preview showed the correct plan (including the new TLS key resource). The assistant then executed pulumi up, which completed successfully. The success message and resource summary appeared in the chat.

Accessing Outputs

Finally, retrieving the kubeconfig was trivial. The assistant used the stack output tool (pulumi_cli_stack_output kubeconfig) to fetch and display the configuration directly.

Why This Changes the Game

Integrating AI assistants with the Pulumi Model Context Protocol offers tangible benefits:

Reduced Context Switching: The entire workflow – discovery, coding, validation, deployment, debugging, output retrieval – happens primarily inside the editor. No more juggling browser tabs and terminal windows.
Accelerated Discovery: MCP integration provides immediate, context-aware access to Pulumi resource schemas and documentation snippets.
Faster Coding: AI generates boilerplate and resource definitions quickly based on registry data and natural language requests.
Tighter Feedback Loop: pulumi preview and pulumi up results are instantly available within the coding environment, enabling rapid iteration.
Intelligent Assistance & Collaboration: The AI assists with debugging complex errors, incorporates developer suggestions, and leverages integrated tooling effectively.

Conclusion

The synergy between AI coding assistants and the Pulumi Model Context Protocol (MCP) integration creates a remarkably efficient environment for Infrastructure as Code development. By bringing cloud resource knowledge, code generation, and the Pulumi CLI workflow directly into the editor, developers can build, deploy, and iterate on their infrastructure faster and with significantly less friction.

Whether you use Cursor, Copilot, Claude Code, Windsurf, or another emerging AI tool, integrating it with the Pulumi MCP server offers a glimpse into the future of streamlined, intelligent IaC development, ultimately boosting productivity and improving the overall developer experience.

Announcing the Next-Generation of Pulumi Components: Enabling Infrastructure Abstractions

Mikhail Shilkov — Thu, 27 Mar 2025 00:00:00 +0000

Pulumi Components enable you to create, share, and consume reusable infrastructure building blocks across your organization and the broader community. Today, we’re excited to announce significant enhancements to Pulumi Components that make them more powerful, accessible, and easier to use than ever before.

With this release, we’ve made it possible to author components in one language and consume them in any other Pulumi language—including Pulumi YAML. This breakthrough enables platform teams to build sophisticated infrastructure abstractions in their preferred programming language while application developers can easily consume these components using simpler interfaces or even YAML, all without sacrificing type safety or functionality.

For platform teams, the new Pulumi Components offer a more intuitive, more secure way to codify organizational standards. For developers, it’s a straightforward way to pull in platform-approved abstractions without having to worry about all the underlying details. It’s a win-win that drastically accelerates innovation and drives productivity across your teams.

Understanding Pulumi Components

At their core, Pulumi Components are logical groupings of resources that encapsulate infrastructure patterns and best practices. Unlike standard resources that map directly to cloud provider resources, components are higher-level abstractions that can contain multiple child resources working together to implement a specific capability. While similar in concept to Terraform modules, Pulumi Components offer more power through full programming language capabilities, stronger typing, and now, cross-language support.

For example, a SecureBucket component might include a bucket, versioning configuration, encryption settings, and tagging policies—all bundled together as a single, reusable unit that enforces your organization’s security and compliance standards.

Pulumi has always supported creating components within a single language - you can simply create a class that extends ComponentResource in your language of choice and use it within programs written in that same language. This approach works well for teams working within a single language ecosystem, requiring no additional setup beyond standard programming patterns. However, these traditional single-language components can’t be consumed from other Pulumi languages, limiting their shareability across teams with different language preferences.

What’s New with Pulumi Components

Our latest enhancements focus on making components more accessible and easier to share:

Cross-language consumption: Components authored in one language can now be consumed in any Pulumi language, now including Pulumi YAML.
Simplified sharing: Share components by simply pushing code to a git repository - no need to manually publish SDKs.
Streamlined consumption: Use the pulumi package add command with any git URL (like github.com/org/repo) to easily incorporate any components into your projects.
YAML authoring support: For the first time, you can now author components directly in YAML, making it easier for teams with limited programming experience to create and maintain reusable infrastructure patterns.

How It Works

Creating Components

The code for creating a cross-language component starts similarly to a single-language component: define a class that extends ComponentResource:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

export interface SecureBucketArgs {
 bucketName?: pulumi.Input<string>;
 versioning?: pulumi.Input<boolean>;
 encryption?: pulumi.Input<boolean>;
 tags?: { [key: string]: pulumi.Input<string> };
}

export class SecureBucket extends pulumi.ComponentResource {
 public readonly bucket: aws.s3.BucketV2;
 public readonly bucketName: pulumi.Output<string>;

 constructor(name: string, args: SecureBucketArgs = {}, opts?: pulumi.ComponentResourceOptions) {
 super("mycomponents:index:SecureBucket", name, {}, opts);

 // Create an S3 bucket with best practices by default
 this.bucket = new aws.s3.BucketV2(`${name}`, {
 bucket: args.bucketName,
 tags: {
 ManagedBy: "Pulumi",
 ...args.tags,
 },
 }, { parent: this });

 // Conditionally enable versioning
 if (args.versioning !== false) {
 new aws.s3.BucketVersioningV2(`${name}-versioning`, {
 bucket: this.bucket.id,
 versioningConfiguration: {
 status: "Enabled",
 },
 }, { parent: this });
 }

 // Conditionally enable encryption
 if (args.encryption !== false) {
 new aws.s3.BucketServerSideEncryptionConfigurationV2(`${name}-encryption`, {
 bucket: this.bucket.id,
 rules: [{
 applyServerSideEncryptionByDefault: {
 sseAlgorithm: "AES256",
 },
 }],
 }, { parent: this });
 }

 this.bucketName = this.bucket.id;

 this.registerOutputs({
 bucket: this.bucket,
 bucketName: this.bucketName,
 });
 }
}

import pulumi
import pulumi_aws as aws
from typing import Optional, Dict, Any, Mapping

class SecureBucketArgs(TypedDict):
 bucket_name: Optional[pulumi.Input[str]]
 versioning: Optional[bool]
 encryption: Optional[bool]
 tags: Optional[Mapping[str, pulumi.Input[str]]]

class SecureBucket(pulumi.ComponentResource):
 bucket_name: pulumi.Output[str]
 def __init__(self, name: str, args: SecureBucketArgs, opts: Optional[pulumi.ResourceOptions] = None):
 super().__init__('mycomponents:index:SecureBucket', name, {}, opts)

 args = args or SecureBucketArgs()

 # Create an S3 bucket with best practices by default
 bucket = aws.s3.BucketV2(
 f"{name}",
 bucket=args.get("bucket_name"),
 tags={
 "ManagedBy": "Pulumi",
 **args.get("tags", {})
 },
 opts=pulumi.ResourceOptions(parent=self)
 )

 # Conditionally enable versioning
 if args.get("versioning", True):
 aws.s3.BucketVersioningV2(
 f"{name}-versioning",
 bucket=bucket.id,
 versioning_configuration=aws.s3.BucketVersioningV2VersioningConfigurationArgs(
 status="Enabled"
 ),
 opts=pulumi.ResourceOptions(parent=self)
 )

 # Conditionally enable encryption
 if args.get("encryption", True):
 aws.s3.BucketServerSideEncryptionConfigurationV2(
 f"{name}-encryption",
 bucket=bucket.id,
 rules=[aws.s3.BucketServerSideEncryptionConfigurationV2RuleArgs(
 apply_server_side_encryption_by_default=aws.s3.BucketServerSideEncryptionConfigurationV2RuleApplyServerSideEncryptionByDefaultArgs(
 sse_algorithm="AES256"
 )
 )],
 opts=pulumi.ResourceOptions(parent=self)
 )

 self.bucket_name = bucket.id

 self.register_outputs({
 "bucketName": self.bucket_name
 })

using System.Collections.Generic;
using Pulumi;
using Pulumi.Aws.S3;
using Pulumi.Aws.S3.Inputs;

namespace MyComponents
{
 public class SecureBucketArgs : ResourceArgs
 {
 [Input("bucketName")]
 public Input<string>? BucketName { get; set; }

 [Input("versioning")]
 public bool? Versioning { get; set; }

 [Input("encryption")]
 public bool? Encryption { get; set; }

 [Input("tags")]
 public Dictionary<string, Input<string>>? Tags { get; set; }
 }

 public class SecureBucket : ComponentResource
 {
 [Output("bucketName")]
 public Output<string> BucketName { get; private set; }

 public SecureBucket(string name, SecureBucketArgs? args, ComponentResourceOptions? opts = null)
 : base("mycomponents:index:SecureBucket", name, opts)
 {
 args ??= new SecureBucketArgs();

 // Create an S3 bucket with best practices by default
 var tags = new InputMap<string>
 {
 { "ManagedBy", "Pulumi" }
 };

 if (args.Tags != null)
 {
 foreach (var tag in args.Tags)
 {
 tags[tag.Key] = tag.Value;
 }
 }

 var bucket = new BucketV2($"{name}", new BucketV2Args
 {
 Bucket = args.BucketName,
 Tags = tags
 }, new CustomResourceOptions { Parent = this });

 // Conditionally enable versioning
 if (args.Versioning != false)
 {
 new BucketVersioningV2($"{name}-versioning", new BucketVersioningV2Args
 {
 Bucket = bucket.Id,
 VersioningConfiguration = new BucketVersioningV2VersioningConfigurationArgs
 {
 Status = "Enabled"
 }
 }, new CustomResourceOptions { Parent = this });
 }

 // Conditionally enable encryption
 if (args.Encryption != false)
 {
 new BucketServerSideEncryptionConfigurationV2($"{name}-encryption", new BucketServerSideEncryptionConfigurationV2Args
 {
 Bucket = bucket.Id,
 Rules = new[]
 {
 new BucketServerSideEncryptionConfigurationV2RuleArgs
 {
 ApplyServerSideEncryptionByDefault = new BucketServerSideEncryptionConfigurationV2RuleApplyServerSideEncryptionByDefaultArgs
 {
 SseAlgorithm = "AES256"
 }
 }
 }
 }, new CustomResourceOptions { Parent = this });
 }

 BucketName = bucket.Id;

 RegisterOutputs(new Dictionary<string, object?>
 {
 { "BucketName", BucketName }
 });
 }
 }
}

package components

import (
 "github.com/pulumi/pulumi-aws/sdk/v7/go/aws/s3"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

// SecureBucketArgs contains the arguments for creating a SecureBucket
type SecureBucketArgs struct {
 BucketName pulumi.StringInput `pulumi:"bucketName,optional"`
 Versioning *bool `pulumi:"versioning,optional"`
 Encryption *bool `pulumi:"encryption,optional"`
 Tags map[string]pulumi.StringInput `pulumi:"tags,optional"`
}

// SecureBucket is a component that creates an S3 bucket with security best practices
type SecureBucket struct {
 pulumi.ResourceState

 BucketName pulumi.StringOutput `pulumi:"bucketName"`
}

// NewSecureBucket creates a new SecureBucket component
func NewSecureBucket(ctx *pulumi.Context, name string, args SecureBucketArgs, opts ...pulumi.ResourceOption) (*SecureBucket, error) {
 component := &SecureBucket{}
 err := ctx.RegisterComponentResource("mycomponents:index:SecureBucket", name, component, opts...)
 if err != nil {
 return nil, err
 }

 // Create an S3 bucket with best practices by default
 tags := pulumi.StringMap{
 "ManagedBy": pulumi.String("Pulumi"),
 }
 if args.Tags != nil {
 for k, v := range args.Tags {
 tags[k] = v
 }
 }

 bucket, err := s3.NewBucketV2(ctx, name, &s3.BucketV2Args{
 Bucket: args.BucketName,
 Tags: tags,
 }, pulumi.Parent(component))
 if err != nil {
 return nil, err
 }

 // Conditionally enable versioning
 if args.Versioning != nil && !*args.Versioning {
 // Skip versioning
 } else {
 _, err = s3.NewBucketVersioningV2(ctx, name+"-versioning", &s3.BucketVersioningV2Args{
 Bucket: bucket.ID(),
 VersioningConfiguration: &s3.BucketVersioningV2VersioningConfigurationArgs{
 Status: pulumi.String("Enabled"),
 },
 }, pulumi.Parent(component))
 if err != nil {
 return nil, err
 }
 }

 // Conditionally enable encryption
 if args.Encryption != nil && !*args.Encryption {
 // Skip encryption
 } else {
 _, err = s3.NewBucketServerSideEncryptionConfigurationV2(ctx, name+"-encryption", &s3.BucketServerSideEncryptionConfigurationV2Args{
 Bucket: bucket.ID(),
 Rules: s3.BucketServerSideEncryptionConfigurationV2RuleArray{
 &s3.BucketServerSideEncryptionConfigurationV2RuleArgs{
 ApplyServerSideEncryptionByDefault: &s3.BucketServerSideEncryptionConfigurationV2RuleApplyServerSideEncryptionByDefaultArgs{
 SseAlgorithm: pulumi.String("AES256"),
 },
 },
 },
 }, pulumi.Parent(component))
 if err != nil {
 return nil, err
 }
 }

 component.BucketName = bucket.ID().ToStringOutput()
 return component, nil
}

package io.mikhail.components;

import com.pulumi.core.Output;
import com.pulumi.aws.s3.BucketV2;
import com.pulumi.aws.s3.BucketV2Args;
import com.pulumi.aws.s3.BucketVersioningV2;
import com.pulumi.aws.s3.BucketVersioningV2Args;
import com.pulumi.aws.s3.inputs.BucketVersioningV2VersioningConfigurationArgs;
import com.pulumi.aws.s3.BucketServerSideEncryptionConfigurationV2;
import com.pulumi.aws.s3.BucketServerSideEncryptionConfigurationV2Args;
import com.pulumi.aws.s3.inputs.BucketServerSideEncryptionConfigurationV2RuleArgs;
import com.pulumi.aws.s3.inputs.BucketServerSideEncryptionConfigurationV2RuleApplyServerSideEncryptionByDefaultArgs;
import com.pulumi.resources.ComponentResource;
import com.pulumi.resources.ComponentResourceOptions;
import com.pulumi.resources.CustomResourceOptions;
import com.pulumi.core.annotations.Export;
import java.util.Map;
import java.util.HashMap;
import java.util.List;

public class SecureBucket extends ComponentResource {
 @Export
 public final Output<String> bucketName;

 public SecureBucket(String name, SecureBucketArgs args, ComponentResourceOptions options) {
 super("mycomponents:index:SecureBucket", name, options);

 // Create an S3 bucket with best practices by default
 Map<String, Output<String>> tags = new HashMap<>();
 tags.put("ManagedBy", Output.of("Pulumi"));
 if (args.tags() != null && !args.tags().isEmpty()) {
 tags.putAll(args.tags());
 }

 var bucket = new BucketV2(name, BucketV2Args.builder()
 .bucket(args.bucketName())
 //.tags(tags)
 .build(),
 CustomResourceOptions.builder().parent(this).build());

 // Conditionally enable versioning
 if (args.versioning() != Boolean.FALSE) {
 new BucketVersioningV2(name + "-versioning", BucketVersioningV2Args.builder()
 .bucket(bucket.id())
 .versioningConfiguration(BucketVersioningV2VersioningConfigurationArgs.builder()
 .status("Enabled")
 .build())
 .build(),
 CustomResourceOptions.builder().parent(this).build());
 }

 // Conditionally enable encryption
 if (args.encryption() != Boolean.FALSE) {
 new BucketServerSideEncryptionConfigurationV2(name + "-encryption",
 BucketServerSideEncryptionConfigurationV2Args.builder()
 .bucket(bucket.id())
 .rules(List.of(BucketServerSideEncryptionConfigurationV2RuleArgs.builder()
 .applyServerSideEncryptionByDefault(
 BucketServerSideEncryptionConfigurationV2RuleApplyServerSideEncryptionByDefaultArgs.builder()
 .sseAlgorithm("AES256")
 .build())
 .build()))
 .build(),
 CustomResourceOptions.builder().parent(this).build());
 }

 this.bucketName = bucket.id();

 this.registerOutputs(Map.of(
 "bucketName", this.bucketName
 ));
 }
}

package io.mikhail.components;

import com.pulumi.core.Output;
import com.pulumi.resources.ResourceArgs;
import com.pulumi.core.annotations.Import;

import java.util.Map;
import java.util.HashMap;

public class SecureBucketArgs extends ResourceArgs {
 @Import(name="bucketName")
 private Output<String> bucketName;

 public Output<String> bucketName() {
 return this.bucketName;
 }

 @Import(name="versioning")
 private Boolean versioning;

 public Boolean versioning() {
 return this.versioning;
 }

 @Import(name="encryption")
 private Boolean encryption;

 public Boolean encryption() {
 return this.encryption;
 }

 @Import(name="tags")
 private Map<String, Output<String>> tags = new HashMap<>();

 public Map<String, Output<String>> tags() {
 return this.tags;
 }

 private SecureBucketArgs() {}

 public SecureBucketArgs(Output<String> bucketName, Boolean versioning, Boolean encryption, Map<String, Output<String>> tags) {
 this.bucketName = bucketName;
 this.versioning = versioning;
 this.encryption = encryption;
 this.tags = tags;
 }
}

# YAML doesn't directly support authoring complex components
# For a complete component, you'd typically write it in another language
# This is a simplified component without conditionals or tags merges
runtime: yaml
name: my-components
components:
 SecureBucket:
 inputs:
 bucketName:
 type: string
 resources:
 bucket:
 type: aws:s3/bucketV2:BucketV2
 properties:
 bucket: ${bucketName}
 tags:
 ManagedBy: Pulumi

 bucketVersioning:
 type: aws:s3/bucketVersioningV2:BucketVersioningV2
 properties:
 bucket: ${bucket.id}
 versioningConfiguration:
 status: Enabled

 bucketEncryption:
 type: aws:s3/bucketServerSideEncryptionConfigurationV2:BucketServerSideEncryptionConfigurationV2
 properties:
 bucket: ${bucket.id}
 rules:
 - applyServerSideEncryptionByDefault:
 sseAlgorithm: AES256

 outputs:
 bucketName: ${bucket.id}

The PulumiPlugin.yaml File

The PulumiPlugin.yaml file tells Pulumi that this directory is a component package that can be consumed across languages (it is not required for a single language component), rather than a standard Pulumi program. In it, you define the language runtime needed to load the plugin:

runtime: nodejs

runtime: python

runtime: dotnet

runtime: go

runtime: java

runtime: yaml

For other languages, you’d use the appropriate runtime value. This file is the key that enables Pulumi to generate SDKs in other languages from your component code.

Define an Entry Point

The entrypoint analyzes components to automatically build a schema, and interact with the Pulumi engine to manage the component lifecycle.

No separate entry point is required for TypeScript. All exported component classes will automatically be exposed to be consumed from other languages.

Create a _main_.py file in your component directory
In the main function, add a call to component_provider_host, specifying a list of components for the components argument

from pulumi.provider.experimental import component_provider_host
from staticpage import MyComponent

if __name__ == "__main__":
 component_provider_host(name="python-components", components=[MyComponent])

Define a main.go file
Declare an instance of NewProviderBuilder, passing in a name, namespace and the components being built

The code below uses the new pulumi-go-provider v1 APIs. Make sure you are using the latest version of github.com/pulumi/pulumi-go-provider.

package main

import (
 "context"

 "github.com/pulumi/pulumi-go-provider/infer"
)

func main() {
 prov, err := infer.NewProviderBuilder().
 WithNamespace("your-org-name").
 WithComponents(
 infer.ComponentF(NewSecureBucket),
 ).
 Build()
 if err != nil {
 panic(err)
 }

 _ = prov.Run(context.Background(), "go-components", "v0.0.1")
}

Create a Program.cs file
Add an entry point that calls the ComponentProviderHost

using System.Threading.Tasks;

class Program
{
 public static Task Main(string []args) =>
 Pulumi.Experimental.Provider.ComponentProviderHost.Serve(args);
}

Create an App.java file
Create a new instance of ComponentProviderHost in the entry point

package com.example.components;

import java.io.IOException;
import com.pulumi.provider.internal.Metadata;
import com.pulumi.provider.internal.ComponentProviderHost;

public class App {
 public static void main(String[] args) throws IOException, InterruptedException {
 new ComponentProviderHost("java-components", App.class.getPackage()).start(args);
 }
}

No separate entry point is required for YAML.

Pulumi Components new capability is designed to make sharing infrastructure building blocks as easy as possible, whether you’re publishing them to a Git repository or referencing them from a local folder in a monorepo.

Publishing to Git

Once you’ve authored your component, sharing it is as simple as pushing your code to a Git repository and optionally tagging a release. From there, anyone (including you) can pull it down and use it in their own Pulumi projects.

For example, after pushing your SecureBucket to GitHub and tagging it as v1.0.0, others can consume it by running:

pulumi package add github.com/myorg/secure-s3-component@v1.0.0

Under the hood, Pulumi:

Fetches your code from GitHub
Auto-generates a local SDK from the component’s schema
Makes it available to your Pulumi program in your chosen language

Referencing Components Locally

For scenarios like monorepos, rapid development iterations, or when you’re working with components that don’t need to be published to a repository, you can reference local source code directly:

pulumi package add ./components/secure-s3-component

Pulumi will identify the folder as a Pulumi component project, generate a local SDK, and make it available for import in your program—even if your consumer program is in a different language. Both relative paths (as shown above) and absolute paths (like /path/to/local/secure-s3-component) are supported, giving you flexibility for different project layouts.

Consuming Components Across Languages

The real power comes in how these components can now be consumed in any Pulumi language:

import * as pulumi from "@pulumi/pulumi";
import { SecureBucket } from "@myorg/secure-s3-component";

const secureBucket = new SecureBucket("my-secure-bucket", {
 bucketName: "my-company-secure-assets",
 tags: {
 Environment: "production",
 Department: "engineering",
 },
});

export const bucketName = secureBucket.bucketName;

import pulumi
from myorg_secure_s3_component import SecureBucket

secure_bucket = SecureBucket("my-secure-bucket",
 bucket_name="my-company-secure-assets",
 tags={
 "Environment": "production",
 "Department": "engineering",
 })

pulumi.export("bucket_name", secure_bucket.bucket_name)

using System.Collections.Generic;
using Pulumi;
using MyOrg.SecureS3Component;

return await Deployment.RunAsync(() =>
{
 var tags = new Dictionary<string, string>
 {
 { "Environment", "production" },
 { "Department", "engineering" }
 };

 var secureBucket = new SecureBucket("my-secure-bucket", new SecureBucketArgs
 {
 bucketName = "my-company-secure-assets",
 Tags = tags
 });

 // Export the bucket name
 return new Dictionary<string, object?>
 {
 ["BucketName"] = secureBucket.BucketName
 };
});

package main

import (
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
 secure "github.com/myorg/secure-s3-component/sdk/go/secure-s3-component"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 tags := make(map[string]string)
 tags["Environment"] = "production"
 tags["Department"] = "engineering"

 secureBucket, err := secure.NewSecureBucket(ctx, "my-secure-bucket", &secure.SecureBucketArgs{
 BucketName: pulumi.String("my-company-secure-assets"),
 Tags: pulumi.ToStringMap(tags),
 })
 if err != nil {
 return err
 }

 ctx.Export("bucketName", secureBucket.BucketName)
 return nil
 })
}

package mypackage;

import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.myorg.secures3component.SecureBucket;
import com.myorg.secures3component.SecureBucketArgs;

import java.util.Map;

public class App {
 public static void main(String[] args) {
 Pulumi.run(ctx -> {
 var tags = Map.of(
 "Environment", "production",
 "Department", "engineering"
 );

 var bucketArgs = SecureBucketArgs.builder()
 .bucketName("my-company-secure-assets")
 .tags(tags)
 .build();

 var secureBucket = new SecureBucket("my-secure-bucket", bucketArgs);

 ctx.export("bucketName", secureBucket.bucketName());
 });
 }
}

name: my-infrastructure
runtime: yaml
description: A stack that uses SecureBucket component

packages:
 secure-s3-component: github.com/myorg/secure-s3-component@v1.0.0

resources:
 secureBucket:
 type: secure-s3-component:SecureBucket
 properties:
 bucketName: my-company-secure-assets
 tags:
 Environment: production
 Department: engineering

outputs:
 bucketName: ${secureBucket.bucketName}

See It In Action

Check out this video of authoring components in Java and consuming them in YAML to see Pulumi Components in action:

Build reusable infrastructure components

Author a component once in your preferred language, then let your teams consume it from TypeScript, Python, Go, or YAML with Pulumi.

Get started

Pro Tips

Tagging Releases: If you’re working with Git-based components, you can tag version releases (both stable and preview versions). This ensures downstream users can easily choose specific versions without chasing a moving target.
Managing Dependencies: Once you’ve added an entry to the packages section of your Pulumi.yaml file, you can run pulumi install to generate a local SDK in your project. This command will process all packages listed in your Pulumi.yaml and create the necessary SDK files. Check in these files if you want fully reproducible builds; or add them to .gitignore if you prefer to regenerate them on each checkout. When using .gitignore, team members will need to run pulumi install after checkout to regenerate the SDK.
Leveraging Private Repos: If you need to keep your components private, you can still point to a Git source in a private repository. Just make sure the correct access tokens are set in the environment as described in our package management documentation.

The Spectrum of Pulumi Components You Can Build

You can use Pulumi Components with more flexibility and control depending on your use case. This table shows the variety of use cases:

Feature	Single language	Multi-language with Auto-Generated SDKs	Manual Schema and SDKs
Best for	Quick development within a single language ecosystem	Cross-language teams needing to share components	More flexibility and control needed or strict API requirements
Cross-language consumption	No - limited to original language	Yes - consume in any Pulumi language	Yes - consume in any Pulumi language
Setup complexity	Minimal - standard programming patterns	Minimal - just requires package management	High - requires schema authoring and validation
Development workflow	Fast iteration with direct code changes	Fast, requires SDK regeneration when component changes	Complex with schema updates and SDK publishing
API control	Full control within your language	Moderate - auto-generated from source	Full - ship the same interface to all consumers
Version management	Simple - standard code versioning	Moderate - requires careful API changes	Complex - strict semantic versioning needed
Typical user	Individual developers or same-language teams	Platform teams sharing with developers	Enterprise teams with strict requirements or package publishers
Ideal use cases	• Rapid prototyping • Single team projects • Simple components	• Organization-wide libraries • Platform engineering • Multi-language environments	• Published packages • Complex validation needs
Limitations	• Single language only • No additional setup required	• SDK regeneration overhead • Runtime dependencies • Some translation limitations	• Complex setup • Steep learning curve • Slower iteration

Conclusion

The ability to author components in one language and consume them in any other Pulumi language—including YAML—provides unprecedented flexibility and collaboration opportunities between platform teams and application developers.

To learn more about Pulumi Components, check out our documentation on components, building a component, packaging and publishing components, and component schemas.

We’re excited to see what you build with Pulumi Components! Share your feedback with us on the Pulumi Community Slack or open an issue in the pulumi/pulumi GitHub repository.

Introducing Customizable Resource Auto-naming in Pulumi

Mikhail Shilkov — Thu, 16 Jan 2025 00:00:00 +0000

I’m thrilled to announce that you can now customize how Pulumi names your cloud resources! Our default auto-naming feature has helped thousands of customers successfully manage cloud resources at scale by automatically ensuring unique, conflict-free resource names across their cloud deployments. This robust naming system has been particularly valuable for teams managing multiple environments, handling zero-downtime deployments, and maintaining clear resource organization. Today, we’re taking it to the next level by giving you control over how these names are generated.

Over the years, we’ve heard from many teams using Pulumi that while they love the power and convenience of our auto-naming system, they need it to work with their organization’s naming standards - whether that’s adding cost center identifiers, following compliance rules, or matching existing naming patterns. The GitHub issue tracking this feature has gathered quite some attention (50 thumbs up!) and lots of great input from the community.

Today, I’m excited to introduce resource auto-naming configuration. Now you can have the best of both worlds: keep the robustness of Pulumi’s auto-naming while making it follow your team’s naming conventions. Want your resources to include environment tags? Project prefixes? Random suffixes of specific length? Disable auto-naming entirely? It’s all possible now, and it works across all major cloud providers.

The Road to Better Resource Naming

The original feature request I opened in June 2018 when I was a Pulumi customer. It has generated extensive discussion, with users sharing various use cases and requirements. Today, I’m happy to finally close that issue with a solution that addresses the community’s needs while maintaining Pulumi’s robust resource management capabilities.

Check out the full story:

Introducing Auto-naming Configuration

With the new auto-naming configuration feature, you now have full control over how Pulumi generates resource names. Here are some common scenarios you can achieve:

Disable Auto-naming

If you want complete control over your resource names, you can disable Pulumi auto-naming entirely:

pulumi:autonaming:
 mode: disabled

In this mode, Pulumi will require you to provide explicit physical names for all resources.

Use Logical Names As-Is

For scenarios where you want Pulumi to copy exactly the logical names to become the physical names, you can use the verbatim mode:

pulumi:autonaming:
 mode: verbatim

No random suffixes will be added to the resource names.

Note, when an update requires replacing the resource, Pulumi’s default behavior is to create the new resource and then delete the old resource. However, when using verbatim names or patterns without random components, resources that need to be replaced will be deleted before creating the new resource. This can lead to downtime.

Custom Naming Patterns

Create your own naming patterns that combine static text, resource information, and random elements:

pulumi:autonaming:
 pattern: ${project}-${stack}-${name}${alphanum(6)}

See the auto-naming configuration documentation to see the full list of available expressions.

See It In Action

Let’s look at a practical example. Say you’re creating an S3 bucket and a DynamoDB table in your Pulumi program:

import * as aws from "@pulumi/aws";

// Create an S3 bucket
const bucket = new aws.s3.Bucket("uploads");

// Create a DynamoDB table
const table = new aws.dynamodb.Table("users", {
 hashKey: "id",
 attributes: [{
 name: "id",
 type: "S",
 }],
 billingMode: "PAY_PER_REQUEST",
});

import pulumi_aws as aws

# Create an S3 bucket
bucket = aws.s3.Bucket("uploads")

# Create a DynamoDB table
table = aws.dynamodb.Table("users",
 hash_key="id",
 attributes=[aws.dynamodb.TableAttributeArgs(
 name="id",
 type="S"
 )],
 billing_mode="PAY_PER_REQUEST"
)

using Pulumi;
using Pulumi.Aws.S3;
using Pulumi.Aws.DynamoDB;
using Pulumi.Aws.DynamoDB.Inputs;

return await Deployment.RunAsync(() =>
{
 // Create an S3 bucket
 var bucket = new Bucket("uploads");

 // Create a DynamoDB table
 var table = new Table("users", new TableArgs
 {
 HashKey = "id",
 Attributes = new[]
 {
 new TableAttributeArgs
 {
 Name = "id",
 Type = "S"
 }
 },
 BillingMode = "PAY_PER_REQUEST"
 });
});

package main

import (
 "github.com/pulumi/pulumi-aws/sdk/v5/go/aws/dynamodb"
 "github.com/pulumi/pulumi-aws/sdk/v5/go/aws/s3"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // Create an S3 bucket
 bucket, err := s3.NewBucket(ctx, "uploads", nil)
 if err != nil {
 return err
 }

 // Create a DynamoDB table
 table, err := dynamodb.NewTable(ctx, "users", &dynamodb.TableArgs{
 HashKey: pulumi.String("id"),
 Attributes: dynamodb.TableAttributeArray{
 &dynamodb.TableAttributeArgs{
 Name: pulumi.String("id"),
 Type: pulumi.String("S"),
 },
 },
 BillingMode: pulumi.String("PAY_PER_REQUEST"),
 })
 if err != nil {
 return err
 }

 return nil
 })
}

package myproject;

import com.pulumi.Pulumi;
import com.pulumi.aws.s3.Bucket;
import com.pulumi.aws.dynamodb.Table;
import com.pulumi.aws.dynamodb.TableArgs;
import com.pulumi.aws.dynamodb.inputs.TableAttributeArgs;

public class App {
 public static void main(String[] args) {
 Pulumi.run(ctx -> {
 // Create an S3 bucket
 var bucket = new Bucket("uploads");

 // Create a DynamoDB table
 var table = new Table("users", TableArgs.builder()
 .hashKey("id")
 .attributes(TableAttributeArgs.builder()
 .name("id")
 .type("S")
 .build())
 .billingMode("PAY_PER_REQUEST")
 .build());
 });
 }
}

resources:
 # Create an S3 bucket
 uploads:
 type: aws:s3:Bucket

 # Create a DynamoDB table
 users:
 type: aws:dynamodb:Table
 properties:
 hashKey: id
 attributes:
 - name: id
 type: S
 billingMode: PAY_PER_REQUEST

By default, Pulumi would generate names like uploads-ae26f3b and users-4c2dd09. But let’s say you want your resources to follow a pattern that includes your project and stack name. You can configure this in your stack configuration file (Pulumi.<stack-name>.yaml):

config:
 pulumi:autonaming:
 pattern: ${project}-${stack}-${name}

Now when you run pulumi up, your resources will be created with predictable names:

S3 bucket: myproject-dev-uploads
DynamoDB table: myproject-dev-users

You can also set different patterns for specific providers or resource types:

config:
 pulumi:autonaming:
 pattern: ${project}-${stack}-${name}
 providers:
 aws:
 resources:
 "aws:s3/bucket:Bucket":
 pattern: ${name}-${stack}-${alphanum(6)}

With this configuration, you’ll get:

S3 bucket: uploads-dev-x7yz9n (with a random suffix for global uniqueness)
DynamoDB table: myproject-dev-users (following the default pattern)

Configuration Syntax

The configuration syntax differs slightly depending on where you define it:

In your project file Pulumi.yaml:

config:
 pulumi:autonaming:
 value:
 mode: verbatim

In your stack configuration file Pulumi.<stack-name>.yaml:

config:
 pulumi:autonaming:
 mode: verbatim

The same applies to other configuration patterns shown above - use the value: key in project-level configuration, but omit it in stack-level configuration.

Getting Started

To use the auto-naming configuration feature, you’ll need:

Pulumi CLI 3.146.0 or later
The following minimum provider versions (as applicable):
- AWS provider 6.66.0 or later
- Azure Native provider 2.78.0 or later
- Azure Classic provider 6.14.0 or later
- Google Cloud Platform provider 8.11.0 or later
- Kubernetes provider 4.20.0 or later
- AWS Cloud Control provider 1.21.0 or later

Once you have the required versions installed, simply add your desired auto-naming configuration to your Pulumi configuration file.

For complete documentation and advanced usage scenarios, visit our resource auto-naming documentation.

General Availability

We’re excited to announce that the auto-naming feature is now generally available across our major cloud providers. This release marks an important milestone in Pulumi’s evolution, delivering a robust and flexible solution for resource naming.

Thank you to everyone who participated in the RFC discussion and the preview period and for providing valuable feedback. Your input has been invaluable in creating a solution that works for diverse use cases while maintaining Pulumi’s core strengths.

If you have any questions or feedback about the resource auto-naming feature, please don’t hesitate to reach out to us on GitHub or in the Pulumi Community Slack.

Pulumi + Azure Deployment Environments: Better Together for Enterprise Developers

Mikhail Shilkov — Tue, 21 May 2024 00:00:00 +0000

We are excited to announce the support for authoring Azure Deployment Environments (ADE) environment definitions in Pulumi Infrastructure as Code (IaC) empowering developers to self-serve app infrastructure required to deploy and test cloud-based applications. With Pulumi support, you can now manage your Azure resources in these environments using the same familiar programming model and the full power of our IaC platform.

Azure Deployment Environments is a service that enables developers to quickly spin up app infrastructure with project-based templates, all while maintaining centralized management and governance. Azure Deployment Environments also provides developers with an intuitive self-service portal where they can choose a curated, project-specific template to deploy new environments with just a few clicks. Thanks to the ADE extensibility model, you can now turn any Pulumi program into an environment definition and use it directly to securely provision application infrastructure in Azure. This post will guide you through creating ADE templates and deploying resources from the Microsoft developer portal.

How It Works

Azure Deployment Environments natively supports the Azure Resource Manager (ARM) and Bicep IaC frameworks. In addition, ADE provides an extensibility model that enables customers to create custom container images to author environment definitions.

The Azure Deployment Environments and Pulumi teams collaborated to publish a standard container image that our joint customers can use directly to run Pulumi programs in ADE: pulumi/azure-deployment-environments. The image leverages ADE’s extensibility model to run Pulumi programs in the container.

When used with Pulumi, each environment definition consists of two components:

An environment.yaml file that describes the environment by referencing the Pulumi Docker image and pointing to a Pulumi program. Here is an example of such a file:

name: MyPulumiEnvironment
version: 1.0.0
summary: My First Pulumi-Enabled Environment
description: Deploys a Pulumi stack into ADE
runner: pulumi/azure-deployment-environments:latest
templatePath: Pulumi.yaml

A Pulumi program that defines the resources to deploy. This program can use any of the Pulumi Azure providers to define resources in the environment. Here is a very simple Pulumi program defined in YAML:

name: ade-pulumi
runtime: yaml
description: My first Pulumi program in ADE
config:
resource-group-name:
 type: string
resources:
sa:
 type: azure-native:storage:StorageAccount
 properties:
 kind: Storage
 resourceGroupName: ${resource-group-name}
 sku:
 name: Standard_LRS

This is all you need to define a Pulumi environment in ADE. If you prefer to use a programming language like C#, TypeScript, Python, or Go, you will also need to add the code next to the Pulumi.yaml file.

Once the environment definition is added to the catalog, a user simply chooses an applicable environment in their developer portal. After accepting parameter values, ADE will run the Pulumi program in the container to provision Azure resources. In the case of the above program, it will create a new Storage Account in the environment, and the user will be able to use the resources once they are provisioned by Pulumi.

Getting Started

To get started, you must have the Azure Deployment Environments service configured. You can manually create one in the Azure Portal by following the instructions or using our sample Pulumi program to provision all the required resources.

Once you have an ADE instance, you can start authoring your environment definitions. You can use the Pulumi Azure Deployment Environments samples as a starting point. The folder contains four simple Pulumi programs in four different languages, each creating a Storage Account in the environment. The programs also show how to use parameters to integrate with ADE’s configuration system.

Once you have the definitions, you will publish them to a GitHub or Azure DevOps repository and attach the repository as a Catalog in Azure Deployment Environments. You can do so manually using the Azure Portal or configure it within your Pulumi project, as illustrated in our sample.

Finally, navigate to the Microsoft developer portal to create a new environment. You will see the list of available templates, including the ones you just published.

Bring a Custom Container Image

The Pulumi container image pulumi/azure-deployment-environments is a great starting point for your environment definitions. However, if you want to customize the behavior to your needs, you can also bring your own container image to run Pulumi programs in ADE. To do so, refer to the Configure a container image to execute deployments with Pulumi article on Microsoft Learn.

One benefit of using a custom image is the ability to provide a Pulumi Cloud access token to the image via the PULUMI_ACCESS_TOKEN environment variable. This allows the image to authenticate with the Pulumi Cloud backend without requiring additional configuration. Pulumi Cloud securely encrypts and stores your infrastructure state, manages secrets, provides search and clear visibility across all your cloud resources, runs remote deployments, integrates with CI/CD pipelines, detects configuration drift, and enables centralized policy enforcement. Additional features such as RBAC and audit logging enable your team to collaborate easily, ship faster, more securely and with confidence for every deployment.

If you have a suggestion for improving the standard Pulumi container image, please open an issue on our GitHub repository. We are always looking for ways to improve the experience for our Azure users.

Conclusion

Together with the Azure Deployment Environments team, Pulumi is excited to enable our shared customers to take advantage of the ADE extensibility model, Pulumi IaC platform, and empower developers with self-service of Azure App infrastructure, within enterprise guardrails. We hope that this new capability will help you streamline your development processes and make it easier for your teams to provision new environments in Azure quickly.

If you want to learn more:

Attend our workshop Platform Engineering with Microsoft Azure and Pulumi that takes place on June 20
Read the announcement on the Azure blog
Explore the code at pulumi/azure-deployment-environments

Announcing 6.0 of the Pulumi AWS Provider

Mikhail Shilkov — Tue, 29 Aug 2023 09:00:00 -0400

We are excited to announce 6.0 of the Pulumi AWS provider. The AWS provider is the most heavily used provider across the entire Pulumi ecosystem, and offers access to the full surface area of the upstream Terraform AWS Provider in Pulumi projects in all supported languages. The 6.0 release brings a substantial set of fixes and improvements to the provider, including a number of breaking changes as part of the major version release.

This blog post walks you through the list of notable changes in the new major version.

Join Pulumi on Thursday, September 7, 2023 for a live workshop on getting started with infrastructure as code on AWS, which will feature the new AWS 6.0 provider.

Terraform Plugin Framework Support

More and more Terraform resources are migrating to the Terraform Plugin Framework. We are excited to announce that 6.0 supports all resources implemented with both the new Terraform Plugin SDK and the legacy Terraform Plugin Framework technologies. This means that you get access to the entire API of the upstream provider.

New Resources and Functions

Since the previous major release of 5.0, we have shipped 57 additional minor and patch releases, enabling you to keep up with the latest and greatest enhancements. The new major version of the AWS provider brings 56 new resources and 23 new functions. Additions include resources for Audit Manager, Chime, Fin Space, Global Accelerator, Open Search, Quicksight, Resource Explorer, Route53, Transfer, and many other AWS services.

You can see a full list of new resources and functions in Release Notes.

SDK Size Optimization

The AWS provider contains more than 1,200 resources with over 12,000 properties between them. As a result, it’s expected that the SDK size of the provider has grown to be fairly large. With that in mind, we invested in some key areas to reduce the size of generated files. Depending on the runtime, the SDK is now up to 90% smaller than it used to be in v5 of the provider!

One particular optimization is related to wafv2 resources, that were particularly difficult to use in Pulumi AWS v5 (issue one, two, three). The new release includes a refactored wafv2 module which uses recursive types. Using recursive types allowed us to remove a lot of excess generated code.

Smaller SDK sizes mean faster downloads, snappier experience with editors and code completion tools, less time needed for pulumi preview and pulumi up operations, and smaller binaries, all of which contribute to a streamlined developer experience for Pulumi users.

Upstream Breaking Changes

Pulumi AWS 6.0 ships all improvements and bug fixes of upstream versions from 5.0.0 to 5.11.0. It also contains a number of upstream breaking changes. Please refer to the changelog to navigate the entire list.

Fixes for Highly Upvoted Issues

The new release ships fixes for many highly upvoted issues along with Pulumi-specific enhancements. This includes resolving constant diff from wafv2 with no changes and unexpected behavior of defaultTags that have been fixed and are available in the new release.

MaxItemsOne and Deprecations

The 6.0 release removes several deprecated resources. All of those resources were replaced with newer versions containing a more logical organization of modules and resource names. The removed resources have been deprecated for years, so it’s unlikely that you were still using them. But in case you do, we applied automatic aliases to make migration as straightforward as possible.

We also shipped a few, small breaking changes to properties that may allow one or many items. We aligned those with the capabilities of the upstream provider and the AWS services ensure Pulumi users can access the full capabilities of the AWS platform.

Migration Guide

You can see a full list of changes and learn more about migrating your existing programs in our Migration Guide.

Get Started Today

You can browse our API reference docs with inline examples or explore the Pulumi AWS Provider repository to get started today! And don’t forget to register for the live workshop on September 7, 2023 to see AWS 6.0 in action!

Pulumi Release Notes: Resource Search, Deployment Actions, Projects in Self-Managed Backends, and more

Mikhail Shilkov — Wed, 24 May 2023 00:00:00 +0000

We have been busy shipping improvements in the last 2 months. Let’s walk through the release highlights across Pulumi engineering areas from March and April. If you want to learn more between release blogs, follow the CLI improvements in the pulumi/pulumi repo changelog and Pulumi Cloud features in the new features blogs.

Quick shortcuts to jump to your favorite area:

Pulumi CLI
Pulumi Automation API
- Manage Stack Tags
- Full “Who Am I” command support
Cloud Providers and Packages
Pulumi Cloud & Pulumi.com

Pulumi CLI

Aligning Projects between Service and Self-Managed Backends

While the majority of Pulumi users do choose to use Pulumi Cloud, we also know that there are good reasons why some organizations would prefer to use Pulumi IaC alone without Pulumi Cloud. The self-managed backends have historically stored all stacks in a single namespace across all projects.

With the 3.61.0 release of Pulumi, we’ve added support for project-scoped stacks to the Self-Managed backend. Stacks created in new or empty self-managed backends will automatically use the new project-scoped layout.

👉 Learn more in the Aligning Projects between Service and Self-Managed Backends blog post.

Improved Interactive Scrolling Experience

We’ve improved the user experience of deploying a large number of resources interactively in the CLI. The recent changes make two improvements to scrolling through a long list of action:

The interactive display can now be scrolled using the page-down and page-up keys. These keys scroll the display N lines at a time, where N is the height of the terminal.
If the display is scrolled to the bottom, it will now autoscroll it as new lines are added to the output.

👉 Learn more in the Improve interactive scrolling pull request.

Shortcut to See the Current Update in Web Browser

While a CLI preview or update operation is running, you can now use the Ctrl-O keyboard shortcut to open the same update in Pulumi Cloud using your web browser.

The shortcut and the Pulumi Cloud URL are shown at the top of the update operation:

$ pulumi up
Previewing update (dev)
View in Browser (Ctrl+O): https://app.pulumi.com/mikhailshilkov/dev-ts/dev/previews/01234567-ec03-4ccc-9b44-02ec864e14ee
Type Name Plan
+ pulumi:pulumi:Stack ts-dev create...

👉 Learn more in the Open in Web Browser shortcut pull request.

New Flag to Specify Teams for a Newly Created Stack

We’ve added a --teams flag to pulumi stack init command. The flag accepts a string and can be provided multiple times. Each team that is provided is assigned read/write permissions on the stack after it has been initialized.

$ pulumi stack init dev-stack --teams devs

👉 Learn more in the Add –teams flag to assign team name to stack pull request.

Visual Indication of Retained Resources

You can use the RetainOnDelete resource option to let Pulumi retain the actual cloud resource while deleting it from your Pulumi stack. This option is useful, for instance, when you need to start managing the same resource from another stack or project.

Now, the CLI will explicitly mark deleted or replaced resources as retained in the cloud. See the Plan column of the preview view:

👉 Learn more in the Retain display pull request.

View More Details in Stack History

Sometimes, to troubleshoot a past action, you would want to see more details about the context of a past CLI operation. We’ve added more details to pulumi stack history command output, so that you could see the names of CLI parameters, the names of environment variables, the version of the Pulumi CLI, and OS architecture:

No sensitive information like parameter or environment variable values is captured or displayed.

👉 Learn more in the Add more details to pulumi stack history pull request.

Pulumi Automation API

Manage Stack Tags

We’ve added support to set, get, list, and remove tags on Pulumi stacks. Shoutout to Matthew Rodrigues for contributions in this area!

👉 Learn more in the Ensure Parity for Stack Tags issue.

Full “Who Am I” command support

The pulumi whoami CLI command now added a --json flag. The same flag is consumed from the Automation API to return user’s organization and URL in addition to the user name.

Another shoutout to Matthew Rodrigues for the contribution!

👉 Learn more in the WhoAmI JSON flag pull request.

Cloud Providers and Packages

Every week we keep shipping provider updates. A few highlights on our native providers:

Kubernetes shipped support for v1.27.0 and all its resources (changelog)
Azure Native shipped six new resources for containerstorage, guestconfiguration, and hybridcloud (releases)
AWS Native shipped 33 new resources in apigatewayv2, appconfig, chatbot, comprehend, connect, datasync, devopsguru, ec2, frauddetector, lambda, msk, neptune, quicksight, ram, sagemaker, ssmcontacts, and vpclattice modules (changelog)

Two new community packages were added to the Registry:

AzAPI - shoutout to Engin Diri
StrongDM - shoutout to Piers Karsenbarg

Pulumi Cloud & Pulumi.com

Resource Search Improvements

More Advanced Filtering

Based on customer feedback we have extended the number of advanced filtering options (Stack, Project, Package and Type) to show the top 10 results by resource count instead of top 3. We know our customers are getting a ton of value from these aggregates to inform and refine their searches and expect the 7 additional values to improve this further.

Pulumi Teams support

Customers who use Pulumi Teams for role based access control on Pulumi Cloud can now filter resources by team access. We have added a new aggregate on the Resource Search page for Teams, allowing you to see at a glance the teams in your organization with the most resources. You can query which teams have access to what resources, track resource usage by team by exporting your resource data and see in the search results table which teams have access to that resource.

Events Dashboard card

We have added a new card to the organizations dashboard which shows upcoming Pulumi events and Pulumi workshops to give users a central place to see events of interest.

Stacks Page Icons

We have added icons on the Stacks page to enable you to see at a glance information about your Pulumi Stacks, such as if Policies are being applied, if Pulumi Deployments is configured for that stack and the resource count. In addition, there is now a quick action menu on each stack to jump to configuring Pulumi Deployments and adding it to a Policy.

Deployment Actions

You can now view the Pulumi Deployments Click-to-Deploy menu on all stack pages, even if not configured to Pulumi Deployments. This allows users to quickly navigate to Deployments Settings to connect the stack.

SAML Admin

For customers on the Enterprise or Business Critical Editions of Pulumi Cloud there is a new role for a SAML admin in your organization. SAML Admins can add another identity to authenticate when SAML is the only permitting authentication method. This could be an email and password or another authentication method, like GitHub. We are slowly rolling this out across our customer base, if it is not yet in your organization you should expect it in the coming weeks.

Pulumi Release Notes: Colorized Stack Traces, Output Methods for JSON, OIDC integration, and much more!

Mikhail Shilkov — Wed, 22 Mar 2023 00:00:00 +0000

We have been busy shipping improvements in the last 2 months. Let’s walk through the release highlights across Pulumi engineering areas from January and February. If you want to learn more between release blogs, follow the CLI improvements in the pulumi/pulumi repo changelog and Pulumi Service features in the new features blogs.

Quick shortcuts to jump to your favorite area:

Pulumi CLI
Pulumi Language SDKs
Pulumi Automation API
- Adding tags to stacks through Automation API
- Support cloning from Azure DevOps in Automation API
Cloud Providers and Packages
Pulumi Service & Pulumi.com
- OIDC integration
- Pulumi Deployments

Pulumi CLI

Colorized stack traces for Node.js errors

When an error happens in a Pulumi program, stack traces can be instrumental to understand the reasons behind the failure. Stack traces can often be large and contain frames unrelated to user code, which makes debugging harder.

We have now changed the CLI to display Node.js stack traces with colors, where the important bits are highlighted and less important parts faded out. The new display technique helps draw attention to relevant details and is similar to running vanilla node programs.

👉 Learn more in the Emit colorized stack traces pull request.

New –shell option for `pulumi stack output`

We added a new --shell option to the pulumi stack output command that writes the outputs for a stack as a shell script:

% pulumi stack output --shell
bucketName=mybucket-1234
websiteURL="http://mybucket-1234.example.com"

On Windows, it will produce a Powershell script:

% pulumi stack output --shell
$bucketName = 'mybucket-1234'
$websiteURL = 'http://mybucket-1234.example.com'

As a result, Pulumi users are now able to write shell scripts like the one below:

eval "$(pulumi stack output --shell)"
echo "Created bucket $bucketName"
# => Created bucket mybucket-1234

echo "Visit at $websiteURL"
# => Visit at http://mybucket-1234.example.com

👉 Learn more in the Add –shell option pull request.

`pulumi new` in directories with source control

By default, pulumi new won’t run unless the current directory is empty to avoid overriding any existing files. This behavior meant that the command failed in directories with initialized source control.

We shipped an improvement where Pulumi CLI will ignore .git, .hg, and .bzr files and directories when considering whether a directory is empty.

👉 Learn more in the Ignore VCS directories pull request.

Support plugins hosted in GitLab

Pulumi comes with a wealth of packages but users can also create and use custom providers and components. By default, Pulumi expects the binaries to be hosted on GitHub. On top of that, the CLI now supports GitLab as a first-class source of package downloads.

If your package binary is hosted on GitLab, you can now set its pluginDownloadURL property (see Publish your package) to point to your GitLab project, e.g. to gitlab://gitlab.com/43429536.

👉 Learn more in the Support custom plugins from other sources issue.

Use access tokens to authenticate to Google Cloud backend

When storing the state files in Google Cloud backend, you can now authenticate to the Google Cloud Storage using Google access tokens. If the environment variable GOOGLE_OAUTH_ACCESS_TOKEN is set during a pulumi login gs://..., Pulumi will utilize the provided token to login to the bucket, assuming this has not been preempted by another authentication type.

👉 Learn more in the Pulumi login to support google oauth access tokens for GCS buckets pull request.

Delete temporary files from plugin downloads

While a plugin is being installed, the CLI downloads the necessary archive and stores the archive in /tmp/pulumi-plugin-tar*. The CLI is now more mindful of cleaning the temporary file after plugin installation completes to save disk space.

👉 Learn more in the Delete temporary files from plugin downloads pull request.

“Yes, using Update Plans” prompt removed

Update Plans is a preview feature to constrain the update only to the operations that were planned during preview. To get extra feedback from our users, we experimented with showing an extra option “yes, using Update Plans” at the update prompt. The experiment is now complete, so we removed the option.

Lean more in the Remove the “yes, using Update Plans” prompt issue.

Pulumi Language SDKs

Output methods to serialize and deserialize JSON

Building JSON from values and parsing strings into JSON are two of the most common operations in Pulumi programs. All programming languages have built-in libraries to do so. However, quite often JSON operations need to be combined with Pulumi Inputs and Outputs, which required writing a cumbersome combination of Apply and serialization method calls.

To streamline this experience, Pulumi SDKs shipped helper serialization and deserialization functions that understand the type Output<T> natively. Here is a simple usage example in Python, where table.arn is an output coming from another resource:

policy = iam.Policy(
 "iam-policy",
 policy= Output.json_dumps({
 "Version": "2012-10-17",
 "Statement": [{
 "Action": [
 "dynamodb:PutItem",
 "dynamodb:GetItem"
 ],
 "Effect": "Allow",
 "Resource": [table.arn]
 }]
 }))

👉 Learn more in the Output methods issue that also tracks a number of further additions to the toolset of helper methods.

Retrieve stack reference outputs as plain values

You can read outputs from one stack in another stack’s program using Stack References. In the past, the resulting values were always wrapped inside an Output<T> container in order to preserve secret values (when needed).

With recent SDKs, we shipped an alternative method StackReference.getOutputDetails. This method returns a plain object with two fields: value and secretValue. At most one of these fields is set, depending on whether the stack reference output is a secret or not.

If you know that the output is not a secret, you can write a program to use the plain value immediately, for example, to iterate though a list in a loop and create resources in each iteration.

👉 Learn more for each language:

Native arm64 binaries in Python SDK

Pulumi Python SDK has been upgraded to use the grpcio library version 1.51.3, which comes with a wheel with native arm64 binaries (universal2). The change eases the installation of the Pulumi Python SDK on Apple M1/M2 machines.

👉 Learn more in the Python grpc 1.51.3 contains universal2 binaries pull request.

Go version raised to 1.18

The minimum version of Go required by Pulumi was raised to 1.18. Raising this directive allows us to use language features like generics in the codbase. The change is applied to both pkg and sdk.

👉 Learn more in the go.mod: Raise to ‘go 1.18 pull request.

Delegate alias computation to the engine

Sometimes, two classes in Pulumi SDKs represent the same cloud resource. To enable migration from one to another, Pulumi sets aliases for those classes and declares them to be compatible.

In the past, each SDK was responsible for calculating all possible alias combinations for a given resource and its inherited parent. We recently added support in the engine for this computation along with an optimization to reduce complexity from a multiplicative factor to a linear one.

As a follow-up step, we updated our SDKs to only provide the base aliases, and allow the engine to do this computation. This enables smaller SDKs and faster runtime performance, especially for longer chains of parented resources. Other than that, the user experience hasn’t changed.

👉 Learn more in the Delegate alias computation from SDKs to Engine issue.

Pulumi Automation API

Adding tags to stacks through Automation API

You can now use Automation API to get, set, list, or remove tags on stacks programmatically. Here is a quick example in TypeScript:

const workspace = await LocalWorkspace.create({
 projectSettings: projectSettings,
});
await workspace.createStack(stackName);
await workspace.setTag(stackName, "foo", "bar");
const tags = await workspace.listTags(stackName);
// tags will contain tags for `pulumi:project`, `pulumi:runtime`, and `foo` keys.

👉 Learn more in the Add support for adding tags to stacks through Automation API issue.

Support cloning from Azure DevOps in Automation API

We implemented a workaround to support Azure DevOps repositories when driving Pulumi from Automation API in Go. This is now also utilized in the Pulumi Kubernetes Operator.

👉 Learn more in the Support clones from Azure DevOps pull request.

Cloud Providers and Packages

Every week we keep shipping provider updates. Two highlights on our native providers:

Azure Native shipped 20 new resources, including support for networkcloud and voiceservices (changelog)
AWS Native shipped 30 new resources in apigatewayv2, appflow, applicationautoscaling, cloudtrail, connect, directoryservice, ec2, fms, gamecast, gamelift, kendraranking, networkmanager, omics, organizations, sagemaker, secretsmanager, simspaceweaver, and systemsmanagersap modules (changelog)

Five new community packages were added to the Registry: Vultr, Zscaler Private Access (ZPA), Zscaler Internet Access (ZIA), Statuscake, and Nuage.

Pulumi Service & Pulumi.com

After launching Pulumi Deployments a few months ago and receiving a ton of community feedback (thank you!) we launched an integration with OpenID Connect (OIDC) to enable security first deployments and industry best practices.

OpenID Connect (OIDC) is an identity layer built on top of the OAuth 2.0 framework. It allows third-party applications to verify the identity of the end-user and to obtain basic user profile information. OIDC uses JSON web tokens (JWTs), which you can obtain using flows conforming to the OAuth 2.0 specifications.

Overview

The main benefits of using the OIDC integration for Pulumi Deployments are as follows:

Temporary credentials: You can move away from long term credentials within your organization to leverage temporary credentials that are generated every time a deployment runs. The Pulumi Service issues a new OIDC token that is limited to that deployment. This token is used to obtain cloud credentials that automatically expire, meaning you do not need to store sensitive credentials that are reused with each deployment.
Granular access control: The OIDC token generated by the Pulumi Service contains detailed information about the deployment with which it is associated. In AWS for example, you can configure our OIDC integration to only use temporary credentials if that token is associated with a particular Pulumi stack or Pulumi organization. This enables you to follow the principle of least privilege access and scope access down to only what is needed. For AWS, Azure, and GCP, Pulumi Deployments can automatically exchange its OIDC token for temporary credentials and make those credentials available to providers. For advanced scenarios or other cloud providers, the OIDC token can be manually exchanged for credentials by the user.

👉 Learn more on how to get started with Pulumi Deployments OIDC by reading the blog post.

If you haven’t yet tried out Pulumi Deployments

You should now see a “Deploy” tab under Stack Settings in the Pulumi Service console and have access to the Deployments REST API. View the documentation for help getting started and join our Slack (#pulumi-deployments channel) for questions and feedback.

New CLI prompt to use Update Plans

Fraser Waters — Wed, 23 Nov 2022 06:00:00 -0700

Earlier this year we announced the experimental introduction of Update Plans as we heard from many of you that you need a strong guarantee about exactly which changes an update will make to your infrastructure, especially in critical and production environments. We have been making steady progress on this feature and are excited to further integrate it into your workflows. In the latest release of the Pulumi CLI (v3.48.0), there’s a new prompt to use experimental Update Plans when running an update.

Why use Update Plans

Update Plans help catch any unexpected changes that might happen between when you preview a change and when you apply that change. Update Plans work by saving the results of a pulumi preview to a plan file, which enables you to restrict subsequent pulumi up operations to only the actions saved in the plan file. This helps you ensure that what you saw in the pulumi preview is what will actually happen when you run pulumi up.

With Update Plans, this same workflow is now possible with infrastructure managed by Pulumi. Assuming you already have CI set up for your pull/merge requests, you can run pulumi preview --save-plan PLAN-FILENAME and output the resulting plan file as a CI artifact. Then, update the CI workflow that runs when changes are merged to run pulumi up --plan PLAN-FILENAME.

Until now to leverage Update Plans users had to pass --save-plan <file> and --plan <file> options while using non-interactive CLI commands preview and up respectively with the PULUMI_EXPERIMENTAL=true environment variable. This restriction limited the feature discoverability, while enabling us to scale gradually. The --save-plan <file> and --plan <file> options continue to be available and still require a PULUMI_EXPERIMENTAL=true environment variable.

New option in CLI interactive prompt

We believe that Update Plans have matured to be evaluated by our larger community. To make it more visible, we are introducing a new option to the update confirmation prompt. If you run a pulumi up command interactively, you will now see the following choice:

Do you want to perform this update? [Use arrows to move, type to filter]
> [experimental] yes, using Update Plans (https://pulumi.com/updateplans)
 yes
 no
 details

The no option is still the default choice, and the yes option is still immediately above it, so the change will hopefully align with user habits. However, there is a top option added, which suggests running an update with a plan built during the preview.

Each interactive preview now calculates an Update Plan. If you choose the yes, using Update Plans option, the Update Plan will be automatically passed to the update command and applied during the deployment. Any inconsistencies between the preview and the update will be reported back to the user.

Try Update Plans

Update Plans are in public preview. We recommend using it only in non-critical, non-production scenarios during the preview period.

We’re eager for you to try the public preview of Update Plans and let us know what you think. To try out the new interactive option, make sure you’ve updated to Pulumi 3.48.0 or higher.

We’d love to hear your thoughts on the Update Plans feature! Feel free to ask questions in our Slack or open an issue on GitHub.

Infrastructure as Code with Java and Pulumi

Mikhail Shilkov — Wed, 04 May 2022 06:59:00 -0700

Infrastructure has become a core part of application development as modern cloud capabilities such as microservices, containers, serverless, and data stores define your application’s architecture. The term “infrastructure” covers all of the cloud resources your application needs to run. Modern architectures require thinking deeply about infrastructure while building your application, instead of treating it as an afterthought. Pulumi’s approach helps developers, infrastructure engineers, and platform teams work together to leverage everything the modern cloud has to offer.

Pulumi has worked with hundreds of companies to get cloud applications into production, and Java has quickly risen to become one of the most frequently requested features by the community. Pulumi is an open source product, and we are grateful to our awesome community members who bootstrapped Pulumi for Java last year and were instrumental in helping us with this public preview. Thank you to Paweł Prażak and his VirtusLab colleagues!

Pulumi supports Java for all of your modern infrastructure as code needs, this means you can build, deploy, and manage your infrastructure on any cloud—including all of AWS, Azure, Google Cloud, Kubernetes, Oracle Cloud, and more—using Java and other JVM languages. With Pulumi, you will have the entire cloud at your fingertips without ever having to leave your code editor, while using production-ready infrastructure as code techniques.

What is Pulumi?

Pulumi lets you build, deploy, and manage infrastructure on any cloud using general-purpose programming languages (TypeScript, Python, Go, .NET, Java) and markup languages (YAML, CUE) to express your application’s infrastructure needs, using a powerful technique called “infrastructure as code.” You declare desired infrastructure, and an engine provisions it for you, so that it’s automated, easy to replicate, and robust enough for demanding production requirements. Pulumi takes this approach a step further by leveraging programming languages and software engineering tools to make modern cloud infrastructure patterns, such as containers and serverless programs, easy and first-class citizens.

With Pulumi for Java you can:

Declare infrastructure using programs, classes, and libraries written in Java or other JVM languages (Kotlin, Scala, Clojure, Groovy, etc.).
Automatically create, update, or delete cloud resources using Pulumi’s infrastructure as code engine, removing manual point-and-clicking in web consoles and ad-hoc scripts.
Use your favorite IDEs and tools, including IntelliJ IDEA and Visual Studio Code, taking advantage of features like auto-completion, refactoring, and interactive documentation.
Catch mistakes early on with standard compiler errors, analyzers, and an infrastructure-specific policy engine for enforcing security, compliance, and best practices.
Reuse any existing Java package, or distribute your own, whether that’s for infrastructure best practices, productivity, or just general programming patterns.
Deploy continuously, predictably, and reliably using GitHub Actions, or one of over a dozen CI integrations.
Build scalable cloud applications using cloud native technologies like Kubernetes, Docker containers, serverless functions, and PaaS services into your core development experience, bringing them closer to your application code.

Pulumi’s free open source SDK, which includes a CLI and an assortment of libraries, enables these capabilities.

Example: Provision a GKE cluster with a Kubernetes namespace

The following Java snippet demonstrates the power of Pulumi for Java (full source code). The program defines a Google Kubernetes Engine cluster, calculates its kubeconfig and exports it for user’s needs, and deploys a Kubernetes namespace into the newly provisioned cluster.

package gke_sample;

import java.util.*;
import java.io.*;
import java.nio.*;
import com.pulumi.*;
import com.pulumi.gcp.container.*;
import com.pulumi.kubernetes.core_v1.*;

public class Program {
 private static void stack(Context ctx) {
 // Create a GKE cluster
 var cluster = new Cluster("mygke",
 ClusterArgs.builder()
 .initialNodeCount(1)
 .minMasterVersion("1.20.7")
 .build()
 );

 // Build and export a Kubeconfig for the newly created cluster.
 var kubeconfig = Utils.buildKubeconfig(cluster);
 ctx.export("kubeconfig", kubeconfig);

 // Create a Kubernetes provider instance that uses our cluster from above.
 var clusterProvider = new Provider("gke-provider",
 ProviderArgs.builder().kubeconfig(kubeconfig).build());

 // Create a Kubernetes Namespace
 var ns = new Namespace("test",
 NamespaceArgs.Empty,
 CustomResourceOptions.builder().provider(clusterProvider).build()
 );
 }

 public static void main(String[] args) {
 Pulumi.run(App::stack);
 }
}

Resources are defined declaratively using class constructors and argument builders. Dependencies between resources are managed automatically by the Pulumi engine based on the way you use variables in the program. You are free to use any libraries, helper functions—for instance, to build the kubeconfig string above, classes, if statements, for loops, and all the other tools available to Java developers.

Why is Java great for infrastructure too?

Many of us love using Java to author our applications, so why not use it for infrastructure as code too? By using Java, you get many helpful features for your infrastructure code:

Familiarity: No need to learn DSLs or markup templating languages.
Expressiveness: Use loops, conditionals, pattern matching, async code, and more, to dynamically create infrastructure that meets the target environment’s needs.
Abstraction: Encapsulate common patterns into classes and functions to hide complexity and avoid copy-and-pasting the same boilerplate repeatedly.
Sharing and reuse: Tap into a community of cloud applications and infrastructure experts by sharing and reusing Maven or Gradle packages with your team or the global community.
Productivity: Use your favorite IDE and get statement completion, go to definition, live error checking, refactoring, static analysis, and interactive documentation.
Project organization: Use common code structuring techniques to manage your infrastructure across one or more projects.
Application lifecycle: Use existing ALM systems and techniques to manage and deploy your infrastructure projects, including source control, code review, testing, and continuous integration (CI) and delivery (CD).

Pulumi unlocks access to the entire JVM ecosystem—something that’s easy to take for granted but is missing from other solutions based on DSLs or CLI scripts. This approach also helps developers and operators work better together using a shared foundation. Add all of the above together, and you get things done faster and more reliably.

Join the community and get started

The first preview of Pulumi for Java includes support for the entire breadth of services in AWS, Azure, Google Cloud, and more. Give Pulumi a try, visit the Pulumi for Java docs.

There you will find several instructions on installing and getting started with Pulumi for Java. The following resources provide additional useful information:

Although Pulumi for Java is listed in “preview” status, it supports all of the most essential Pulumi programming model features (and the rest is on its way). Our goal is to gather feedback over the next few weeks, and we will be working hard to improve the Java experience across the board, including more examples and better documentation.

Pulumi is open source on GitHub and you can find the Java plugin at pulumi/pulumi-java. And you’re welcome to join the community in Slack to discuss your scenarios, ideas, and to get any needed assistance from the team and other end users.

We look forward to seeing the new and amazing cloud applications you build with Pulumi for Java!

Exploring how to solve circular dependencies

Daniel Bradley — Wed, 12 Jan 2022 00:00:00 +0000

This blog post originated from a hackathon and exploratory exercise, and should be considered in that context. None of the solutions exist in the Pulumi SDKs. For more information on this, please view our blog post on Circular Dependencies.

As part of our hackathon near the end of last year, we decided to explore solutions to a common problem when people are using Pulumi for their systems. A question that’s been asked in a few different forms is how to resolve circular dependencies between resources in a Pulumi program.

A simple example of this idea is a modern web application with a static front-end and an API, where the front-end needs to know the URL of the API to be able to call it and the API needs to know the source domain of the front-end to allow it access via CORS. As these two resources rely on one another to be created, they are circular dependencies.

For a deployable example, let’s build a serverless game of ping-pong—two functions that invoke one another until one misses!

const ping = new CallbackFunction("ping", {
 callback: pingPongHandler,
 environment: {
 variables: {
 OPPONENT_FN_NAME: pong.name, // <-- Can't access `pong` yet!
 },
 },
});

const pong = new CallbackFunction("pong", {
 callback: pingPongHandler,
 environment: {
 variables: {
 OPPONENT_FN_NAME: ping.name,
 },
 },
});

Programming languages don’t let us create one object that takes a value from another object that doesn’t yet exist. Therefore, we can’t express a circular dependency in this way.

The simplest workaround for this error is to get a reference to our own stack, which contains the outputs from the previous deployment. Now, we set the reference to pong within ping from the output of the last deployment.

const currentStack = new pulumi.StackReference(
 `acme-org/${pulumi.getProject()}/${pulumi.getStack()}`
);

const ping = new CallbackFunction("ping", {
 callback: pingPongHandler,
 environment: {
 variables: {
 OPPONENT_FN_NAME: currentStack
 .getOutput("pongName")
 .apply((pongName) => pongName ?? ""),
 },
 },
});

const pong = new CallbackFunction("pong", {
 callback: pingPongHandler,
 environment: {
 variables: {
 OPPONENT_FN_NAME: ping.name,
 },
 },
});

export const pingName = ping.name;
export const pongName = pong.name;

The downside of this approach is that we have to deploy twice to reach the desired state because we need the results of the previous deployment, though this approach might be fine for simple one-off circumstances.

How then could we make the process easier and clearer for users who need to resolve this kind of circular dependency regularly? We explored four options as part of the hackathon, so let’s go through each one, getting progressively closer to a possible solution.

Option 1: State

Instead of referencing outputs directly from our stack’s previous deploy and manually deploying twice, we could introduce a special type called “State”.

The State object does 3 tasks:

Read the value from the previous deployment, or provide a default if it’s the first deployment.
Update the value in the stack state using a “set” method.
Automatically triggers another deployment if its value has changed.

const pongName = new pulumi.State("pong-name", {
 initialValue: "",
});

const ping = new CallbackFunction("ping", {
 callback: pingPongHandler,
 environment: {
 variables: {
 OPPONENT_FN_NAME: pongName.value,
 },
 },
});

const pong = new CallbackFunction("pong", {
 callback: pingPongHandler,
 environment: {
 variables: {
 OPPONENT_FN_NAME: ping.name,
 },
 },
});

// Engine re-runs "up" if this changes the value
pongName.set(pong.name);

The first deployment will match the preview, but when triggering the re-deployment, we’re making a change that wasn’t in the first preview. This action could also result in a never-ending loop where every deployment makes another change to the state and triggers another deployment. So let’s explore a different option that might not have this issue.

Option 2: Refs

This option looks almost identical to the “state” concept above, except we’re calling them references (refs):

const pongName = new pulumi.Ref("pong-name", {
 initialValue: "",
});

const ping = new CallbackFunction("ping", {
 callback: pingPongHandler,
 environment: {
 variables: {
 OPPONENT_FN_NAME: pongName.value, // Special type the Pulumi engine will track
 },
 },
});

const pong = new CallbackFunction("pong", {
 callback: pingPongHandler,
 environment: {
 variables: {
 OPPONENT_FN_NAME: ping.name,
 },
 },
});

// Determine affected resources and re-update
pongName.set(pong.name);

The difference between refs and state is that, instead of triggering a whole re-deploy when the value changes, refs would be integrated into the dependency graph similarly to resources. When the value of the ref changes, we could internally work out which resources depend on that ref and change the target state for only those resources. This avoids the possibility of a never-ending update loop.

Both the “state” and “refs” approaches have the downside of not making the final outputs of the updated resource available to the rest of the program. The resource object would still have the values from after it first deployed. This means that other parts of the program might still deploy differently on subsequent deployments. Let’s explore another option that might address that problem.

Option 3: Create patch resources

We could explicitly model the third step by using a new resource type suffixed with “patch”:

const ping = new CallbackFunction("ping", {
 callback: pingPongHandler,
});

const pong = new CallbackFunction("pong", {
 callback: pingPongHandler,
 environment: {
 variables: {
 OPPONENT_FN_NAME: ping.name,
 },
 },
});

const pingPatch = new CallbackFunctionPatch("ping-patch", {
 id: ping.id,
 environment: {
 variables: {
 OPPONENT_FN_NAME: pong.name,
 },
 },
});

The provider presents pseudo-resources for update or patch operations. The patch resource is using the same underlying resource as the original resource. This option is a very flexible approach that could be implemented by providers right now. These patch resources could also be used for resources that exist outside the Pulumi stack. The drawback is that the patch operation has to be implemented in each provider individually, and it significantly increases the overall size of each provider and its associated SDK. We’re still evaluating whether this option would be a good one considering that size increase, which increases download times as an example impact, and the engineering resources to implement the change. We have one more option to consider, as well.

Option 4: Allow a resource to update its goal state

When using the SDK, we could write something like the following code to create then update a little later on:

const ping = new CallbackFunction("ping", {
 callback: pingPongHandler,
});

const pong = new CallbackFunction("pong", {
 callback: pingPongHandler,
 environment: {
 variables: {
 OPPONENT_FN_NAME: ping.name,
 },
 },
});

const updatedPing = ping.update({
 environment: {
 variables: {
 OPPONENT_FN_NAME: pong.name,
 },
 },
});

When declaring a resource in a Pulumi program, you’re essentially saying, “This is the state I want my resource to be in.” The update method returns a whole new copy of the resource with new properties containing the outputs of the update.

What’s next?

The final option is quite close to a fully working solution, but there still remains a few tasks to complete before we could release it as a feature:

Finalise what the internal messages between the Pulumi’s internal components should look like.
Keep a record of circular dependencies in the stack state without causing knock-on issues elsewhere in Pulumi’s internal workings.
Decide what should happen if an update triggers a replacement—if this is potentially valid behaviour or not. We should be able to generate the set of properties which are able to be updated without causing replacement and provide these properties to the user as the arguments for the update method to avoid these issues before execution of the program.

Some other ideas that would be interesting to explore in the future are as follows:

How could we utilise patching of resources where a provider can support it?
Could we delete a resource that was created earlier in the program to allow the creation of temporary resources?

This hackathon project has opened up a number of new pathways for us to explore. If you want to participate, we always welcome contributions in our open-source repos at https://github.com/pulumi. We hope you have enjoyed this dive into how our team explored this concept as part of the hackathon. We’ll be exploring more of the hackathon projects soon on the blog!

Deploy Azure Container Apps with Docker and Pulumi

Mikhail Shilkov — Tue, 02 Nov 2021 09:00:00 -0700

Today, Microsoft announced a new general-purpose serverless container platform: Azure Container Apps. Container Apps is a fully managed platform for microservice applications that runs on top of Kubernetes and open-source technologies like KEDA, Envoy, and Dapr.

Container Apps are designed to abstract infrastructure management with flexible serverless containers. Developers can run containers at scale without the burden of standing up and managing a Kubernetes cluster manually.

We are happy to announce same-day support for Azure Container Apps in the Pulumi Azure Native Provider, which covers 100% of the Azure Resource Manager APIs and gives you highest fidelity integration with Azure’s resources.

Azure Container Apps QuickStart

Ready to get up and running quickly right away?

Bootstrap a project $ pulumi new https://github.com/pulumi/apps/container-apps
Add your container logic to the app folder
Deploy with $ pulumi up
Test with $ curl $(pulumi stack output url)

For additional information on how Azure Container Apps work and advanced options, please read on.

The new service supports a broad range of usage scenarios, including

Microservices over HTTP or gRPC
HTTP APIs and websites
Event processing workers
Long-running background jobs

Serverless containers

Container Apps provide serverless containers for microservice developers.

Managed infrastructure. Microsoft operates the control plane that takes care of orchestrating containers and their configuration, allowing developers to focus on apps, not cloud infrastructure.
Fully integrated auto-scaling with scale to zero. The platform relies on Kubernetes Event-driven Autoscaling (KEDA) to scale apps and microservices dynamically based on HTTP traffic or event workload.
Consumption pricing. The billing model is based on the actual resource consumption with per-second granularity. Applications incur charges per request, compute time and memory used with no need to pre-provision a fixed capacity.
Any language, any base image. Container Apps put no limitation on the container images. You may use an arbitrary base image and host any web server or a console application, ensuring flexibility and interoperability with other container orchestrators.

How it works

Microsoft has built Container Apps as a managed service on the foundation of open-source technology in the Kubernetes ecosystem. This enables teams to build microservices without having to manage Kubernetes directly while providing application portability by leveraging open standards and APIs. Behind the scenes, every container app runs on the Azure Kubernetes Service (AKS). Enabling open-source integrations include:

Envoy, a built-in ingress controller that exposes user containers internally and externally via HTTP endpoints. It supports multiple container versions with dynamic load balancing and several traffic rollout strategies.

KEDA enables dynamic auto-scaling for user applications based on the current workload, including scaling down to zero during idle periods.

Container Apps are also tightly integrated with Dapr—an open-source runtime system designed to support cloud-native microservice applications. It provides extra blocks for service discovery, state management, asynchronous message passing.

How it’s different

A few other services provided by Microsoft Azure and other cloud providers operate in the space of running container-based workloads:

Azure Kubernetes Service delivers the full power of Kubernetes but requires expertise in configuring and operating the cluster. When building on AKS, users handle most infrastructure management aspects themselves. Instead, Container Apps provide a platform built on top of AKS, focusing on developer productivity and switching to consumption-based pricing.

Azure Container Instances (ACI) provides atomic infrastructure units with per-usage pricing. However, ACI comes without higher-level functionality like auto-scaling, load balancing, versioning, and managed rollouts.

Azure App Service is a Platform-as-a-Service comparable to Container Apps in terms of being simple-to-operate and developer-friendly. App Service can also run arbitrary containers. However, it is best suited to run websites. The billing model is mostly capacity-based with some built-in autoscaling but without scaling to zero.

Azure Functions is a developer-oriented truly-serverless offering. However, it comes with an opinionated programming model that is focused on achieving a high developer productivity as long as your application can use the Azure Functions SDK or container base images. Unlike Container APps, it does not support long-running jobs in consumption mode.

Comparing to other vendors: Azure Container Apps are in the same space as Google Cloud Run and AWS App Runner. In contrast to the competition, Azure Container Apps is built on Kubernetes and related open-source projects, which should benefit its users in terms of interoperability and transparency. Additionally, Container Apps is the only service that provides features like service discovery for microservices-style communication out of the box.

Example: Run an HTTP API with Azure Container Apps and Pulumi

Let’s walk through the steps to build an example application with Azure Container Apps using infrastructure as code in familiar languages. In this scenario, we create an HTTP application that is available via a public domain name. We’ll use Pulumi to provision the necessary resources. In this example, we will use TypeScript however you could also use JavaScript, Python, Go, and C#.

You can check out the complete source code in the Pulumi Examples:

Define a Dockerfile and app

Here are the key features of the container image for our application:

Based on the node:8.9.3-alpine image
Installs express with NPM
Runs a node web app using index.js and index.html user files

Full Dockerfile.

Set up the environment

The resource KubeEnvironment defines a cluster that can host multiple Container Apps. Behind the scenes, it creates an AKS cluster in a subscription managed internally by Microsoft and deploys the Apps control plane.

import * as web from "@pulumi/azure-native/web/v20210301";

const env = new web.KubeEnvironment("env", {
 resourceGroupName: resourceGroup.name,
 type: "Managed",
});

using Pulumi.AzureNative.Web.V20210301;

var kubeEnv = new KubeEnvironment("env", new KubeEnvironmentArgs
{
 ResourceGroupName = resourceGroup.Name,
 Type = "Managed"
});

import pulumi_azure_native.web.v20210301 as web

kube_env = web.KubeEnvironment("env",
 resource_group_name=resource_group.name,
 type="Managed")

import web "github.com/pulumi/pulumi-azure-native/sdk/go/azure/web/v20210301"

kubeEnvironment, err := web.NewKubeEnvironment(ctx, "kubeEnvironment", &web.KubeEnvironmentArgs{
 ResourceGroupName: resourceGroup.Name,
 Type: pulumi.String("Managed"),
})

Build and publish a container image

We can build the Docker image and publish it to a new Azure Container Registry (ACR) repository. The code below assumes

import * as docker from "@pulumi/docker";
import * as containerregistry from "@pulumi/azure-native/containerregistry";

const customImage = "node-app";
const registry = new containerregistry.Registry("registry", {
 resourceGroupName: resourceGroup.name,
 sku: {
 name: "Basic",
 },
 adminUserEnabled: true,
});

const credentials = containerregistry.listRegistryCredentialsOutput({
 resourceGroupName: resourceGroup.name,
 registryName: registry.name,
});
const adminUsername = credentials.apply(c => c.username!);
const adminPassword = credentials.apply(c => c.passwords![0].value!);

const myImage = new docker.Image(customImage, {
 imageName: pulumi.interpolate`${registry.loginServer}/${customImage}:v1.0.0`,
 build: { context: `./${customImage}` },
 registry: {
 server: registry.loginServer,
 username: adminUsername,
 password: adminPassword,
 },
});

using Pulumi.AzureNative.ContainerRegistry;
using Pulumi.Docker;

var registry = new Registry("registry", new RegistryArgs
{
 ResourceGroupName = resourceGroup.Name,
 Sku = new SkuArgs { Name = "Basic" },
 AdminUserEnabled = true
});

var credentials = Output.Tuple(resourceGroup.Name, registry.Name)
 .Apply(items =>
 ListRegistryCredentials.InvokeAsync(new ListRegistryCredentialsArgs
 {
 ResourceGroupName = items.Item1,
 RegistryName = items.Item2
 }));
var adminUsername = credentials.Apply(c => c.Username);
var adminPassword = credentials.Apply(c => c.Passwords[0].Value);

var customImage = "node-app";
var myImage = new Image(customImage, new ImageArgs
{
 ImageName = Output.Format($"{registry.LoginServer}/{customImage}:v1.0.0"),
 Build = new DockerBuild { Context = $"./{customImage}" },
 Registry = new ImageRegistry
 {
 Server = registry.LoginServer,
 Username = adminUsername,
 Password = adminPassword
 }
});

from pulumi_azure_native import containerregistry
import pulumi_docker as docker

registry = containerregistry.Registry("registry",
 resource_group_name=resource_group.name,
 sku=containerregistry.SkuArgs(name="Basic"),
 admin_user_enabled=True)

credentials = pulumi.Output.all(resource_group.name, registry.name).apply(
 lambda args: containerregistry.list_registry_credentials(
 resource_group_name=args[0],
 registry_name=args[1]))
admin_username = credentials.username
admin_password = credentials.passwords[0]["value"]

custom_image = "node-app"
my_image = docker.Image(custom_image,
 image_name=registry.login_server.apply(
 lambda login_server: f"{login_server}/{custom_image}:v1.0.0"),
 build=docker.DockerBuild(context=f"./{custom_image}"),
 registry=docker.ImageRegistry(
 server=registry.login_server,
 username=admin_username,
 password=admin_password))

import (
 "github.com/pulumi/pulumi-azure-native/sdk/go/azure/containerregistry"
 "github.com/pulumi/pulumi-docker/sdk/v3/go/docker"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

registry, err := containerregistry.NewRegistry(ctx, "registry", &containerregistry.RegistryArgs{
 ResourceGroupName: resourceGroup.Name,
 Sku: containerregistry.SkuArgs{
 Name: pulumi.String("Basic"),
 },
 AdminUserEnabled: pulumi.Bool(true),
})
if err != nil {
 return err
}
credentials := pulumi.All(resourceGroup.Name, registry.Name).ApplyT(
 func(args []interface{}) (*containerregistry.ListRegistryCredentialsResult, error) {
 resourceGroupName := args[0].(string)
 registryName := args[1].(string)
 return containerregistry.ListRegistryCredentials(ctx, &containerregistry.ListRegistryCredentialsArgs{
 ResourceGroupName: resourceGroupName,
 RegistryName: registryName,
 })
 },
)

adminUsername := credentials.ApplyT(func(result interface{}) (string, error) {
 credentials := result.(*containerregistry.ListRegistryCredentialsResult)
 return *credentials.Username, nil
}).(pulumi.StringOutput)
adminPassword := credentials.ApplyT(func(result interface{}) (string, error) {
 credentials := result.(*containerregistry.ListRegistryCredentialsResult)
 return *credentials.Passwords[0].Value, nil
}).(pulumi.StringOutput)

newImage, err := docker.NewImage(ctx, "node-app", &docker.ImageArgs{
 ImageName: pulumi.Sprintf("https://%s/node-app:v1.0.0", registry.LoginServer),
 Build: docker.DockerBuildArgs{
 Context: pulumi.String("/node-app"),
 },
 Registry: docker.ImageRegistryArgs{
 Server: registry.LoginServer,
 Username: adminUsername,
 Password: adminPassword,
 },
})
if err != nil {
 return err
}

Ship containers to Azure with Pulumi

Use the Azure Native provider to build a Docker image, publish it to Azure Container Registry, and deploy a serverless Container App from one program. Provision and manage it all in the language you already use.

Get started

Deploy the container app

Finally, we can define the Container App itself. We point the App to the environment resource and instruct it to run our custom image. Image container credentials are specified in the configuration block, with the password marked as a secret. We’ve also enabled external ingress to publish the app on the web.

const containerApp = new web.ContainerApp("app", {
 resourceGroupName: resourceGroup.name,
 kubeEnvironmentId: env.id,
 configuration: {
 ingress: {
 external: true,
 targetPort: 80,
 },
 registries: [{
 server: registry.loginServer,
 username: adminUsername,
 passwordSecretRef: "pwd",
 }],
 secrets: [{
 name: "pwd",
 value: adminPassword,
 }],
 },
 template: {
 containers: [{
 name: "myapp",
 image: myImage.imageName,
 }],
 }
});

export const url = pulumi.interpolate`https://${containerApp.configuration.ingress.fqdn}`;

var containerApp = new ContainerApp("app", new ContainerAppArgs
{
 ResourceGroupName = resourceGroup.Name,
 KubeEnvironmentId = kubeEnv.Id,
 Configuration = new ConfigurationArgs
 {
 Ingress = new IngressArgs
 {
 External = true,
 TargetPort = 80
 },
 Registries = {
 new RegistryCredentialsArgs
 {
 Server = registry.LoginServer,
 Username = adminUsername,
 PasswordSecretRef = "pwd"
 }
 },
 Secrets = {
 new SecretArgs
 {
 Name = "pwd",
 Value = adminPassword
 }
 },
 },
 Template = new TemplateArgs
 {
 Containers = {
 new ContainerArgs
 {
 Name = "myapp",
 Image = myImage.ImageName
 }
 }
 }
});

this.Url = Output.Format($"https://{containerApp.Configuration.Apply(c => c.Ingress).Apply(i => i.Fqdn)}");

container_app = web.ContainerApp("app",
 resource_group_name=resource_group.name,
 kube_environment_id=kube_env.id,
 configuration=web.ConfigurationArgs(
 ingress=web.IngressArgs(
 external=True,
 target_port=80),
 registries=[web.RegistryCredentialsArgs(
 server=registry.login_server,
 username=admin_username,
 password_secret_ref="pwd")],
 secrets=[web.SecretArgs(
 name="pwd",
 value=admin_password)],
 ),
 template=web.TemplateArgs(
 containers=[web.ContainerArgs(
 name="myapp",
 image=my_image.image_name)]))

pulumi.export("url", container_app.configuration.apply(lambda c: c.ingress).apply(lambda i: i.fqdn))

containerApp, err := web.NewContainerApp(ctx, "app", &web.ContainerAppArgs{
 ResourceGroupName: resourceGroup.Name,
 KubeEnvironmentId: kubeEnvironment.ID(),
 Configuration: web.ConfigurationArgs{
 Ingress: web.IngressArgs{
 External: pulumi.Bool(true),
 TargetPort: pulumi.IntPtr(80),
 },
 Registries: web.RegistryCredentialsArray{
 web.RegistryCredentialsArgs{
 Server: registry.LoginServer,
 Username: adminUsername,
 PasswordSecretRef: pulumi.String("pwd")},
 },
 Secrets: web.SecretArray{
 web.SecretArgs{
 Name: pulumi.String("pwd"),
 Value: adminPassword,
 },
 },
 },
 Template: web.TemplateArgs{
 Containers: web.ContainerArray{
 web.ContainerArgs{
 Name: pulumi.String("myapp"),
 Image: newImage.ImageName,
 },
 },
 },
})
if err != nil {
 return err
}

ctx.Export("url", pulumi.Sprintf("https://%s", containerApp.LatestRevisionFqdn))

Test the app

And that is it! We run pulumi up to get the application up and running. Once the deployment completes, we can send an HTTP request and check the response.

$ curl $(pulumi stack output url)
<html>
<body>
<h1>Your custom docker image is running in Azure Container Apps!</h1>
</body>
</html>

Conclusion

Azure Container Apps enable you to build applications in your favorite language with any dependencies and tools, package them as a container image, and deploy them in seconds. Container Apps abstract away infrastructure management by automatically scaling up and down to zero and only charging for the exact resources you use.

This post shows how to use Pulumi to build a container image and publish it as an Azure Container App. Pulumi makes it easy to create artifacts and provision and manage cloud infrastructure on any cloud using familiar programming languages, including C#, TypeScript, Python, and Go. Docker images, ACR registries, container environments, and Apps can be managed within the same infrastructure definition.

Further steps

Complete Azure Container Apps example

Full Coverage of Azure Resources with Azure-Native

Mikhail Shilkov — Tue, 02 Mar 2021 00:00:00 +0000

Last September, we announced the beta release of Pulumi Azure NextGen: a new Microsoft Azure provider for Pulumi that combines same-day access to the entire Azure API surface with the excellent Pulumi experience you know and love, including version-less resources, auto-naming, and auto-location.

Today, we’re excited to announce that this new provider is now the default way to manage Azure resources with Pulumi. We’re also excited to announce its final name: the native Azure provider for Pulumi, or “Azure-Native” for short. You can get started with the new provider using our newly-updated getting started guide.

Already use the classic Pulumi-Azure provider? You can migrate to Azure-Native now using the migration guide. If you prefer to wait until general availability, stay tuned for updates: we expect to reach general availability in the spring of this year.

Alongside this announcement, we’re also formalizing a new concept: a Pulumi “native provider”. A native provider provides functionality mapped directly from the underlying API; in the case of the new Azure Provider, functionality is mapped directly from the Azure Resource Manager API surface. Going forward, you’ll be able to find these next-generation Pulumi providers directly from the name by looking for the “-Native” suffix.

Native to Azure

The native Azure provider implements the best possible support for the Azure platform in Pulumi. It exposes the entire API surface of Azure to developers and operators, now and forever. As of today, the native provider supports 1,010 resource types, 120 of them added since the initial preview in September 2020. Every property of each resource is always represented in the SDKs.

Unlike the classic Azure provider, which requires manual work to keep updated, we designed the native provider to stay always up-to-date with Azure API additions and changes. We generate Pulumi SDKs for the native Azure provider automatically from Azure API specifications published by Microsoft. We publish daily builds of the provider and have published 210 versions of the provider in the last six months.

Native to Pulumi

The native Azure provider comes with several significant improvements over the Azure NextGen preview. These changes bring more features of the Pulumi experience to the native Azure provider.

Top-level Resources

Every resource is now available in the top-level (versionless) module of its resource provider. You don’t have to choose the API version explicitly: we will apply the default for you.

import * as azure from "@pulumi/azure-native";

const resourceGroup = new azure.resources.ResourceGroup("resourceGroup");

const storageAccount = new azure.storage.StorageAccount("sa", {
 resourceGroupName: resourceGroup.name,
 sku: {
 name: azure.storage.SkuName.Standard_LRS,
 },
 kind: azure.storage.Kind.StorageV2,
});

import pulumi_azure_native as azure

resource_group = azure.resources.ResourceGroup('resource_group')

account = azure.storage.StorageAccount('sa',
 resource_group_name=resource_group.name,
 sku=storage.SkuArgs(
 name=azure.storage.SkuName.STANDARD_LRS,
 ),
 kind=azure.storage.Kind.STORAGE_V2)

using Pulumi;
using Pulumi.AzureNative.Resources;
using Pulumi.AzureNative.Storage;
using Pulumi.AzureNative.Storage.Inputs;

class MyStack : Stack
{
 public MyStack()
 {
 var resourceGroup = new ResourceGroup("resourceGroup");

 var storageAccount = new StorageAccount("sa", new StorageAccountArgs
 {
 ResourceGroupName = resourceGroup.Name,
 Sku = new SkuArgs
 {
 Name = SkuName.Standard_LRS
 },
 Kind = Kind.StorageV2
 });
 }
}

package main

import (
 "github.com/pulumi/pulumi-azure-native/sdk/go/azure/resources"
 "github.com/pulumi/pulumi-azure-native/sdk/go/azure/storage"
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 resourceGroup, err := resources.NewResourceGroup(ctx, "resourceGroup", nil)
 if err != nil {
 return err
 }

 account, err := storage.NewStorageAccount(ctx, "sa", &storage.StorageAccountArgs{
 ResourceGroupName: resourceGroup.Name,
 Sku: &storage.SkuArgs{
 Name: storage.SkuName_Standard_LRS,
 },
 Kind: storage.KindStorageV2,
 })

 return err
 })
}

The top-level resources will use stable API versions and take no arbitrary breaking changes within a given major version of the package once we reach general availability; during the preview, breaking changes may appear in 0.x minor versions.

We still provide access to every version of every ARM API if you want to use the latest version or a specific version from the past. Each API version is mapped to its own module or namespace within the resource provider.

Auto-Naming

The original preview of the native Azure provider (then called “Azure NextGen”) didn’t yet support auto-naming, a common feature of other Pulumi providers. Now, it provides full support for auto-naming. Azure resources will get a random suffix in the name by default. In the following example, Pulumi will create a resource group called, for instance, rg53acf911.

import * as azure from "@pulumi/azure-native";
const resourceGroup = new azure.resources.ResourceGroup("rg");

import pulumi_azure_native as azure
resource_group = azure.resources.ResourceGroup('rg')

using Pulumi;
using Pulumi.AzureNative.Resources;
class MyStack : Stack
{
 public MyStack()
 {
 var resourceGroup = new ResourceGroup("rg");
 }
}

package main

import (
 resources "github.com/pulumi/pulumi-azure-native/sdk/go/azure/resources"
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 _, err := resources.NewResourceGroup(ctx, "resourceGroup", nil)
 return err
 })
}

Pass an explicit value to the resourceGroupName property if you want to control the resource’s exact name.

However, suppose you create a sub-resource, e.g., a container under a storage account, a database under a database account, or a slot under a web app. In that case, Pulumi will not append a random suffix because those resources names don’t have to be globally unique.

You can also apply transformations to define your own naming schemas: see this example from our community.

Auto-Location

Like the classic Azure provider, all resources’ location property is now optional in the native Azure provider. In the examples above, we did not have to specify the location of resources. Instead, we can run the following command to configure it for the whole stack:

pulumi config set azure-native:location WestUS2

You can also specify the location explicitly for a resource group but leave it out for resources under that resource group: the resources will deploy to the same region by default. Of course, you are free to override the defaults at any time.

Beyond ARM API

Unfortunately, some Azure capabilities aren’t yet available via the Azure Resource Manager APIs used by the native Azure provider. To ensure that you can still use some of the missing capabilities, we’ve introduced several resources that go beyond what’s currently possible with ARM (and ARM templates):

Exposing Azure Storage as a static website.
Uploading files and directories as Azure Storage Blobs.
Managing Azure KeyVault secrets and keys.

By combining full support for the ARM APIs with these additional extensions, Pulumi’s native Azure provider provides the broadest support for managing Azure resources with infrastructure-as-code.

Open-Source

The native Azure provider for Pulumi is fully open-source. The code is available on GitHub at pulumi-azure-native.

Migrate from Azure NextGen to Azure-Native

Migration from Azure NextGen to Azure-Native is simple and non-disruptive:

Reference the pulumi-azure-native package from your program.
Update all import/using statements from nextgen to native.
Run pulumi up and make sure the program ran successfully. The preview should show no changes.
If so, choose yes to run the update that will overwrite the state to the native provider.

Note that the Azure resources are not affected by this migration.

Getting Started

If you are new to Pulumi, follow our Get Started guide.

From now on, the azure-* templates will reference the native Azure provider. It’s easier to get started than ever:

$ pulumi new azure-typescript

$ pulumi new azure-python

$ pulumi new azure-csharp

$ pulumi new azure-go

Several more extensive examples are available in the Pulumi Examples repo:

Web Applications with Azure App Service and Docker: TypeScript, C#, Python, Go
Azure AKS cluster: TypeScript, C#, Python, Go
Web Application with Azure Container Instances: TypeScript, C#, Python, Go
Web Server Using Azure Virtual Machine: TypeScript, Python

You can browse API reference docs with inline examples or explore the Pulumi Azure-Native SDKs repository.

Get Up and Running with Azure Synapse and Pulumi

Mikhail Shilkov — Fri, 04 Dec 2020 00:00:00 +0000

Azure Synapse is an integrated analytics service that combines enterprise data warehousing of Azure SQL Data Warehouse and Big Data analytics of Apache Spark. Azure Synapse is a managed service well integrated with other Azure services for data ingestion and business analytics.

You could use the Azure portal to get started with Azure Synapse, but it can be hard to define sophisticated infrastructure for your analytics pipeline using the portal alone, and many users need to apply version control to their cloud configurations.

The alternative is to use an infrastructure as code tool to automate building and deploying cloud resources. This article demonstrates how to provision an Azure Synapse workspace using Pulumi and general-purpose programming languages like Python and C#.

Azure Synapse Quick Start

Ready to get up and running quickly right away?

Bootstrap a project $ pulumi new https://github.com/pulumi/examples/tree/master/azure-py-synapse.
Follow the steps outlined in README.
Go to https://web.azuresynapse.net and sign in to your new workspace.

For additional information on how provisioning Azure Synapse with infrastructure as code, please read on.

Azure Synapse Components

Let’s start by introducing the components required to provision a basic Azure Synapse workspace. To follow along with the Synapse Getting Started Guide, you need the following key Azure infrastructure components:

Resource Group to contain all other resources.
Storage Account to store input data and analytics artifacts.
Azure Synapse Workspace—a collaboration boundary for cloud-based analytics in Azure.
SQL Pool—a dedicated Synapse SQL pool to run T-SQL based analytics.
Spark Pool to use Apache Spark analytics.
IP Filters and Role Assignments for secure access control.

Infrastructure as Code

Let’s walk through the steps to build a workspace with all the components mentioned above. We’ll use Pulumi to provision the necessary resources. Feel free to pick the language of your choice that will apply to all code snippets.

You can check out the full source code in the Pulumi Examples.

Resource Group

Let’s start by defining a resource group to contain all other resources. Be sure to adjust its name and region to your preferred values.

Choose the language below to adjust the contents of this blog. Your choice is applied throughout the article.

resource_group = resources.ResourceGroup("resourceGroup",
 resource_group_name="synapse-rg",
 location="westus2")

var resourceGroup = new ResourceGroup("resourceGroup", new ResourceGroupArgs
{
 ResourceGroupName = "synapse-rg",
 Location = "westus2
});

const resourceGroup = new resources.ResourceGroup('resourceGroup', {
 resourceGroupName: 'synapse-rg',
 location: 'westus2',
});

Data Lake Storage Account

Synapse workspace will store data in a data lake storage account. We use a Standard Read-Access Geo-Redundant Storage account (SKU Standard_RAGRS) for this purpose. Make sure to change the accountName to your own globally unique name.

storage_account = storage.StorageAccount("storageAccount",
 resource_group_name=resource_group.name,
 location=resource_group.location,
 account_name="yoursynapsesa",
 access_tier="Hot",
 enable_https_traffic_only=True,
 is_hns_enabled=True,
 kind="StorageV2",
 sku=storage.SkuArgs(
 name="Standard_RAGRS",
 ))

var storageAccount = new StorageAccount("storageAccount", new StorageAccountArgs
{
 ResourceGroupName = resourceGroup.Name,
 Location = resourceGroup.Location,
 AccountName = "synapsesa",
 AccessTier = "Hot",
 EnableHttpsTrafficOnly = true,
 IsHnsEnabled = true,
 Kind = "StorageV2",
 Sku = new SkuArgs
 {
 Name = "Standard_RAGRS"
 },
});

const storageAccount = new storage.StorageAccount("storageAccount", {
 resourceGroupName: resourceGroup.name,
 location: resourceGroup.location,
 accountName: "yoursynapsesa",
 accessTier: "Hot",
 enableHttpsTrafficOnly: true,
 isHnsEnabled: true,
 kind: "StorageV2",
 sku: {
 name: "Standard_RAGRS",
 },
});

Let’s build a data lake URL for this storage account.

data_lake_storage_account_url = storage_account.name.apply(lambda name: f"https://{name}.dfs.core.windows.net")

var dataLakeStorageAccountUrl = Output.Format($"https://{storageAccount.Name}.dfs.core.windows.net");

const dataLakeStorageAccountUrl = pulumi.interpolate`https://${storageAccount.name}.dfs.core.windows.net`;

We’ll use the users blob container as the analytics file system.

users = storage.BlobContainer("users",
 resource_group_name=resource_group.name,
 account_name=storage_account.name,
 container_name="users",
 public_access="None")

var users = new BlobContainer("users", new BlobContainerArgs
{
 ResourceGroupName = resourceGroup.Name,
 AccountName = storageAccount.Name,
 ContainerName = "users",
 PublicAccess = "None"
});

new storage.BlobContainer("users", {
 resourceGroupName: resourceGroup.name,
 accountName: storageAccount.name,
 containerName: "users",
 publicAccess: "None",
});

Synapse Workspace

It’s time to use all of the above to provision an Azure Synapse workspace! Adjust the name and the SQL credentials in the definition below.

workspace = synapse.Workspace("workspace",
 resource_group_name=resource_group.name,
 location=resource_group.location,
 workspace_name="mysynapse",
 default_data_lake_storage=synapse.DataLakeStorageAccountDetailsArgs(
 account_url=data_lake_storage_account_url,
 filesystem="users",
 ),
 identity=synapse.ManagedIdentityArgs(
 type="SystemAssigned",
 ),
 sql_administrator_login="sqladminuser",
 sql_administrator_login_password=random.RandomPassword("workspacePwd", length=12).result)

var workspace = new Workspace("workspace", new WorkspaceArgs
{
 ResourceGroupName = resourceGroup.Name,
 Location = resourceGroup.Location,
 WorkspaceName = "my-workspace",
 DefaultDataLakeStorage = new DataLakeStorageAccountDetailsArgs
 {
 AccountUrl = dataLakeStorageAccountUrl,
 Filesystem = "users"
 },
 Identity = new ManagedIdentityArgs
 {
 Type = "SystemAssigned"
 },
 SqlAdministratorLogin = "sqladminuser",
 SqlAdministratorLoginPassword = "YourStrongPassword"
});

const workspace = new synapse.Workspace("workspace", {
 resourceGroupName: resourceGroup.name,
 location: resourceGroup.location,
 workspaceName: "mysynapse",
 defaultDataLakeStorage: {
 accountUrl: dataLakeStorageAccountUrl,
 filesystem: "users",
 },
 identity: {
 type: "SystemAssigned",
 },
 sqlAdministratorLogin: "sqladminuser",
 sqlAdministratorLoginPassword: "YourSecurePassword",
});

Note that we also defined a system-assigned managed identity for the workspace.

Security Setup

You need to allow access to the workspace with a firewall rule. The following is a blank access rule but feel free to restrict it to your target IP range.

allow_all = synapse.IpFirewallRule("allowAll",
 resource_group_name=resource_group.name,
 workspace_name=workspace.name,
 rule_name="allowAll",
 end_ip_address="255.255.255.255",
 start_ip_address="0.0.0.0")

var allowAll = new IpFirewallRule("allowAll", new IpFirewallRuleArgs
{
 ResourceGroupName = resourceGroup.Name,
 WorkspaceName = workspace.Name,
 RuleName = "allowAll",
 EndIpAddress = "255.255.255.255",
 StartIpAddress = "0.0.0.0"
});

new synapse.IpFirewallRule("allowAll", {
 resourceGroupName: resourceGroup.name,
 workspaceName: workspace.name,
 ruleName: "allowAll",
 endIpAddress: "255.255.255.255",
 startIpAddress: "0.0.0.0",
});

The following snippet assigns the Storage Blob Data Contributor role to the workspace managed identity and your target user. If you use the Azure CLI, run az ad signed-in-user show --query=objectId to look up your user ID.

subscription_id = resource_group.id.apply(lambda id: id.split('/')[2])
role_definition_id = subscription_id.apply(lambda id: f"/subscriptions/{id}/providers/Microsoft.Authorization/roleDefinitions/ba92f5b4-2d11-453d-a403-e96b0029c9fe")

authorization.RoleAssignment("storageAccess",
 role_assignment_name=random.RandomUuid("roleName").result,
 scope=storage_account.id,
 principal_id=workspace.identity.principal_id.apply(lambda v: v or "<preview>"),
 principal_type="ServicePrincipal",
 role_definition_id=role_definition_id)

authorization.RoleAssignment("userAccess",
 role_assignment_name=random.RandomUuid("userRoleName").result,
 scope=storage_account.id,
 principal_id=config.get("userObjectId"),
 principal_type="User",
 role_definition_id=role_definition_id)

var subscriptionId = resourceGroup.Id.Apply(id => id.Split('/')[2]);
var roleDefinitionId = $"/subscriptions/{subscriptionId}/providers/Microsoft.Authorization/roleDefinitions/ba92f5b4-2d11-453d-a403-e96b0029c9fe";

new RoleAssignment("storageAccess", new RoleAssignmentArgs
{
 RoleAssignmentName = new RandomUuid("roleName").Result,
 Scope = storageAccount.Id,
 PrincipalId = workspace.Identity.Apply(identity => identity.PrincipalId).Apply(v => v ?? "<preview>"),
 PrincipalType = "ServicePrincipal",
 RoleDefinitionId = roleDefinitionId
});
new RoleAssignment("userAccess", new RoleAssignmentArgs
{
 RoleAssignmentName = new RandomUuid("userRoleName").Result,
 Scope = storageAccount.Id,
 PrincipalId = config.Require("userObjectId"),
 PrincipalType = "User",
 RoleDefinitionId = roleDefinitionId
});

const subscriptionId = resourceGroup.id.apply(id => id.split('/')[2]);
const roleDefinitionId = pulumi.interpolate`/subscriptions/${subscriptionId}/providers/Microsoft.Authorization/roleDefinitions/ba92f5b4-2d11-453d-a403-e96b0029c9fe`;

new authorization.RoleAssignment("storageAccess", {
 roleAssignmentName: new random.RandomUuid("roleName").result,
 scope: storageAccount.id,
 principalId: workspace.identity.principalId,
 principalType: "ServicePrincipal",
 roleDefinitionId: roleDefinitionId,
});

new authorization.RoleAssignment("userAccess", {
 roleAssignmentName: new random.RandomUuid("userRoleName").result,
 scope: storageAccount.id,
 principalId: "your-synapse-user-object-uuid",
 principalType: "User",
 roleDefinitionId: roleDefinitionId,
});

SQL and Spark Pools

Finally, let’s add two worker pools to the Synapse workspace. A SQL pool for T-SQL analytic queries…

sql_pool = synapse.SqlPool("sqlPool",
 resource_group_name=resource_group.name,
 location=resource_group.location,
 workspace_name=workspace.name,
 sql_pool_name="SQLPOOL1",
 collation="SQL_Latin1_General_CP1_CI_AS",
 create_mode="Default",
 sku=synapse.SkuArgs(
 name="DW100c",
 ))

var sqlPool = new SqlPool("sqlPool", new SqlPoolArgs
{
 ResourceGroupName = resourceGroup.Name,
 Location = resourceGroup.Location,
 WorkspaceName = workspace.Name,
 SqlPoolName = "SQLPOOL1",
 Collation = "SQL_Latin1_General_CP1_CI_AS",
 CreateMode = "Default",
 Sku = new Pulumi.AzureNative.Synapse.Inputs.SkuArgs
 {
 Name = "DW100c"
 },
});

const sqlPool = new synapse.SqlPool("sqlPool", {
 resourceGroupName: resourceGroup.name,
 location: resourceGroup.location,
 workspaceName: workspace.name,
 sqlPoolName: "SQLPOOL1",
 collation: "SQL_Latin1_General_CP1_CI_AS",
 createMode: "Default",
 sku: {
 name: "DW100c",
 },
});

… and a Spark pool for Big Data analytics.

spark_pool = synapse.BigDataPool("sparkPool",
 resource_group_name=resource_group.name,
 location=resource_group.location,
 workspace_name=workspace.name,
 big_data_pool_name="Spark1",
 auto_pause=synapse.AutoPausePropertiesArgs(
 delay_in_minutes=15,
 enabled=True,
 ),
 auto_scale=synapse.AutoScalePropertiesArgs(
 enabled=True,
 max_node_count=3,
 min_node_count=3,
 ),
 node_count=3,
 node_size="Small",
 node_size_family="MemoryOptimized",
 spark_version="2.4")

var sparkPool = new BigDataPool("sparkPool", new BigDataPoolArgs
{
 ResourceGroupName = resourceGroup.Name,
 Location = resourceGroup.Location,
 WorkspaceName = workspace.Name,
 BigDataPoolName = "Spark1",
 AutoPause = new AutoPausePropertiesArgs
 {
 DelayInMinutes = 15,
 Enabled = true,
 },
 AutoScale = new AutoScalePropertiesArgs
 {
 Enabled = true,
 MaxNodeCount = 3,
 MinNodeCount = 3,
 },
 NodeCount = 3,
 NodeSize = "Small",
 NodeSizeFamily = "MemoryOptimized",
 SparkVersion = "2.4"
});

const bigDataPoolResource = new synapse.BigDataPool("sparkPool", {
 resourceGroupName: resourceGroup.name,
 location: resourceGroup.location,
 workspaceName: workspace.name,
 bigDataPoolName: "Spark1",
 autoPause: {
 delayInMinutes: 15,
 enabled: true,
 },
 autoScale: {
 enabled: true,
 maxNodeCount: 3,
 minNodeCount: 3,
 },
 nodeCount: 3,
 nodeSize: "Small",
 nodeSizeFamily: "MemoryOptimized",
 sparkVersion: "2.4",
});

Ready to Dive into Analytics

Our resource definitions are ready. Run pulumi up to provision your Azure Synapse infrastructure.

$ pulumi up
...
Do you want to perform this update? yes
Updating (dev)

 Type Name Plan
 + pulumi:pulumi:Stack azure-py-synapse-dev created
 + ├─ azure-native:resources:ResourceGroup resourceGroup created
 + ├─ azure-native:storage:StorageAccount storageAccount created
 + ├─ azure-native:storage:BlobContainer users created
 + ├─ azure-native:synapse:Workspace workspace created
 + ├─ random:index:RandomUuid roleName created
 + └─ azure-native:authorization:RoleAssignment storageAccess created
 + ├─ random:index:RandomUuid userRoleName created
 + ├─ azure-native:authorization:RoleAssignment userAccess created
 + ├─ azure-native:synapse:IpFirewallRule allowAll created
 + ├─ azure-native:synapse:SqlPool sqlPool created
 + ├─ azure-native:synapse:BigDataPool sparkPool created

 Resources:
 + 12 created

Duration: 10m51s

You can now navigate to the Azure Synapse Quickstart, Step 2, and follow along with the data analysis tutorial.

Conclusion

Azure Synapse is a managed analytics service that accelerates time to insight across data warehouses and big data workloads. A Synapse workspace is a critical component of your cloud infrastructure that you should provision with infrastructure as code and other management best practices.

Pulumi and the native Azure provider open up full access to all types of Azure resources using your favorite programming languages, including Python, C#, and TypeScript. Navigate to the complete Azure Synapse example in Python, C#, or TypeScript and get started today.

Running Container Images in AWS Lambda

Mikhail Shilkov — Tue, 01 Dec 2020 00:00:00 +0000

TL;DR — To run a container image in AWS Lambda, build an OCI image that implements the Lambda Runtime API (most teams start from an AWS-provided base image), push it to Amazon ECR, and create a Lambda function with packageType: "Image" pointing at the image URI. Lambda containers support images up to 10 GB, up to 10 GB of memory, up to 10 GB of /tmp ephemeral storage, and a 15-minute execution ceiling. Pulumi automates the build, push, and function wiring in a single program. Pick Lambda containers when your workload is event-driven and bursty but your dependencies (binaries, ML models, system libraries) outgrow the 250 MB ZIP limit; pick AWS Fargate or ECS when you need long-running tasks, persistent connections, or multi-container pods.

How do I run a container image in AWS Lambda?

You package your function as a Linux container image, push it to Amazon ECR, and create a Lambda function whose packageType is Image. The image must implement the Lambda Runtime API, which is an HTTP protocol Lambda uses to deliver invocation events and collect responses. The simplest path is to start FROM an AWS-provided base image (public.ecr.aws/lambda/nodejs:22, public.ecr.aws/lambda/python:3.13, public.ecr.aws/lambda/java:21, and so on) because they ship with the Runtime Interface Client preinstalled. If you want to use a different OS or runtime, add the open-source Runtime Interface Client yourself.

With Pulumi, one program covers the entire path: ECR repository, Docker build and push, IAM role, Lambda function, and event source. It works in TypeScript, Python, Go, .NET, Java, or YAML. The rest of this post walks through that program with a real video-thumbnailer example.

import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";

const repo = new awsx.ecr.Repository("repo", {
 forceDelete: true,
});

const image = new awsx.ecr.Image("image", {
 repositoryUrl: repo.url,
 context: "./app",
 platform: "linux/amd64",
});

const role = new aws.iam.Role("thumbnailerRole", {
 assumeRolePolicy: aws.iam.getPolicyDocument({
 statements: [{
 actions: ["sts:AssumeRole"],
 principals: [{ type: "Service", identifiers: ["lambda.amazonaws.com"] }],
 }],
 }).then(doc => doc.json),
});

new aws.iam.RolePolicyAttachment("lambdaExecute", {
 role: role.name,
 policyArn: aws.iam.ManagedPolicy.AWSLambdaExecute,
});

const thumbnailer = new aws.lambda.Function("thumbnailer", {
 packageType: "Image",
 imageUri: image.imageUri,
 role: role.arn,
 timeout: 900,
 memorySize: 2048,
});

import pulumi_aws as aws
import pulumi_awsx as awsx

repo = awsx.ecr.Repository("repo", force_delete=True)

image = awsx.ecr.Image(
 "image",
 repository_url=repo.url,
 context="./app",
 platform="linux/amd64",
)

role = aws.iam.Role(
 "thumbnailerRole",
 assume_role_policy=aws.iam.get_policy_document(statements=[{
 "actions": ["sts:AssumeRole"],
 "principals": [{"type": "Service", "identifiers": ["lambda.amazonaws.com"]}],
 }]).json,
)

aws.iam.RolePolicyAttachment(
 "lambdaExecute",
 role=role.name,
 policy_arn=aws.iam.ManagedPolicy.AWS_LAMBDA_EXECUTE,
)

thumbnailer = aws.lambda_.Function(
 "thumbnailer",
 package_type="Image",
 image_uri=image.image_uri,
 role=role.arn,
 timeout=900,
 memory_size=2048,
)

pulumi up builds the image from ./app, pushes it to a fresh ECR repository, and deploys the Lambda — in one step, with one command, with no shell scripts.

What is a Lambda container image?

A Lambda container image is an OCI (or Docker v2) image whose entrypoint speaks the Lambda Runtime API. The image is stored in Amazon ECR, and Lambda pulls it on cold start. Functionally, a container-image function behaves the same as a ZIP-package function: it is invoked by the same triggers (S3, SQS, API Gateway, EventBridge, and so on), it bills the same way (per request and per GB-second), and it is bound by the same 15-minute execution ceiling. The only meaningful difference is how the code is packaged and what limits apply to that package.

Lambda container images were announced at re:Invent 2020 and have been generally available ever since. They are not a separate service — they are a deployment format for the same Lambda runtime.

What are the AWS Lambda container limits in 2026?

The hard limits below apply to every Lambda function regardless of packaging, except for the image-size column, which is specific to container images.

Limit	Container image	ZIP package
Maximum package size	10 GB (uncompressed)	250 MB (unzipped, including layers)
Maximum memory	10,240 MB	10,240 MB
Maximum ephemeral `/tmp`	10,240 MB	10,240 MB
Maximum execution timeout	15 minutes	15 minutes
Maximum payload (sync)	6 MB request / 6 MB response	6 MB request / 6 MB response
Architectures	`x86_64`, `arm64`	`x86_64`, `arm64`
SnapStart support	Not supported for container images	Java, Python, .NET

Sources: Lambda quotas, Lambda images, SnapStart documentation.

A note on cold starts: the public guidance from AWS is that container-image cold starts are typically comparable to ZIP cold starts when the image is well-optimized — AWS caches image layers and uses on-demand block-level loading so only the bytes touched at startup are downloaded. The biggest lever you have is image size: keep the image small, put rarely changed dependencies in lower layers, and prefer a minimal base. SnapStart, which is the most effective cold-start mitigation for Java, Python, and .NET ZIP functions, does not apply to container-image functions; if you need SnapStart, ship a ZIP.

How much do Lambda container images cost?

Container-image functions are billed identically to ZIP functions:

Requests: $0.20 per 1M requests (x86_64 and arm64 alike).
Duration: $0.0000166667 per GB-second on x86_64, $0.0000133334 on arm64 (Graviton2) — roughly 20% lower per GB-second, with up to 34% better price-performance per AWS.
Free tier: 1M requests and 400,000 GB-seconds per month, every month.
Storage: ECR charges $0.10 per GB-month for image storage; data transfer to Lambda within the same Region is free.

Always confirm rates against the live Lambda pricing and ECR pricing pages, since AWS adjusts them periodically.

How do Lambda containers compare to Fargate, ECS, and ZIP Lambdas?

The four options below all run containers, but they sit on a spectrum from “fully event-driven, scale to zero” to “long-running, always on.”

Dimension	Lambda (ZIP)	Lambda (container image)	AWS Fargate	Amazon ECS (EC2)
Packaging	ZIP up to 250 MB	OCI image up to 10 GB	OCI image (no Lambda runtime API needed)	OCI image
Cold start	Milliseconds to seconds; SnapStart available	Seconds (image-size dependent); no SnapStart	Tens of seconds for new tasks	Tens of seconds (capacity dependent)
Max execution time	15 minutes	15 minutes	Unbounded	Unbounded
Scale to zero	Yes	Yes	No (minimum task count)	No (minimum container instances)
Billing	Per-ms duration + per-request	Per-ms duration + per-request	Per-second of vCPU and memory	Per-EC2-instance-hour + per-task overhead
Networking	Optional VPC attachment	Optional VPC attachment	First-class VPC, ENIs, persistent connections	First-class VPC, ENIs, persistent connections
Native event triggers	100+ AWS services	100+ AWS services	EventBridge / Step Functions integrations	EventBridge / Step Functions integrations
Best for	Short, bursty, event-driven workloads	Same as ZIP, but with heavy dependencies, native binaries, large ML models	Long-running services, daemons, persistent workloads	Workloads needing custom EC2 sizing or GPU

The decision tree is short: if your workload runs for less than 15 minutes per invocation and you can fit it in 10 GB, Lambda is almost always cheaper and operationally simpler than Fargate. If you cannot fit in a 250 MB ZIP, use a Lambda container image. If your workload is long-running or needs persistent network connections, choose Fargate or ECS.

How do I deploy a Lambda container image with Pulumi?

Let’s walk through a complete program: a video thumbnailer that runs every time a .mp4 is uploaded to an S3 bucket. The function uses FFmpeg, a binary that is impractical to ship in a 250 MB ZIP, to extract a thumbnail and write it back to the bucket. The full source is in pulumi/examples.

Quick start

Bootstrap a project: pulumi new https://github.com/pulumi/examples/tree/master/aws-ts-lambda-thumbnailer
Edit ./app/Dockerfile and ./app/index.js with your function’s logic.
Deploy: pulumi up.

1. Define a Dockerfile

The container image is based on the AWS-provided Node.js 22 image, which preinstalls the Runtime Interface Client. It also installs the AWS CLI and FFmpeg, copies the handler, and points to index.handler as the entrypoint.

FROM public.ecr.aws/lambda/nodejs:22

RUN dnf install -y tar xz unzip \
 && curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" \
 && unzip awscliv2.zip && ./aws/install \
 && curl -O https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz \
 && tar -xf ffmpeg-release-amd64-static.tar.xz \
 && mv ffmpeg-*-amd64-static/ffmpeg /usr/local/bin/ \
 && rm -rf awscliv2.zip aws ffmpeg-*-amd64-static*

COPY index.js ${LAMBDA_TASK_ROOT}
CMD ["index.handler"]

2. Create an S3 bucket for inputs and outputs

import * as aws from "@pulumi/aws";

const bucket = new aws.s3.Bucket("bucket");

import pulumi_aws as aws

bucket = aws.s3.Bucket("bucket")

3. Build the image and push it to ECR

Pulumi Crosswalk for AWS wraps the ECR repository, the Docker build, and the push into two resources:

import * as awsx from "@pulumi/awsx";

const repo = new awsx.ecr.Repository("repo", {
 forceDelete: true,
});

const image = new awsx.ecr.Image("image", {
 repositoryUrl: repo.url,
 context: "./app",
 platform: "linux/amd64",
});

import pulumi_awsx as awsx

repo = awsx.ecr.Repository("repo", force_delete=True)

image = awsx.ecr.Image(
 "image",
 repository_url=repo.url,
 context="./app",
 platform="linux/amd64",
)

The ./app directory holds the Dockerfile and the handler. platform: "linux/amd64" is explicit so the build is reproducible from Apple Silicon and Graviton developer machines. Switch to linux/arm64 if you want the cheaper Graviton runtime.

4. Set up the IAM role

const role = new aws.iam.Role("thumbnailerRole", {
 assumeRolePolicy: aws.iam.getPolicyDocument({
 statements: [{
 actions: ["sts:AssumeRole"],
 principals: [{ type: "Service", identifiers: ["lambda.amazonaws.com"] }],
 }],
 }).then(doc => doc.json),
});

new aws.iam.RolePolicyAttachment("lambdaExecute", {
 role: role.name,
 policyArn: aws.iam.ManagedPolicy.AWSLambdaExecute,
});

role = aws.iam.Role(
 "thumbnailerRole",
 assume_role_policy=aws.iam.get_policy_document(statements=[{
 "actions": ["sts:AssumeRole"],
 "principals": [{"type": "Service", "identifiers": ["lambda.amazonaws.com"]}],
 }]).json,
)

aws.iam.RolePolicyAttachment(
 "lambdaExecute",
 role=role.name,
 policy_arn=aws.iam.ManagedPolicy.AWS_LAMBDA_EXECUTE,
)

AWSLambdaExecute is the AWS-managed policy that grants the read/write S3 access and CloudWatch Logs access this function needs. For a real production workload, scope this down to a custom policy that names the bucket explicitly.

5. Configure the Lambda function

const thumbnailer = new aws.lambda.Function("thumbnailer", {
 packageType: "Image",
 imageUri: image.imageUri,
 role: role.arn,
 timeout: 900,
 memorySize: 2048,
 architectures: ["x86_64"],
});

thumbnailer = aws.lambda_.Function(
 "thumbnailer",
 package_type="Image",
 image_uri=image.image_uri,
 role=role.arn,
 timeout=900,
 memory_size=2048,
 architectures=["x86_64"],
)

packageType: "Image" is the only setting that distinguishes a container function from a ZIP function. The imageUri is the digest-pinned URI of the image we just pushed, so any change to the Dockerfile or handler triggers a redeploy on the next pulumi up.

6. Wire up the S3 trigger

new aws.s3.BucketNotification("onNewVideo", {
 bucket: bucket.id,
 lambdaFunctions: [{
 lambdaFunctionArn: thumbnailer.arn,
 events: ["s3:ObjectCreated:*"],
 filterSuffix: ".mp4",
 }],
});

new aws.lambda.Permission("allowBucket", {
 action: "lambda:InvokeFunction",
 function: thumbnailer.name,
 principal: "s3.amazonaws.com",
 sourceArn: bucket.arn,
});

aws.s3.BucketNotification(
 "onNewVideo",
 bucket=bucket.id,
 lambda_functions=[{
 "lambda_function_arn": thumbnailer.arn,
 "events": ["s3:ObjectCreated:*"],
 "filter_suffix": ".mp4",
 }],
)

aws.lambda_.Permission(
 "allowBucket",
 action="lambda:InvokeFunction",
 function=thumbnailer.name,
 principal="s3.amazonaws.com",
 source_arn=bucket.arn,
)

That is the entire program. pulumi up provisions everything in the right order, and a subsequent edit to the Dockerfile rebuilds and rolls out a new image automatically.

Deploy Lambda containers with Pulumi

Build the image, push it to Amazon ECR, wire up the IAM role, and deploy the function from one program in TypeScript, Python, Go, or the language you already use.

Get started

When should I choose Lambda containers over ZIP packages?

Reach for a container image when one or more of the following is true:

Your dependencies exceed 250 MB unzipped. Native binaries (FFmpeg, ImageMagick, headless Chromium), large ML model weights, and full Python data-science stacks all blow past the ZIP limit quickly.
You need a custom OS, runtime, or system library. Lambda’s managed runtimes are good but opinionated. A container lets you pin a specific glibc version, install OS packages, or run a language Lambda doesn’t natively support.
Your team’s CI/CD already builds containers. Reusing your existing image-build pipeline (and its scanning, signing, and SBOM tooling) is often easier than maintaining a parallel ZIP-packaging pipeline.
You want a single artifact format across Lambda, Fargate, and ECS. A container image that runs in Lambda for short bursts can also run in Fargate for long jobs — convenient for hybrid workloads.

Stick with ZIP when your code is small, you want SnapStart, or you want the fastest possible cold starts.

You might also like:

Conclusion

AWS Lambda’s container-image support brings the industry-standard packaging format to event-driven serverless functions, and Pulumi covers the full deploy (ECR repository, image build, IAM role, function, and event source) in one program in the language you already use. If you have outgrown the ZIP package, this is the pattern you want.

Watch a demo of the thumbnailer below.

Next steps:

Browse the full Lambda + container image example in pulumi/examples.
Read the Pulumi AWS Lambda guide for event sources, IAM patterns, and packaging tips.
See AWS Lambda SnapStart with Pulumi if you need faster cold starts on a ZIP-packaged Java, Python, or .NET function.
Cut x86 costs with Lambda functions powered by Graviton2.
Get started with Pulumi for AWS today.

Announcing Next Generation Pulumi Azure Provider

Mikhail Shilkov — Mon, 21 Sep 2020 00:00:00 +0000

The next-generation Azure provider is now Azure-Native.

We are excited to announce the beta release of a next generation Microsoft Azure provider for Pulumi. Azure has been a rapidly growing cloud platform among Pulumi users over the last year, and with the next generation Azure provider, we are doubling down on providing the best support possible for the Azure platform in Pulumi. We designed the new provider to expose the entire API surface of Azure to developers and operators, now and forever.

The new Azure provider for Pulumi (azure-nextgen) works directly with the Azure Resource Manager (ARM) platform instead of depending on a handwritten layer as with the previous provider. This approach ensures higher quality and higher fidelity with the Azure platform.

Full API Coverage

The next generation Pulumi Azure provider covers 100% of the resources available in Azure Resource Manager. The new provider supports 890 resource types at launch, nearly double the number supported by the previous Pulumi Azure Provider. Every property of each resource is always represented in the SDKs.

The provider also contains functions to retrieve keys, secrets, and connection strings from all resources that expose them.

The new SDKs include full coverage for Azure services, including Azure Static Web Apps, Azure Synapse Analytics, Azure Logic Apps, Azure Service Fabric, Azure Blockchain Service, Azure API Management, and dozens of other services.

If you can deploy a resource with ARM Templates, you can deploy it with the next Generation Pulumi Azure provider!

Always Up-to-Date

Unlike the original Azure provider, which requires manual work to keep updated, the new provider is designed to be always up-to-date with additions and changes to Azure APIs.

We generate Pulumi SDKs for azure-nextgen automatically from Azure API specifications published by Microsoft. An automated pipeline releases updated resources within hours after any current API specifications are merged. We publish daily updates via automated builds and cut minor SDK versions every two weeks.

By generating the SDKs directly from the Azure Resource Manager resource model specifications maintained by Azure service teams, the Azure NextGen provider is robust and reliable, with fewer moving parts involved and fewer sources of potential bugs and incompatibilities.

Excited about a new service announced by Microsoft? Chances are it’s already in the latest Azure NextGen package!

API Versions

The Azure Resource Manager API is structured around Resource Providers—high-level groups like “storage”, “compute”, or “web”. We map Resource Providers to top-level modules or namespaces in Pulumi SDKs.

Each resource provider defines one or more API versions, for example, “2015-05-01”, “2020-09-01”, or “2020-08-01-preview”. Every version of every ARM API is available in Pulumi SDKs, and each version has its own module or namespace.

In addition, a latest version in the Pulumi SDK maps to the latest stable API version as determined by the Azure SDK teams. Using a specific version ensures full compatibility, while using the latest version is a quick way to start, but does not guarantee compatibility between minor versions of the provider.

Now, you always have access to the latest and greatest, but at the same time, we don’t force you to upgrade your production resources unless you are ready.

Every Language

The next generation Azure provider is available in preview today for all Pulumi languages. The new azure-nextgen SDKs are open source on GitHub and available in NPM, NuGet, PyPI, and Go Modules.

import * as resources from "@pulumi/azure-nextgen/resources/latest";
import * as storage from "@pulumi/azure-nextgen/storage/latest";

const resourceGroup = new resources.ResourceGroup("resourceGroup", {
 resourceGroupName: "my-rg",
 location: "WestUS",
});

const storageAccount = new storage.StorageAccount("sa", {
 resourceGroupName: resourceGroup.name,
 accountName: "mystorageaccount",
 location: resourceGroup.location,
 sku: {
 name: "Standard_LRS",
 tier: "Standard",
 },
 kind: "StorageV2",
});

from pulumi_azure_nextgen.storage import latest as storage
from pulumi_azure_nextgen.resources import latest as resources

resource_group = resources.ResourceGroup('resource_group',
 resource_group_name='my-rg',
 location='WestUS')

account = storage.StorageAccount('sa',
 account_name='mystorageaccount',
 resource_group_name=resource_group.name,
 location=resource_group.location,
 sku=storage.SkuArgs(
 name='Standard_LRS',
 tier='Standard',
 ),
 kind='StorageV2')

using Pulumi;
using Pulumi.AzureNextGen.Resources.Latest;
using Pulumi.AzureNextGen.Storage.Latest;
using Pulumi.AzureNextGen.Storage.Latest.Inputs;

class MyStack : Stack
{
 public MyStack()
 {
 var resourceGroup = new ResourceGroup("resourceGroup", new ResourceGroupArgs
 {
 ResourceGroupName = "my-rg",
 Location = "WestUS"
 });

 var storageAccount = new StorageAccount("sa", new StorageAccountArgs
 {
 ResourceGroupName = resourceGroup.Name,
 AccountName = "mystorageaccount",
 Location = resourceGroup.Location,
 Sku = new SkuArgs
 {
 Name = "Standard_LRS",
 Tier = "Standard"
 },
 Kind = "StorageV2"
 });
 }
}

package main

import (
 resources "github.com/pulumi/pulumi-azure-nextgen/sdk/go/azure/resources/latest"
 storage "github.com/pulumi/pulumi-azure-nextgen/sdk/go/azure/storage/latest"
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 resourceGroup, err := resources.NewResourceGroup(ctx, "resourceGroup", &resources.ResourceGroupArgs{
 ResourceGroupName: pulumi.String("my-rg"),
 Location: pulumi.String("WestUS"),
 })
 if err != nil {
 return err
 }

 account, err := storage.NewStorageAccount(ctx, "sa", &storage.StorageAccountArgs{
 ResourceGroupName: resourceGroup.Name,
 AccountName: pulumi.String("mystorageaccount"),
 Location: resourceGroup.Location,
 Sku: &storage.SkuArgs{
 Name: pulumi.String("Standard_LRS"),
 Tier: pulumi.String("Standard"),
 },
 Kind: pulumi.String("StorageV2"),
 })

 return err
 })
}

API documentation is available at Azure NextGen API Reference and includes more than 1,000 resource examples.

Integrated with Azure Ecosystem

Relying on the shape of the Azure API, we can integrate the NextGen provider into the broader Azure ecosystem. In the coming weeks, we will release several capabilities to simplify the adoption of the new provider:

Command-line and web-based tools to convert Azure Resource Manager Templates to Pulumi programs in the language of your choice
A flow to import an existing Azure Resource Group and all its resources to your Pulumi project
A multi-language component resource to embed ARM Templates in Pulumi programs, including per-resource previews
Integrations with ARM Template generation tools like Project Bicep and Farmer

Bring your existing resources and assets and start managing them with the NextGen Azure provider!

Provider Coexistence

We will continue to invest in the existing azure provider, and Pulumi users can use either provider or both, side-by-side, in their applications.

A single Pulumi project can use both providers. Each provider must be configured independently, but they accept mostly the same configuration options. The outputs of a resource from one provider can flow to inputs of a resource from the other provider. In both cases, they are the values available from Azure itself, such as the name or id of an Azure resource.

Pulumi has a separate azuread provider, which will continue to manage Azure Active Directory resources.

If you have existing projects using the current Pulumi Azure provider, you can continue to use that provider, and it will be updated indefinitely. You can add new cloud resources using the existing provider or start creating new infrastructure with the new provider.

New Pulumi Azure projects will default to use the azure-nextgen provider once the provider reaches general availability.

Getting Started

New Pulumi templates are available for the new azure-nextgen provider:

$ pulumi new azure-nextgen-typescript

$ pulumi new azure-nextgen-python

$ pulumi new azure-nextgen-csharp

$ pulumi new azure-nextgen-go

Several larger examples are available in the Pulumi Examples repo:

Web Applications with Azure App Service and Docker: TypeScript, C#, Python, Go
Azure AKS cluster: TypeScript, C#, Python, Go
Web Application with Azure Container Instances: TypeScript, C#, Python, Go
Web Server Using Azure Virtual Machine: TypeScript, Python

You can browse API reference docs with inline examples or explore the Pulumi Azure NextGen SDKs repository.

Unit Testing Cloud Deployments with .NET

Mikhail Shilkov — Thu, 21 May 2020 00:00:00 +0000

Because Pulumi uses general-purpose programming languages to provision cloud resources, you can take advantage of native tools and perform automated tests of your infrastructure. The full power of each language is available, including access to libraries and frameworks for testing.

This blog post takes a deeper dive into mock-based unit testing of Pulumi programs written in C# and F#.

How Unit Testing Works in Pulumi

Let’s start with a picture showing how Pulumi components interact during a typical deployment process.

Whenever a new resource is instantiated in code, the program makes a remote call to the engine. The engine receives the resource’s input values, validates them, and translates the request to an invocation of a cloud API. The API returns some data, which are translated to resource outputs and sent back to the program.

The remote calls may be slow, unreliable, and non-deterministic, which makes testing of these interactions hard.

Our testing guide outlines several testing methods, but today I want to focus on unit testing.

The Pulumi SDK provides a hook to replace all the remote calls with mocks. Mocks run in the same process and can respond immediately with hard-coded or calculated on-the-fly data.

We can write fully deterministic and blazingly fast automated tests as all remote calls and uncertainty are eliminated. There is no cloud to respond to resource creation, so it’s a developer’s responsibility to mimic the cloud behavior with mocks adequately.

This blog post walks you through an example of unit testing and mocking with the .NET SDK.

Define the Base Structure

In this article, I build a program that deploys a static website to Azure. I use TDD to add new tests and deployment components bit-by-bit.

Let’s start with an empty project—go ahead and create a .NET Core Console Application with the .NET CLI or your favorite IDE.

Install NuGet packages

You are free to choose your unit testing frameworks, mocking and assertions libraries. I’m using NUnit with FluentAssertions, and my program tests Azure resources, so this is my project file:

<Project Sdk="Microsoft.NET.Sdk">

 <PropertyGroup>
 <OutputType>Exe</OutputType>
 <TargetFramework>netcoreapp3.1</TargetFramework>
 <Nullable>enable</Nullable>
 </PropertyGroup>

 <ItemGroup>
 <PackageReference Include="FluentAssertions" Version="5.10.2" />
 <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.5.0" />
 <PackageReference Include="NUnit" Version="3.12.0" />
 <PackageReference Include="NUnit3TestAdapter" Version="3.16.1" />
 <PackageReference Include="Pulumi.Azure" Version="3.*" />
 </ItemGroup>

</Project>

<Project Sdk="Microsoft.NET.Sdk">

 <PropertyGroup>
 <OutputType>Exe</OutputType>
 <TargetFramework>netcoreapp3.1</TargetFramework>
 <Nullable>enable</Nullable>
 </PropertyGroup>

 <ItemGroup>
 <PackageReference Include="FluentAssertions" Version="5.10.2" />
 <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.5.0" />
 <PackageReference Include="NUnit" Version="3.12.0" />
 <PackageReference Include="NUnit3TestAdapter" Version="3.16.1" />
 <PackageReference Include="Pulumi.Azure" Version="3.*" />
 <PackageReference Include="Pulumi.FSharp" Version="2.*" />
 </ItemGroup>

 <ItemGroup>
 <Compile Include="Testing.fs" />
 <Compile Include="WebsiteStack.fs" />
 <Compile Include="WebsiteStackTests.fs" />
 </ItemGroup>

</Project>

Stack

A Pulumi stack is the “unit” of our testing. Every test instantiates a stack, retrieves the resources that the stack defines, and makes assertions about them.

Here is my starting point in WebsiteStack file:

using System.IO;
using Pulumi;
using Pulumi.Azure.Core;
using Storage = Pulumi.Azure.Storage;

public class WebsiteStack : Stack
{
 public WebsiteStack()
 {
 // <-- Cloud resources go here
 }
}

namespace UnitTesting

open Pulumi
open Pulumi.FSharp
open Pulumi.Azure.Core

type WebsiteStack() =
 inherit Stack()

 // <-- Cloud resources go here

This class is precisely what Pulumi expects to start deploying your resources to the cloud. Once the test suite is ready and green, you can deploy the stack with pulumi up.

Mocks

As I explained above, unit tests replace the real Pulumi engine with mocks. Mocks get all calls and respond with predefined values.

A mock class has to implement two methods of Pulumi.Testing.IMocks interface. NewResourceAsync is called whenever a new resource is defined, while CallAsync is invoked when our program retrieves information about existing cloud resources. We’ll focus on the former in this post, but we still need to define both.

While you are free to use your favorite mocking library, I’ll keep it simple and define the mocks as a plain class.

using System.Collections.Immutable;
using System.Threading.Tasks;
using Pulumi;
using Pulumi.Testing;

namespace UnitTesting
{
 class Mocks : IMocks
 {
 public Task<(string id, object state)> NewResourceAsync(
 string type, string name, ImmutableDictionary<string, object> inputs, string? provider, string? id)
 {
 var outputs = ImmutableDictionary.CreateBuilder<string, object>();

 // Forward all input parameters as resource outputs, so that we could test them.
 outputs.AddRange(inputs);

 // <-- We'll customize the mocks here

 // Default the resource ID to `{name}_id`.
 id ??= $"{name}_id";
 return Task.FromResult((id, (object)outputs));
 }

 public Task<object> CallAsync(string token, ImmutableDictionary<string, object> inputs, string? provider)
 {
 // We don't use this method in this particular test suite.
 // Default to returning whatever we got as input.
 return Task.FromResult((object)inputs);
 }
 }
}

namespace UnitTesting

open System
open System.Collections.Immutable
open System.Threading.Tasks
open Pulumi.Testing

type Mocks() =
 interface IMocks with
 member this.NewResourceAsync(typeName: string, name: string, inputs: ImmutableDictionary<string, obj>, provider: string, id: string): Task<ValueTuple<string,obj>> =
 // Forward all input parameters as resource outputs, so that we could test them.
 let dict = inputs.ToImmutableDictionary() :> obj

 // <-- We'll customize the mocks here

 // Default the resource ID to `{name}_id`.
 let id = if id = null then sprintf "%s_id" name else id
 Task.FromResult(struct (id, outputs :> obj))

 member this.CallAsync(token: string, inputs: ImmutableDictionary<string, obj>, provider: string): Task<obj> =
 // We don't use this method in this particular test suite.
 // Default to returning whatever we got as input.
 Task.FromResult(null :> obj)

Both methods receive several parameters and need to return a result. Notably, they get the list of inputs—the arguments that our program defined for the given resource. The goal of the mocks is to return the list of outputs, similarly to what a cloud provider would do.

My default implementation returns the outputs that include all the inputs and nothing else. That’s a sensible default: usually, real outputs are a superset of inputs. We’ll extend this behavior as we need later on.

Test Fixture

The next class to add is the container for my future unit tests. I named the file WebsiteStackTests.cs and it defines a test container, called a “test fixture” in NUnit:

using System.Linq;
using System.Threading.Tasks;
using FluentAssertions;
using NUnit.Framework;
using Pulumi;
using Pulumi.Azure.Core;
using Storage = Pulumi.Azure.Storage;

namespace UnitTesting
{
 [TestFixture]
 public class WebserverStackTests
 {
 // <-- Tests go here

 private static Task<ImmutableArray<Resource>> TestAsync()
 {
 return Deployment.TestAsync<WebsiteStack>(new Mocks(), new TestOptions {IsPreview = false});
 }
 }
}

namespace UnitTesting

open System
open System.Collections.Immutable
open System.Threading.Tasks
open NUnit.Framework
open FluentAssertions
open Pulumi
open Pulumi.Azure.Core
open Pulumi.Azure.Storage
open Pulumi.Testing

[<TestFixture>]
type WebserverStackTests() =
 let runTest(): ImmutableArray<Resource> =
 let options = new TestOptions(IsPreview = Nullable<bool> false)
 Deployment.TestAsync<WebsiteStack>(new Mocks(), options)
 |> Async.AwaitTask
 |> Async.RunSynchronously

 // <-- Tests go here

I defined a helper method TestAsync that points Pulumi’s Deployment.TestAsync to our stack and mock classes.

As we progress through the article, I will add tests to this class one-by-one.

Retrieve output values

Finally, I need a small extension method to extract values from outputs. Every Pulumi resource returns values wrapped inside an Output<T> container. While the real engine runs, those values may be known or unknown, but my mock-based tests always return known values. It’s safe to get those values and convert them to a Task<T>:

public static class TestingExtensions
{
 public static Task<T> GetValueAsync<T>(this Output<T> output)
 {
 var tcs = new TaskCompletionSource<T>();
 output.Apply(v => { tcs.SetResult(v); return v; });
 return tcs.Task;
 }
}

let getValue(output: Output<'a>): 'a =
 let tcs = new TaskCompletionSource<'a>()
 output.Apply(fun v -> tcs.SetResult(v); v) |> ignore
 tcs.Task.Result

To learn more about outputs, read this article.

First Test

The setup is done, time to write my first unit test! True to the TDD spirit, I write the tests before I define any resources.

Every Azure resource has to live inside a resource group, so my first test checks that a new resource group is defined.

[Test]
public async Task SingleResourceGroupExists()
{
 var resources = await TestAsync();

 var resourceGroups = resources.OfType<ResourceGroup>().ToList();
 resourceGroups.Count.Should().Be(1, "a single resource group is expected");
}

[<Test>]
member this.SingleResourceGroupExists() =
 let resources = runTest()

 let resourceGroupCount = resources.OfType<ResourceGroup>() |> Seq.length
 resourceGroupCount.Should().Be(1, "a single resource group is expected") |> ignore

This code is great to illustrate the overall structure of each test:

Call the TestAsync method to evaluate the stack.
Find the target resource in the collection of created resources.
Assert a property of the target resource.

In this case, the test validates that there is one and only one resource group defined.

I can run dotnet test and see the expected test failure:

$ dotnet test
...
X SingleResourceGroupExists [238ms]
Error Message:
Expected resourceGroups.Count to be 1 because a single resource group is expected, but found 0.
...
Total tests: 1
Failed: 1
Total time: 0.9270 Seconds

I go ahead and add the following definition to the WebsiteStack constructor.

var resourceGroup = new ResourceGroup("www-prod-rg");

let resourceGroup = new ResourceGroup("www-prod-rg")

This change is enough to make the tests green!

$ dotnet test
...
Test Run Successful.
Total tests: 1
Passed: 1
Total time: 0.8462 Seconds

Storage Account

It’s time to add a test for a second resource in the stack: a Storage Account. To make things more interesting, I want to check that the storage account belongs to our resource group.

There’s no direct link between a storage account object and a resource group object. Instead, we need to check the ResourceGroupName property of the account. The test below expects the account to belong to a resource group called www-prod-rg.

[Test]
public async Task StorageAccountBelongsToResourceGroup()
{
 var resources = await TestAsync();
 var storageAccount = resources.OfType<Storage.Account>().SingleOrDefault();
 storageAccount.Should().NotBeNull("Storage account not found");

 var resourceGroupName = await storageAccount.ResourceGroupName.GetValueAsync();
 resourceGroupName.Should().Be("www-prod-rg");
}

[<Test>]
member this.StorageAccountBelongsToResourceGroup() =
 let resources = runTest()
 let storageAccount = resources.OfType<Account>() |> Seq.tryHead |> Option.toObj
 storageAccount.Should().NotBeNull("Storage account not found") |> ignore

 let resourceGroupName = getValue storageAccount.ResourceGroupName
 resourceGroupName.Should().Be("www-prod-rg", null) |> ignore

Of course, the test fails. I try to fix it with what I think is the minimal change to make the test pass by adding a new resource to the stack and pointing it to the resource group.

var storageAccount = new Storage.Account("wwwprodsa", new Storage.AccountArgs
{
 ResourceGroupName = resourceGroup.Name,
});

let storageAccount =
 new Account(
 "wwwprodsa",
 new AccountArgs(ResourceGroupName = io resourceGroup.Name))

However, when I rerun the test suite, all three tests fail. They complain about the missing required properties AccountReplicationType and AccountTier on the storage account, so I have to add those.

var storageAccount = new Storage.Account("wwwprodsa", new Storage.AccountArgs
{
 ResourceGroupName = resourceGroup.Name,
 AccountTier = "Standard",
 AccountReplicationType = "LRS",
 StaticWebsite = new Storage.Inputs.AccountStaticWebsiteArgs
 {
 IndexDocument = "index.html"
 }
});

let storageAccount =
 new Account(
 "wwwprodsa",
 new AccountArgs(
 ResourceGroupName = io resourceGroup.Name,
 AccountTier = input "Standard",
 AccountReplicationType = input "LRS",
 StaticWebsite = input (new AccountStaticWebsiteArgs(IndexDocument = input "index.html"))))

I defined the StaticWebsite property as well: I’ll leave testing these values as an exercise for the reader.

Two resource group tests are back to green again, but the storage account test fails with a new error message.

X StorageAccountBelongsToResourceGroup [84ms]
Error Message:
Expected resourceGroupName to be "www-prod-rg", but found <null>.

What’s going on, and why is it null?

I defined the account name like this: ResourceGroupName = resourceGroup.Name. However, if we look closely, the resourceGroup resource doesn’t have an input property Name defined. www-prod-rg is a logical name for Pulumi deployment, not the physical name of the resource.

Under normal circumstances, the Pulumi engine would use the logical name to produce the physical name of the resource group automatically (see resource names for details). However, my mocks don’t do that.

That’s a good reason to change the Mocks implementation. I add the following lines to the NewResourceAsync method.

// Set the name to resource name if it's not set explicitly in inputs.
if (!inputs.ContainsKey("name"))
 outputs.Add("name", name);

let inputKVs = inputs |> Seq.map(fun kv -> kv.Key, kv.Value) |> Seq.toList
let nameKVs = if inputs.ContainsKey("name") then [] else [("name", name :> obj)]

let outputs = [inputKVs; nameKVs] |> Seq.concat |> Seq.map KeyValuePair
let dict = outputs.ToImmutableDictionary() :> obj

Note that mocks operate on weakly-typed dictionaries, so I need to get the property name right. Pulumi SDKs are open source, so I looked here to double-check the exact value.

After this change in Mocks, my tests go green again.

Test Run Successful.
Total tests: 3
Passed: 3

Website Files

The next step is to upload some files to the static website. Well, instead, to write an automated test that validates the upload with mocks.

I create a wwwroot folder with two HTML files in it and copy the folder to the build’s output. Here is my change in the project file.

<ItemGroup>
 <None Update="wwwroot\**\*.*">
 <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
 </None>
</ItemGroup>

Now, the test is straightforward: it expects two Blob resources in the stack.

[Test]
public async Task UploadsTwoFiles()
{
 var resources = await TestAsync();
 var files = resources.OfType<Storage.Blob>().ToList();
 files.Count.Should().Be(2, "Should have uploaded files from `wwwroot`");
}

[<Test>]
member this.UploadsTwoFiles() =
 let resources = runTest()
 let filesCount = resources.OfType<Blob>() |> Seq.length
 filesCount.Should().Be(2, "Should have uploaded files from `wwwroot`") |> ignore

As intended, the test fails immediately.

X UploadsTwoFiles [95ms]
Error Message:
Expected files.Count to be 2 because Should have uploaded files from `wwwroot`, but found 0.

I extend my stack with the following loop that navigates through the files and creates a storage blob for each of them.

var files = Directory.GetFiles("wwwroot");
foreach (var file in files)
{
 var blob = new Storage.Blob(file, new Storage.BlobArgs
 {
 ContentType = "application/html",
 Source = new FileAsset(file),
 StorageAccountName = storageAccount.Name,
 StorageContainerName = "$web",
 Type = "Block"
 });
}

let files =
 let fileNames = Directory.GetFiles("wwwroot")
 fileNames
 |> Seq.map(fun file ->
 new Blob(
 file,
 new BlobArgs(
 ContentType = input "application/html",
 Source = input (new FileAsset(file) :> AssetOrArchive),
 StorageAccountName = io storageAccount.Name,
 StorageContainerName = input "$web",
 Type = input "Block")))
 |> List.ofSeq

However, this change breaks the stack and all tests in the suite:

Error Message:
Pulumi.RunException : Running program failed with an unhandled exception:
System.InvalidOperationException: Unsupported value when converting to protobuf: Pulumi.FileAsset
at Pulumi.Serialization.Serializer.CreateValue(Object value)
...

Once again, our mocks fail to represent the behavior of the engine accurately. The Pulumi engine knows about the FileAsset class pointing to a file on the disk and how to convert it to an uploaded blob. But, the engine doesn’t copy this property to outputs. I need to adjust the mocks again.

I’m not particularly interested in testing the binary contents of the files now, so I’ll change the Mocks class to ignore the source property and not to include it into the output dictionary.

if (type == "azure:storage/blob:Blob")
{
 // Assets can't directly go through the engine.
 // We don't need them in the test, so blank out the property for now.
 outputs.Remove("source");
}

let outputs =
 [inputKVs; nameKVs]
 |> Seq.concat
 |> Seq.filter (fun (k, _) -> typeName <> "azure:storage/blob:Blob" || k <> "source")
 |> Seq.map KeyValuePair

This change makes my test suite happy again.

Test Run Successful.
Total tests: 4
Passed: 4

Validate Website URL

So far, I’ve been validating the input parameters of resources. What if I want to test something that is usually done by the cloud provider?

For instance, when Azure creates a static website, it automatically assigns a public endpoint. The endpoint is then available in the PrimaryWebEndpoint property after the Pulumi program ran and resources are created. I may want to export this value from stack outputs and validate it in a unit test.

[Test]
public async Task StackExportsWebsiteUrl()
{
 var resources = await TestAsync();
 var stack = resources.OfType<WebsiteStack>().First();

 var endpoint = await stack.Endpoint.GetValueAsync();
 endpoint.Should().Be("https://wwwprodsa.web.core.windows.net");
}

[<Test>]
member this.StackExportsWebsiteUrl() =
 let resources = runTest()
 let stack = resources.OfType<WebsiteStack>() |> Seq.head

 let endpoint = getValue stack.Endpoint
 endpoint.Should().Be("https://wwwprodsa.web.core.windows.net", null) |> ignore

This test doesn’t even compile yet, so I have to define the Endpoint property and assign a value to it. Here is the complete implementation of the stack with the output property defined at the end.

public class WebsiteStack : Stack
{
 public WebsiteStack()
 {
 var resourceGroup = new ResourceGroup("www-prod-rg", new ResourceGroupArgs
 {
 Tags = { { "Environment", "production" } }
 });

 var storageAccount = new Storage.Account("wwwprodsa", new Storage.AccountArgs
 {
 ResourceGroupName = resourceGroup.Name,
 AccountTier = "Standard",
 AccountReplicationType = "LRS",
 StaticWebsite = new Storage.Inputs.AccountStaticWebsiteArgs
 {
 IndexDocument = "index.html"
 }
 });

 var files = Directory.GetFiles("wwwroot");
 foreach (var file in files)
 {
 var blob = new Storage.Blob(file, new Storage.BlobArgs
 {
 ContentType = "application/html",
 Source = new FileAsset(file),
 StorageAccountName = storageAccount.Name,
 StorageContainerName = "$web",
 Type = "Block"
 });
 }

 this.Endpoint = storageAccount.PrimaryWebEndpoint;
 }

 [Output] public Output<string> Endpoint { get; set; }
}

type WebsiteStack() =
 inherit Stack()

 let resourceGroup =
 new ResourceGroup(
 "www-prod-rg",
 new ResourceGroupArgs(Tags = inputMap(["Environment", input "production"])))

 let storageAccount =
 new Account(
 "wwwprodsa",
 new AccountArgs(
 ResourceGroupName = io resourceGroup.Name,
 AccountTier = input "Standard",
 AccountReplicationType = input "LRS",
 StaticWebsite = input (new AccountStaticWebsiteArgs(IndexDocument = input "index.html"))))

 let files =
 let fileNames = Directory.GetFiles("wwwroot")
 fileNames
 |> Seq.map(fun file ->
 new Blob(
 file,
 new BlobArgs(
 ContentType = input "application/html",
 Source = input (new FileAsset(file) :> AssetOrArchive),
 StorageAccountName = io storageAccount.Name,
 StorageContainerName = input "$web",
 Type = input "Block")))
 |> List.ofSeq

 [<Output>]
 member val Endpoint = storageAccount.PrimaryWebEndpoint with get, set

Now, the test compiles but fails when executed:

X StackExportsWebsiteUrl [85ms]
Error Message:
Expected endpoint to be "https://wwwprodsa.web.core.windows.net", but found <null>.

That’s because there is no call to Azure to populate the endpoint. It’s time to make the last change to my Mocks class to emulate that assignment.

// For a Storage Account...
if (type == "azure:storage/account:Account")
{
 // ... set its web endpoint property.
 // Normally this would be calculated by Azure, so we have to mock it.
 outputs.Add("primaryWebEndpoint", $"https://{name}.web.core.windows.net");
}

let endpointKVs =
 if typeName = "azure:storage/account:Account"
 then ["primaryWebEndpoint", sprintf "https://%s.web.core.windows.net" name :> obj]
 else []

let outputs =
 [inputKVs; nameKVs; endpointKVs]
// ...

And that’s it! My static website is ready and tested!

Test Run Successful.
Total tests: 5
Passed: 5
Total time: 0.9338 Seconds

Note that it still takes less than a second to run my test suite so that I can iterate very quickly.

Get Started

The tests above cover the basics of unit testing with Pulumi .NET SDK. You can take it from here and apply the techniques and practices that you use while testing the application code. You may also try more advanced practices like property-based testing or behavior-driven development—we believe that the mocks enable many testing styles.

Here are several useful pointers to get started with testing in Pulumi:

Testing Guide
Full code for this blog post: C#, F#.
Another example of unit testing in C#

Announcing Pulumi Azure Provider 2.0

Mikhail Shilkov — Mon, 16 Mar 2020 00:00:00 +0000

This blog post is outdated and no longer accurate. Go to our Azure Classic package page for up to date details.

We are happy to announce the release of a new major version of the Pulumi Azure provider. Pulumi Azure 2.0 is based on the 2.0 release of the upstream provider and brings several improvements and breaking changes.

This article outlines the most significant changes:

Improvements in Azure Storage resources
New resources for VMs & VM Scale Sets
Removed deprecated resources and fields
Latest versions in Callback Functions
Configuring custom timeouts
How to migrate and where to get help if needed

The provider release is not directly related to our Pulumi 2.0 plans.

Improvements in Azure Storage

We streamlined the experience of working with Azure Storage resources.

The Storage Account resource has finally received native support for static website provisioning: it’s as simple as declaring a staticWebsite property:

const storageAccount = new azure.storage.Account("mywebsite", {
 resourceGroupName: resourceGroup.name,
 accountReplicationType: "LRS",
 accountKind: "StorageV2",
 staticWebsite: {
 indexDocument: "index.html",
 },
});

In version 1.x, the Blob resource could upload a file to a storage container, while the ZipBlob would take a local directory, zip it, and upload the archive file to a storage container. In 2.0, we deprecated the ZipBlob resource and combined both capabilities in Blob.

Blob can now accept an instance of the FileAsset class. FileAsset watches the contents of the file on disk, so if the file changes, Pulumi re-uploads the file on the next update.

The following snippet uploads a list of files from a local disk to the newly created static website:

foreach (let file in files)
 new azure.storage.Blob(file, {
 storageAccountName: storageAccount.name,
 storageContainerName: "$web",
 type: "Block",
 source: new pulumi.asset.FileAsset(`./wwwroot/${file}`),
 contentType: "text/html",
 }),
);

Note that resourceGroupName properties were removed from all storage resources except the account: blobs or queues can’t belong to a resource group on their own.

New Resources for Virtual Machine & Virtual Machine Scale Set

There are four new resources for Virtual Machine & Virtual Machine Scale Set separated by operating system: LinuxVirtualMachine, WindowsVirtualMachine, LinuxVirtualMachineScaleSet, WindowsVirtualMachineScaleSet. The old-style generic VirtualMachine and VirtualMachineScaleSet are still available, so you can take your time before migrating existing stacks.

Here is a snippet that defines an Ubuntu VM:

const vm = new azure.compute.LinuxVirtualMachine("server-vm", {
 resourceGroupName,
 networkInterfaceIds: [networkInterface.id],
 size: "Standard_A0",
 sourceImageReference: {
 publisher: "Canonical",
 offer: "UbuntuServer",
 sku: "16.04-LTS",
 version: "latest",
 },
 osDisk: {
 caching: "ReadWrite",
 storageAccountType: "Standard_LRS",
 },
 computerName: "hostname",
 adminUsername: username,
 adminPassword: password,
 disablePasswordAuthentication: false,
);

Removing Deprecated Resources, Invokes, and Fields

Azure Active Directory now has its own Pulumi provider, so all the AD resources were removed from the AD namespace of the Azure Provider 2.0.

A number of other resources, invokes, and fields were removed too, following the changes in the upstream provider.

Default Versions in Callback Functions

Serverless as Callbacks running on Azure Functions have bumped the default version of the Azure Functions runtime to ~3 and Node.js to ~12. As always, you can override the default to set the versions that you require.

Callback Functions also moved from ZipBlob to Blob deploy as described above, so expect another replacement operation while upgrading.

Custom Timeouts

Some types of resources take longer to create than others. For instance, provisioning a new Cosmos DB account may take from 10 minutes to over an hour depending on the number of geo locations and other factors.

If the creation of a target resource times out, you can override the default timeout using resource options:

const cosmosdbAccount = new azure.cosmosdb.Account("cosmosdb", {
 resourceGroupName: resourceGroup.name,
 offerType: "Standard",
 geoLocations: manyLocations,
 consistencyPolicy: { consistencyLevel: "Session" },
}, {
 customTimeouts: {
 create: "30m",
 update: "60m",
 },
});

You shouldn’t need this ability often, but it can be a lifesaver when you do.

Explicit Adoption

Pulumi comes with a way to adopt existing cloud resources by specifying import options.

With the 1.x versions of the Azure provider, some resources could become adopted accidentally: if a name would match an existing resource, the upsert operation could succeed, and Pulumi would start managing the existing resource.

The 2.0 version of the provider is rigorous in checking the presence of existing resources with matching names before attempting to create new ones. This measure prevents inadvertent adoption and undesired side effects.

Getting Started with Migration

To get started with Azure provider 2.0, update the versions in your package manager:

// package.json
"@pulumi/azure": "^2.0.0",

// csproj/fsproj/vbproj
<PackageReference Include="Pulumi.Azure" Version="2.1.0-preview" />

# requirements.txt
pulumi-azure>=2.0.0

// Gopkg.toml
[constraint]]
 version = "v2.0.0"
 name = "github.com/pulumi/pulumi-azure"

There are quite a few breaking changes, so you may need to adjust your code before you can run the program successfully again.

To help with this process, we upgraded all our Azure templates and examples to 2.0.

If you want to stay with the 1.x version, please pin your version to ^1.0.0 or 1.14.0 in the package manager.

If you have issues or questions, reach out to us on Pulumi Community Slack, and we’ll help you through migration.

Deploy Serverless Containers on Google Cloud Run with Pulumi

Mikhail Shilkov — Tue, 04 Feb 2020 00:00:00 +0000

Google Cloud Run is the latest addition to the serverless compute family. While it may look similar to existing services of public cloud, the feature set makes Cloud Run unique:

Docker as a deployment package enables using any language, runtime, framework, or library that can respond to an HTTP request.
Automatic scaling, including scale to zero, means you pay for what you consume with no fixed cost and no management overhead.
HTTP load-balancing out of the box simplifies the usage.

Cloud Run is targeted very specifically at stateless web applications. It uses ephemeral containers, and each execution is limited to 15 minutes.

Today, we will deploy our first Cloud Run services with Pulumi. Then, we’ll discuss pricing and compare Cloud Run to the competition.

Hello Cloud Run

We’ll start by deploying a pre-built container image provided by Google to a Cloud Run service.

To follow along, create a new Pulumi project with the Get Started with Google Cloud guide.

Enable Cloud Run for the project

As with any other service of Google Cloud, you need to enable it for the target project before the first deployment. You can do so with the gcp.projects.Service resource:

const enableCloudRun = new gcp.projects.Service("EnableCloudRun", {
 service: "run.googleapis.com",
});

Note: If you destroy the Pulumi stack and delete the projects.Service resource, the Cloud Run service will be disabled again. If you use Cloud Run in multiple Pulumi stacks, you should move the service management to a central location, i.e., a shared stack.

Choose a location

Cloud Run is a regional service: all container instances run in a single location of your choice.

Set the region setting to one of the currently supported locations: us-central1 (Iowa), us-east1 (South Carolina), europe-west1( Belgium), or asia-northeast1 (Tokyo).

pulumi config set gcp:region <region>

Then, the program can read the value and reuse it for all resources:

// Location to deploy Cloud Run services
const location = gcp.config.region || "us-central1";

Deploy a Cloud Run service

Google provides a pre-deployed “Hello Cloud Run” image at gcr.io/cloudrun/hello. The following resource deploys that image to your GCP project with the default settings.

const helloService = new gcp.cloudrun.Service("hello", {
 location,
 template: {
 spec: {
 containers: [
 { image: "gcr.io/cloudrun/hello" },
 ],
 },
 },
}, { dependsOn: enableCloudRun });

Expose unrestricted HTTP access

By default, Google does not expose HTTP endpoints of a Cloud Run service to the Internet. To make it publicly available, you should grant the roles/run.invoker role to allUsers.

const iamHello = new gcp.cloudrun.IamMember("hello-everyone", {
 service: helloService.name,
 location,
 role: "roles/run.invoker",
 member: "allUsers",
});

Note that the Contributor role is required for the user who executes Pulumi deployment to grant the access.

Try it out

Each Cloud Run service automatically gets a run.app subdomain. You can export the exact URL as a Pulumi output.

// Export the URL
export const helloUrl = helloService.status.url;

Run pulumi up to deploy all the resources.

$ pulumi up
...
Outputs:
~ helloUrl: "https://hello-585ad15-q4wszdxb2a-ew.a.run.app"

Navigate to the URL to see the welcome screen:

Congratulations, your first Cloud Run service is up and running. Now, it’s time to deploy some custom code.

Deploy a Custom Application

Cloud Run can run any Docker container so that you can write the application code in a language of your choice. I picked Ruby as an example of a language that is not supported by Cloud Functions.

Create a Ruby web app

Let’s create an app subfolder and place all application files there. The first file app.rb is a Hello World web application in Ruby.

require 'sinatra'

set :bind, '0.0.0.0'

get '/' do
 target = ENV['TARGET'] || 'Pulumi'
 "Hello #{target}!\n"
end

Gemfile and Gemfile.lock files configure the required gems (packages).

Define a Docker image

Dockerfile defines the container image to deploy. It is based on a generic Ruby image, copies the application files, and runs the web application.

# Use the official lightweight Ruby image.
FROM ruby:2.7-slim

# Install production dependencies.
WORKDIR /usr/src/app
COPY Gemfile Gemfile.lock ./
ENV BUNDLE_FROZEN=true
RUN bundle install

# Copy local code to the container image.
COPY . ./

# Run the web service on container startup.
CMD ["ruby", "./app.rb"]

Note that nothing in this web application is specific to Cloud Run.

Build and publish the Docker image

Cloud Run can only deploy images from Google Cloud Registry (GCR). Therefore, our Pulumi program needs to build the Docker image with the sample Ruby application and push it to GCR.

Run gcloud auth configure-docker in your command line to configure your local Docker installation to use GCR endpoints.

Run npm i @pulumi/docker to install the Pulumi Docker SDK. Then, add the docker.Image resource to the Pulumi program.

const myImage = new docker.Image("ruby-image", {
 imageName: pulumi.interpolate`gcr.io/${gcp.config.project}/my-ruby-app:v1.0.0`,
 build: {
 context: "./app",
 },
});

Pulumi takes care of building the image in the app folder and uploading it to GCR.

Deploy to Cloud Run

Now, you can deploy another Cloud Run service and point it to the custom image.

const rubyService = new gcp.cloudrun.Service("ruby", {
 location,
 template: {
 spec: {
 containers: [{
 image: myImage.imageName,
 resources: {
 limits: {
 memory: "1Gi",
 },
 },
 }],
 containerConcurrency: 50,
 },
 },
}, { dependsOn: enableCloudRun });

This snippet illustrates two additional configuration values. The memory limit defines the amount of RAM available to each container. Container concurrency defines how many requests each container instance may process in parallel at any given time. Both parameters influence the price of the deployment, as explained in the Pricing section below.

Currently, it’s not possible to use partial vCPU or multiple vCPUs per instance: your instance has always one vCPU assigned. You can adjust the memory from 128Mb to 2Gb.

Finally, don’t forget to enable unrestricted access and export the public URL.

const iamRuby = new gcp.cloudrun.IamMember("ruby-everyone", {
 service: rubyService.name,
 location,
 role: "roles/run.invoker",
 member: "allUsers",
});

export const rubyUrl = rubyService.status.url;

After deploying the program, you should be able to access the application with an HTTP call.

$ curl "$(pulumi stack output rubyUrl)"
Hello Pulumi!

You can find the complete example in the Pulumi examples Github repository.

Pricing

Cloud Run pricing is entirely consumption-based. After you exhaust the free tier, you start paying for four components, combined:

Per CPU time ($24.00 / million seconds)
Per memory consumption ($2.50 / million GB-seconds)
Per request ($0.40 / million requests)
Egress traffic, same as any other Google Cloud usage

Both time-based measurements are rounded up to the nearest 100 ms. All the provisioned memory is charged, the actual Mb/Gb consumption doesn’t matter.

It’s important to understand that the CPU and memory metrics are calculated per active instance (host, provisioned container), not per request. Each Cloud Run instance can handle multiple requests concurrently. Overlapping executions aren’t double charged: billable time begins with the start of the first request and ends at the end of the last request.

The following picture illustrates the pricing for three executions running on the same container host.

Multiple requests can share the allocated CPU and memory, so it makes sense to set the concurrency setting as high as possible for a given application (but not higher). This model is a big difference to Cloud Functions, which are charged for each request independently.

Comparing Cloud Run to Other Services

Despite the existence of multiple services that look somewhat similar, Cloud Run is unique in its capabilities.

Google Cloud Run vs Functions

Google Cloud Functions (GCF) service deploys snippets of code as functions, while Cloud Run deploys a web application packaged as a Docker image. Currently, GCF only supports three runtimes (Node.js, Python, and Go), while Cloud Run can run practically any language and any runtime.

GCF has a notion of events and triggers: it can natively integrate with Pub/Sub, Cloud Storage, Cloud Firestore. Cloud Run is all about handling HTTP requests: Any connection to another service has to go via HTTP.

Google Cloud Run vs AWS Fargate

AWS Fargate deploys container images. It requires an ECS cluster to run on and imposes more configuration burden on the user, including networking, load balancing, auto-scaling, and service discovery. Pulumi Crosswalk for AWS can help with these tasks.

Fargate is capable of hosting long-running workloads. Therefore, Fargate’s scaling model is not tied to individual requests, and there is no scale-to-zero out of the box.

With Cloud Run, there’s no notion of a cluster to manage, the scaling and billing models are based on individual requests. However, the workloads are HTTP-only, with a maximum duration of 15 minutes per call.

Google Cloud Run vs Azure Container Instances

Azure Container Instances (ACI) can also run arbitrary containers and has built-in HTTP endpoints. However, there’s no auto-scaling: you get a single host for each instance. Also, there’s no load balancing capability across multiple instances.

Cloud Run is elastically scalable, which positions it much better to host applications with variable workloads.

Conclusion

Cloud Run is a service with a unique combination of serverless ease of use and pricing with the full power of container deployments. It is easier to manage than Fargate, has automatic scaling missing from ACI, and greater flexibility compared to GCF. With concurrent requests model, Cloud Run is well-positioned to be significantly cheaper than Cloud Functions for some scenarios.

If you are excited, follow the example to Get Started with Cloud Run and Pulumi.

Provisioned Concurrency: Avoiding Cold Starts in AWS Lambda

Mikhail Shilkov — Thu, 19 Dec 2019 00:00:00 +0000

AWS Lambda cold starts (the time it takes for AWS to assign a worker to a request) are a major frustration point of many serverless programmers. In this article, we will take a look at the problem of latency-critical serverless applications, and how Provisioned Concurrency impacts the status-quo.

Concurrency Model of AWS Lambda

Despite being serverless, AWS Lambda uses lightweight containers to process incoming requests. Every container, or worker, can process only a single request at any given time.

Overlapping executions land on separate workers

Because of this, the number of concurrent requests defines the number of required workers that a specific AWS Lambda function needs to serve a response at any given moment.

Cold Starts

How does AWS know how many workers it needs to run for a given function? Well, it doesn’t know in advance. AWS allocates new workers on-demand as the Lambda gets invoked.

Whenever Lambda receives a request but it has no idle workers, the control plane assigns a new generic worker to it. The worker then has to download the custom code or binaries of your Lambda and load them into memory before it can service the request. This process takes time, which significantly increases response latency.

The issue of sporadically slow responses caused by the need to increase the pool of workers is known as Cold Start. Cold starts are consistently the top concern about the applicability of serverless technologies to latency-sensitive workloads. There are numerous articles about the problem, including many articles I have written in the Cold Starts section on my website

Warming

Cold starts don’t occur for the majority of requests because AWS Lambda reuses workers between subsequent invocations. However, if a particular worker is idle for too long (usually, several minutes), AWS may decide to recycle and return it to the generic pool.

A trick known as Lambda Warmers uses kept-alive workers to reduce the frequency of cold starts. The idea is to have a CloudWatch timer that fires every few minutes and sends N parallel requests to the target Lambda. If all those requests land at the same time, AWS has to provision at least N workers to process them. The actual operation doesn’t have to do any useful work, but it resets the recycling timer back to zero.

There are a few drawbacks with this approach:

If a valid request comes at the same time as warming requests, it might hit a cold start.
Extra logic is needed in the Lambda code to detect warming requests and short-circuit them instead of doing useful work.
CloudWatch rules require additional setup and management.
The result is still best-effort: Lambda still recycles workers from time to time.

You can find more details about how Pulumi can help with some of these issues in AWS Lambda Warmer as Pulumi Component.

Provisioned Concurrency

At re:Invent 2019, AWS introduced Lambda Provisioned Concurrency—a feature to work around cold starts. The base concurrency model doesn’t change. However, you can request a given number of workers to be always-warm and dedicated to a specific Lambda.

Here is an example of configuring the provisioned concurrency with Pulumi in TypeScript:

const lambda = new aws.lambda.Function("mylambda", {
 code: new pulumi.asset.FileArchive("./app"),
 handler: "handler",
 role: role.arn,
 publish: true,
});

const fixed = new aws.lambda.ProvisionedConcurrencyConfig("fixed-concurrency", {
 functionName: lambda.name,
 qualifier: lambda.version,
 provisionedConcurrentExecutions: 2,
});

The snippet sets the provisioned concurrency for mylambda to a fixed value of 2. Note that concurrency is provisioned per Lambda version, and it can’t be set for the $LATEST version alias. Therefore, I set the publish attribute of my Lambda to true and reference a specific version of the Lambda in provisioning.

Dynamic Provisioned Concurrency

A fixed level of provisioned concurrency works well for stable workloads.

Fixed provisioned concurrency for uniform workloads

However, many workloads fluctuate a lot. Extreme elasticity and lack of configuration parameters have always been the essential benefits of AWS Lambda. It works great if you can tolerate the cold starts that come during scale-out. If not, you can explore more advanced scenarios for provisioning concurrency dynamically.

Instead of choosing a permanently fixed value, you can configure provisioned concurrency to autoscale. The first required component is the autoscaling target:

const resourceId = pulumi.interpolate`function:${lambda.name}:${lambda.version}`;
const target = new aws.appautoscaling.Target("target", {
 resourceId,
 serviceNamespace: "lambda",
 scalableDimension: "lambda:function:ProvisionedConcurrency",
 minCapacity: 1,
 maxCapacity: 10,
});

The second component is the autoscaling policy, which defines the conditions when scaling starts. There are two important ways to adapt your concurrency provisioning to traffic patterns.

Scheduled profile

Quite often, increases in request rates are partially predictable. For example, usage increases during business hours and decreases at night.

Provisioned concurrency matches predictable workload changes

The following snippet sets two scheduled rules that switch between two levels of provisioned concurrency every day.

function scheduledConcurrency(name: string, cron: string, capacity: number) {
 return new aws.appautoscaling.ScheduledAction(`schedule-${name}`, {
 resourceId,
 serviceNamespace: "lambda",
 scalableDimension: "lambda:function:ProvisionedConcurrency",
 scalableTargetAction: {
 minCapacity: capacity,
 maxCapacity: capacity,
 },
 schedule: `cron(${cron})`,
 }, { dependsOn: [target]});
}

scheduledConcurrency("day", "0 8 * * ? *", 10);
scheduledConcurrency("night", "0 18 * * ? *", 2);

The example defines a helper function and calls it twice to set two schedules with different parameters.

Autoscaling based on utilization

If your workload pattern is less predictable, you can configure autoscaling for provisioned concurrency based on the measured utilization.

Provisioned concurrency matches the variation in workload

Here is a basic example of a dynamic scaling policy.

const scaledConcurrency = new aws.appautoscaling.Policy("autoscaling", {
 resourceId,
 serviceNamespace: "lambda",
 scalableDimension: "lambda:function:ProvisionedConcurrency",
 policyType: "TargetTrackingScaling",
 targetTrackingScalingPolicyConfiguration: {
 targetValue: 0.8,
 predefinedMetricSpecification: {
 predefinedMetricType: "LambdaProvisionedConcurrencyUtilization",
 },
 },
}, { dependsOn: [target]});

Currently, there are issues with autoscaling based on the metrics, which makes provisioned concurrency scale less effectively than what the chart above shows. Hopefully, Amazon will fix these issues soon.

AWS Pricing

The pricing information in this section is outdated. For the most current AWS Lambda pricing, including Provisioned Concurrency costs, please refer to the official AWS Lambda pricing page.

While hand-crafted Lambda warmers are virtually free, provisioned concurrency can be costly. The pricing model for Provisioned Concurrency differs from the standard on-demand Lambda model: Instead of purely per-call billing, AWS charges per hour for provisioned capacity.

You pay for the Provisioned Concurrency you configure, even if a worker handled zero requests, plus a reduced rate for actual usage of those pre-provisioned instances.

Comparison of the cost of a 1GB worker for two billing models

The equilibrium point is at 60% utilization. Note that because the billed duration is rounded up to the nearest 100 ms for each execution, the utilization is not limited to 100%. A series of sequential executions can be processed by a single worker. If each execution is 10 ms, the charge is still 100 ms, and the total utilization can be as high as 1000% in terms of the chart above!

Finally, be careful to clean up your resources after any experiments: Leaving running workers with provisioned concurrency can be expensive! Using pulumi destroy removes resources after you finish experimenting.

Conclusion

Provisioned concurrency brings the long-awaited solution to cold starts in AWS Lambda. However, it comes at a cost in terms of a direct impact on billing and management overhead to provision concurrency at optimal levels.

For high-load functions where utilization is continuously above a known level of requests, it makes sense to set the provisioned concurrency to that level to save money and have the guarantee of warm workers.

If your Lambda hosts a latency-sensitive API, especially with runtimes like Java and .NET, you should strive to find the right balance between the percentage of requests that result in a cold start and the money spent on concurrency. Autoscaling should help once AWS has fixed the initial glitches that slipped into their current services.

If you want to try this for yourself, we’ve updated the Serverless App example to show off the scenario of configurable AWS Lambda provisioned concurrency.

Azure Functions on Kubernetes with KEDA

Mikhail Shilkov — Thu, 10 Oct 2019 00:00:00 +0000

Azure Functions is a managed service for serverless applications in the Azure cloud. More broadly, Azure Functions is a runtime with multiple hosting possibilities. KEDA (Kubernetes-based Event-Driven Autoscaling) is an emerging option to host this runtime in Kubernetes.

In the first part of this post, I compare KEDA with cloud-based scaling and outline the required components. In the second part, I define infrastructure as code to deploy a sample KEDA application to an Azure Kubernetes Service (AKS) cluster.

The result is a fully working example and a high-level idea of how it works. Kubernetes expertise is not required!

Autoscaling Primer

When you deploy an Azure Function, it runs within the Azure Functions runtime. The runtime is a host process which knows how to pull events from an event source (defined by the function trigger) and pass those to your function:

However, one instance of runtime rarely provides adequate processing capacity. If you only get one message per day, having an instance always running is wasteful. If you get thousands of events per second, one instance won’t be able to process all of them.

Automatic horizontal scaling solves the problem. At any point in time, several identical workers are crunching the events. The number N is optimized continuously to fit the current workload by adding new workers and removing underutilized ones.

Components of an automatically scaled application

Instance Provisioner is an extra component in the auto-scaled system. It monitors the stream of metrics from the event source and decides to add or remove workers. A massive standby pool of idle generic workers provides the workforce. Such a generic worker pulls the artifact of the assigned function, plugs it into the runtime, and starts processing events.

Azure Functions Consumption Plan

Consumption Plan is the serverless hosting option where Azure manages all the scaling components internally. Let’s consider an example of an Azure Function triggered by a Storage Queue:

Azure Functions Consumption Plan

The deployment artifact is just the code packaged as a zip archive and uploaded to Blob Storage. Scale Controller is an internal Azure component that observes the target queue and allocates Function App instances based on the queue length fluctuations. Each instance bootstraps itself with a zip file, connects to the queue, and pulls messages to process.

The cloud provider manages all the components of the system, so developers can focus on writing business logic code. It can be as simple as a JavaScript callback:

queue.onEvent("MyHandler", async (context, msg) => {
 console.log("New message: " + JSON.stringify(msg));
});

KEDA

Over the last few years, Kubernetes has gained traction across many industries. KEDA is provides a way to design and run event-driven applications inside a Kubernetes cluster. KEDA implements the autoscaling components in terms of Kubernetes tools.

The target Function App is packaged together with the Azure Functions runtime into a custom Docker image and published to a registry. A Kubernetes deployment utilizes that image and configures the parameters to connect it to the target event source (for instance, a queue):

The application can then run on one or many instances, or pods in Kubernetes terms. Kubernetes has a built-in component to scale out the pods: Horizontal Pod Autoscaler (HPA). By default, HPA makes scaling decisions based on the processor utilization of existing pods. CPU turns out to be a poor metric for event-driven applications: many workloads are not CPU-bound, so the scale-out won’t be aggressive enough to keep the queue empty. Therefore, KEDA introduces a ScaledObject—a custom resource which pulls metric values from the event source and feeds them as a custom metric to HPA:

Kubernetes-based Event-Driven Autoscaling

At the time of the initial KEDA preview announcement and until Kubernetes version 1.16, Horizontal Pod Autoscaler wasn’t able to scale a deployment down to zero pods. Therefore, KEDA includes an extra controller to disable the deployment if the event source is empty and re-enable it when new events come in. This duty might be delegated back to HPA in the future versions.

Now, when we know what KEDA is and how it works, it’s time to deploy a Function App!

Deploy a Function App with KEDA and Pulumi

Here is the list of the components required to run a Function App in Kubernetes with KEDA:

Azure Kubernetes Service cluster
Azure Container Registry
Helm chart keda-edge with KEDA service and custom resource definition
Azure Storage Queue as a sample event source
Docker image of the queue processor application
Deployment of this image
Scaled Object custom resource

All these components can be defined and deployed within a single Pulumi program. Below I highlight the main blocks of the program. Navigate to the following section to look at a short and reusable component.

AKS Cluster

KEDA can run on any Kubernetes cluster, but I choose to do so on managed AKS. My custom AksCluster component defines a cluster, including the required Active Directory and networking configuration. It makes multiple assumptions, so I only need to specify the main properties:

const resourceGroup = new azure.core.ResourceGroup("keda-sample");

const aks = new AksCluster("keda-cluster", {
 resourceGroupName: resourceGroup.name,
 kubernetesVersion: "1.13.5",
 vmSize: "Standard_B2s",
 vmCount: 3,
});

aks.cluster now has all the required output values, for instance, aks.cluster.kubeConfigRaw configuration.

Shared Cluster Components

The next group of components needs to be deployed just once.

Azure Container Registry is not part of the AKS cluster. Its purpose is to host Docker images of applications.

const registry = new azure.containerservice.Registry("registry", {
 resourceGroupName: args.resourceGroup.name,
 adminEnabled: true,
 sku: "Premium",
});

A Helm chart kedacore/keda-edge deploys the KEDA service and the ScaledObject custom resource definition.

const keda = new k8s.helm.v2.Chart("keda-edge", {
 repo: "kedacore",
 chart: "keda-edge",
 version: "0.0.1-2019.07.24.21.37.42-8ffd9a3",
 values: {
 logLevel: "debug",
 },
}, { providers: { kubernetes: aks.provider } });

Event Source

KEDA supports multiple types of event sources, and the current list is available here. An event source is supported if there is a scaler which knows how to pull metrics out of it and turn them into a custom metric for HPA.

My example uses an Azure Storage Queue as the event source:

const storageAccount = new azure.storage.Account("kedasa", {
 resourceGroupName: resourceGroup.name,
 accountTier: "Standard",
 accountReplicationType: "LRS",
});

const queue = new azure.storage.Queue("kedaqueue", {
 storageAccountName: storageAccount.name,
});

Docker Image

The same Pulumi program is capable of building a Docker image and uploading it to the Container Registry.

const dockerImage = new docker.Image("image", {
 imageName: pulumi.interpolate`${registry.loginServer}/${args.queue.name}:v1.0.0`,
 build: {
 context: "./functionapp",
 },
 registry: {
 server: registry.loginServer,
 username: registry.adminUsername,
 password: registry.adminPassword,
 },
});

The image refers to the folder with a Dockerfile in it and uses the registry variables to fill the credentials.

Deployment

Now, we can define a Deployment which uses the Docker image to run our Function App on Kubernetes pods.

const appLabels = { app: name };
const deployment = new k8s.apps.v1.Deployment(name, {
 apiVersion: "apps/v1",
 kind: "Deployment",
 metadata: {
 labels: appLabels,
 },
 spec: {
 selector: { matchLabels: appLabels },
 template: {
 metadata: {
 labels: appLabels,
 },
 spec: {
 containers: [{
 name,
 image: dockerImage.imageName,
 env: [{ name: "queuename", value: args.queue.name }],
 envFrom: [{ secretRef: {name: secretQueue.metadata.name } }],
 }],
 imagePullSecrets: [{ name: args.service.registrySecretName }],
 },
 },
 },
}, { provider: aks.provider });

The deployment refers to the secret value which stores the connection string to our target Storage Queue:

const secretQueue = new k8s.core.v1.Secret("queue-secret", {
 data: {
 queueConnectionString:
 args.storageAccount.primaryConnectionString.apply(c => Buffer.from(c).toString("base64")),
 },
}, { provider: aks.provider });

Scaled Object

Finally, we need to deploy an instance of the ScaledObject custom resource, which takes care of feeding the metrics from the queue to the Horizontal Pod Autoscaler.

const scaledObject = new k8s.apiextensions.CustomResource("scaledobject", {
 apiVersion: "keda.k8s.io/v1alpha1",
 kind: "ScaledObject",
 metadata: {
 labels: { deploymentName: name },
 },
 spec: {
 scaleTargetRef: { deploymentName: name },
 triggers: [{
 type: "azure-queue",
 metadata: {
 type: "queueTrigger",
 connection: "queueConnectionString",
 queueName: args.queue.name,
 name: "myQueueItem",
 },
 }],
 },
}, { provider: aks.provider });

Reusable Components

We can simplify the program further by creating reusable Pulumi components for a Cluster, a Service, and a Function app. After those are done, here is all the code required to deploy a Kubernetes cluster with a Function App and KEDA.

// Create an AKS K8s cluster
const aks = new AksCluster("keda-cluster", {
 resourceGroupName: resourceGroup.name,
 kubernetesVersion: "1.13.5",
 vmSize: "Standard_B2s",
 vmCount: 3,
});

// Deploy shared components of KEDA (container registry, kedacore/keda-edge Helm chart)
const service = new KedaService("keda-edge", {
 resourceGroup,
 k8sProvider: aks.provider,
});

// Deploy a Function App which subscribes to the Storage Queue
const app = new KedaStorageQueueHandler("queue-handler", {
 resourceGroup,
 storageAccount,
 queue,
 service,
 path: "./functionapp",
});

You can find the full code of the components, the sample program, and steps to run it in this example.

Conclusion

If your company is betting on Kubernetes as the standard application platform and you see high value in event-driven fine-grained applications, KEDA might be an exciting option for the future. As of March 31, 2020, Kubernetes Event-driven Autoscaling (KEDA) has joined the CNCF as an official sandbox project.

Pulumi is a great tool to get your KEDA application running quickly and effortlessly. You can combine Docker image creation, Kubernetes cluster provisioning, Helm chart installation, and KEDA deployment in a single program, written in your favorite general-purpose language.

Get started with Azure Kubernetes Service (AKS) Cluster and Azure Functions with KEDA.

How To Build Globally Distributed Apps with Azure Cosmos DB

Mikhail Shilkov — Tue, 24 Sep 2019 00:00:00 +0000

In a previous blog post, I shared how easy it is to create a globally distributed, highly-available, low-latency application with Azure Functions, Azure Cosmos DB, and Pulumi.

Today, I want to show how the same approach can be generalized for any cloud compute service, including containers, virtual machines, and serverless functions.

E-commerce Example

Let’s consider an example of an e-commerce website which targets a worldwide audience. The architecture of such a website should address several challenges.

Multiple Teams

A large e-commerce website is a sophisticated software application and consists of many subcomponents. Many engineers in multiple teams work on it simultaneously.

A typical approach is to organize engineers into smaller teams and make each team autonomous and responsible for one or several business domains. This strategy leads to microservices architecture when the application is built out of loosely coupled components. Each component represents a “vertical slice”: a team owns functionality across the vertical stack, from its user interface and service API to the data storage layer and underlying infrastructure.

Here is an example of such a breakdown:

Multiple components on a single web page

Diversity of Technology Decisions

One of the guiding principles of microservices architecture is that each team makes decisions regarding which tools and technologies to use. They are free to choose programming languages, databases, or cloud services that are the best fit for the service. In practice, some restrictions do apply—for instance, the teams might need to share the same cloud provider.

In our hypothetical example, the whole product is hosted on Azure cloud, however other aspects differ:

Product Page team runs JavaScript serverless functions and stores data in a document database
Shopping Cart team uses C# and .NET Core, packages the code in a Docker container, and uses event sourcing to store every action ever made by users while shopping
Pricing Engine team has a mix of Java and C++ code with a bunch of third-party libraries and deploys the services to Ubuntu VMs that use MongoDB drivers for storage

Global Presence

Regardless of the tech, the user-facing services must serve customers across the globe and serve them fast. This requirement demands a geo-distributed infrastructure so that user requests could be handled by the nearest data center to minimize the latency and ensure excellent user experience.

Each team has to solve the following problems in a way which fits their tech stack:

Distribute the data across the globe
Run their compute workload in each region next to the data
Manage all this infrastructure in a reliable way

Let’s consider these goals and possible solutions.

Azure Cosmos DB

Distributing data across many data centers while keeping them consistent and available is one of the hardest problems in computer science and engineering. Delegating this challenge to a specialized service or product is usually a sensible idea.

Azure Cosmos DB is the database of choice for global applications running in the Azure cloud. Cosmos DB provides a turn-key global distribution to any number of regions worldwide. Regions can be added or removed along the way while running production workloads and without having any impact on high availability. The accounts can be provisioned with a single write region or in multi-master mode with every region being writable, thus enabling writes that are local to each region.

Azure Cosmos DB with multiple locations

Cosmos DB supports multiple models of data: key-value, document, column family, graph, with wire-protocol compatible APIs for MongoDB, Apache Cassandra, SQL, etcd, Gremlin, etc. It also supports multiple well-defined and practical consistency levels, from eventual to strong consistency, so that each application can strike the right balance between latency, availability, and consistency guarantees. All of this is offered as a fully-managed service, with the comprehensive SLAs covering latency, throughput, consistency, and 99.999% high availability for both reads and writes.

Sounds great, but how do we build applications to leverage this power?

Distributed Web Applications

In contrast to Cosmos accounts, compute services in Azure are located in a single region. This holds true for Virtual Machines, Container Instances, Serverless Functions, and managed Azure Kubernetes Service.

If an application is deployed to a single location, we are not making good use of geographic redundancy of the database. If a user happens to be far from the application region, their request has to travel across the globe. Moreover, the application connects to the nearest region of Cosmos DB, while all other regions stay idle:

Application deployed in a single location

Instead, we should deploy a copy of application infrastructure to each of the target regions:

Application deployed in multiple locations

Each application instance has to be configured to connect to the endpoint of the Cosmos DB in the same region to enjoy local single-digit-milliseconds latency for reads and writes.

While beneficial to end-users, such setup brings much extra complexity in terms of infrastructure management:

Application regions must stay in sync with Cosmos DB regions, including the correct configuration of preferred locations.
With the growth of the number of regions, the traditional capacity planning gets harder and harder. This makes a dynamic auto-scaling configuration increasingly important.
A central routing service must be configured to flow the traffic from end-users to the closest application instance.
Each team must execute all these tasks, independently but coherently.

Let’s see how infrastructure as code and a general-purpose programming language come to the rescue.

Search for a Reusable Abstraction

In our example, three teams have a common goal of providing high-quality service to customers around the world, but they differ in the compute services they want to use.

Product Card team relies on serverless functions, so their infrastructure may look like this:

Global application with Azure Functions

Shopping Cart team packages their code as a Docker container, which they put into Azure Container Registry, and then use Azure Container Instances to run the application. The Registry is a global Azure resource, while Container Instances are deployed separately per region.

Global application with Azure Container Instances

Pricing Engine team runs their code on Infrastructure-as-a-Service (IaaS), so they have a task of creating Virtual Machine images, and then deploying those to an Azure Virtual Machine Scale Set, fronting it with a Load Balancer, and configuring all the networking infrastructure including Virtual Networks, Subnets, Public IPs, Network Security Groups, etc. Once again, this has to be done for each location independently.

Global application with Virtual Machines

Each team has the same global structure of infrastructure, including a Traffic Manager front-end and a Cosmos DB data store. At the same time, the per-region compute infrastructure differs substantially.

Instead of copying the common pieces of infrastructure between the teams, we can think of a reusable abstraction which could be shared by every service.

Global application pattern

For this blog post, I created a custom Pulumi component called CosmosApp which implements such abstraction and simplifies the provisioning of global web applications. The component creates distributed Cosmos DB resources, as well as the front-end routing component, while allowing pluggable compute layer implementation.

Cosmos App with Azure Functions

Let’s look at an example of how the component can be used. Creating a globally-distributed application requires three steps:

List the Regions

We define a configuration value to contain a comma-separated list of Azure regions that we’ll deploy the application to.

$ pulumi config set locations WestUS,WestEurope,SouthEastAsia,SouthBrazil,AustraliaCentral

This value can be loaded and parsed into a JavaScript array:

const locations = new pulumi.Config().require("locations").split(",");

Build a Region Template

We define a factory function to create the infrastructure in each location. Its signature is a bit unusual in that we return a function from another function. I’ll explain the motivation in the next example. For now, it’s important to understand the following:

We get a Cosmos account and a location as input arguments
We create a Function App in the specified location with connection settings pointing to the local Cosmos DB replica
We return the identifier of the Function App so that a Traffic Manager endpoint could point to it

Here is the function:

function buildProductApp({ cosmosAccount }: GlobalContext) {
 return ({ location }: RegionalContext) => {
 const app = new azure.appservice.ArchiveFunctionApp("function-app", {
 resourceGroupName: resourceGroup.name,
 location,
 archive: new pulumi.asset.FileArchive("./app"),
 appSettings: {
 COSMOSDB_ENDPOINT: cosmosAccount.endpoint,
 COSMOSDB_KEY: cosmosAccount.primaryMasterKey,
 COSMOSDB_LOCATION: location,
 },
 });

 return {
 id: app.functionApp.id,
 };
 };
}

Instantiate a Cosmos App

Finally, we use the buildProductApp factory function to create the geo-distributed application:

const products = new CosmosApp("products", {
 resourceGroup,
 locations,
 databaseName: "productsdb",
 containerName: "products",
 factory: buildProductApp,
});

[ Full Example ]

Cosmos App with Container Instances

The Shopping Cart team will have to follow the same steps but define their compute infrastructure accordingly.

A significant difference to the Functions-based example is the fact that the Container-based application needs some infrastructure to be shared across regions. The top block of the factory function defines an instance of Azure Container Registry, builds a Docker image, and puts that image into the Registry:

function buildShoppingCartApp({ cosmosAccount, database, container }: GlobalContext) {

 const registry = new azure.containerservice.Registry("global", {
 resourceGroupName: resourceGroup.name,
 adminEnabled: true,
 sku: "Premium",
 }, opts);

 const dockerImage = new docker.Image("node-app", {
 imageName: pulumi.interpolate`${registry.loginServer}/mynodeapp:v1.0.0`,
 build: {
 context: "./container",
 },
 registry: {
 server: registry.loginServer,
 username: registry.adminUsername,
 password: registry.adminPassword,
 },
 });

 return ({ location }: RegionalContext) => {
 // ... see the next sample
 };
}

Now, the factory function that builds per-region infrastructure can directly reference the Registry and the image to deploy an Azure Container Instances group. Note how registry and dockerImage are used as any other TypeScript variable inside the closure factory function:

function buildShoppingCartApp({ cosmosAccount, database, container }: GlobalContext) {

 const registry = new azure.containerservice.Registry(/* ... */);
 const dockerImage = new docker.Image(/* ... */);

 return ({ location }: RegionalContext) => {
 const group = new azure.containerservice.Group(`aci-${location}`, {
 resourceGroupName: resourceGroup.name,
 location,
 imageRegistryCredentials: [{
 server: registry.loginServer,
 username: registry.adminUsername,
 password: registry.adminPassword,
 }],
 osType: "Linux",
 containers: [
 {
 cpu: 0.5,
 image: dockerImage.imageName,
 memory: 1.5,
 name: "web-server",
 ports: [{
 port: 80,
 protocol: "TCP",
 }],
 environmentVariables: {
 ENDPOINT: cosmosAccount.endpoint,
 MASTER_KEY: cosmosAccount.primaryMasterKey,
 DATABASE: database.name,
 COLLECTION: container.name,
 LOCATION: location,
 },
 },
 ],
 ipAddressType: "public",
 dnsNameLabel: `acishop-${location}`,
 });

 return {
 url: group.fqdn,
 };
 };
}

Finally, a CosmosApp is defined:

const aci = new CosmosApp("aci", {
 resourceGroup,
 locations,
 databaseName: "cartdb",
 containerName: "items",
 factory: buildShoppingCartApp,
 enableMultiMaster: true,
});

Shopping Cart workload expects both reads and writes from the end-users, so the team decides to enable multi-master support in their Cosmos DB instance.

[ Full Example ]

Cosmos App with Virtual Machine Scale Sets

The Pricing Engine team has to go through the same steps but employ IaaS-based infrastructure to run application code. The factory function creates a dozen resources related to virtual machines, load balancing, networking, and auto-scaling. You may define as many resources as needed in the code, as long as it returns a pointer to the proper resource to link the endpoint.

function buildVMScaleSetApp({ cosmosAccount, database }: GlobalContext) {

 // Build a VM image here...

 // Define a MongoDB compatible collection
 const collection = new azure.cosmosdb.MongoCollection(`mongo-${name}`, {
 resourceGroupName: resourceGroup.name,
 accountName: cosmosAccount.name,
 databaseName: database.name,
 });

 return ({ location }: RegionalContext) => {

 const publicIp = new azure.network.PublicIp(/* ... */);
 const loadBalancer = new azure.lb.LoadBalancer(/* ... */);
 const bpepool = new azure.lb.BackendAddressPool(/* ... */);
 const probe = new azure.lb.Probe(/* ... */);
 const rule = new azure.lb.Rule(/* ... */);
 const vnet = new azure.network.VirtualNetwork(/* ... */);
 const subnet = new azure.network.Subnet(/* ... */);
 const scaleSet = new azure.compute.ScaleSet(/* ... */);
 const autoscale = new azure.monitoring.AutoscaleSetting(/* ... */);

 return {
 id: publicIp.id,
 };
 };
}

The code also uses a collection with the MongoDB protocol compatibility because that’s what the application expects.

[ Full Example ]

Conclusion

Azure Cloud Platform offers an amazing array of capabilities for application developers. Azure Cosmos DB takes care of distributing the data globally, reliably, consistently, and with low-latency access anywhere around the world. Nonetheless, developers still need to take care of adequately provisioning the application layer for efficiently serving users around the globe.

Cosmos App component is an excellent example of the power of using a general-purpose programming language to define cloud infrastructure. Such a component can let you codify the best practices and let multiple teams or companies share the implementation. The component and applications can evolve independently, with tools like versioning, package managers, compilers, and automated tests helping to maintain high-quality standards.

Ten Pearls With Azure Functions in Pulumi

Mikhail Shilkov — Wed, 21 Aug 2019 00:00:00 +0000

In this post, we’ll take a look at 10 “pearls”—bite-sized code snippets—that demonstrate using Pulumi to build serverless applications with Azure Functions and infrastructure as code. These pearls are organized into four categories, each demonstrating a unique scenario:

Function App Deployment: Deploy an existing Azure Functions application using infrastructure as code.
HTTP Functions as Callbacks: Mix JavaScript or TypeScript functions with your infrastructure definition to produce strongly-typed, self-contained, serverless HTTP endpoints.
Cloud Event Handling: Leverage a variety of event sources available to Azure Functions with lightweight event handlers.
Data Flows with Function Bindings: Take advantage of function bindings—declarative connectors to Azure services.

Our Azure Functions pearls are:

Function App Deployment

Deploy a Function App written in .NET or any other supported runtime
Configure Functions to run on an Elastic Premium Plan

HTTP Functions as Callbacks

Define Node.js Functions as inline callbacks
Implement REST APIs as multiple Functions
“Warm” the Functions to avoid Cold Starts

Cloud Event Handling

Process events from Azure Event Hub
Subscribe to Blob creation with Azure Event Grid
Run a Function every time an Azure resource is modified

Data Flows with Function Bindings

Push a message to a Storage Queue with an output binding
Pull a Storage Table row with an input binding

Function App Deployment

Azure Functions can be written in many languages, and Pulumi supports whatever choice you make. You can take any existing serverless application and deploy it using infrastructure-as-code.

1. Deploy a .NET Function App

[ Runnable Example ]

Many Function Apps are .NET applications created with native tooling like Visual Studio or the Functions CLI. Pulumi can simply deploy such an application in only a few lines of JavaScript:

const dotnetApp = new azure.appservice.ArchiveFunctionApp("http-dotnet", {
 resourceGroupName: resourceGroup.name,
 archive: new pulumi.asset.FileArchive("./app/bin/Debug/netcoreapp2.1/publish"),
 appSettings: { runtime: ".NET" },
});

There are only four things required:

the Function App name (“http-dotnet”)
the resource group it belongs to
the path to the compiled .NET assemblies, and
the desired runtime

Pulumi takes care of the rest for you. It handles the following tasks:

Creating a Storage Account and a Blob Container
Zipping up the binaries and uploading them to the blob container
Calculating a SAS token
Preparing a Consumption Plan and a Function App using this Consumption Plan
Configuring the required application settings, including a reference to the zip archive with the SAS token

While a few values are required, you are not restricted to the default behavior and can change any setting.

2. Run Functions using an Elastic Premium Plan

While the Consumption Plan is the ultimate serverless option, there are other ways to host Azure Functions. Azure also offers the Premium Plan: a combination of the power and guarantees of a fixed App Service Plan with the elasticity of Consumption.

If you want to take advantage of a Premium Plan, go ahead and define it as a Pulumi resource, then link to it in the Function App definition:

const premiumPlan = new azure.appservice.Plan("my-premium", {
 resourceGroupName: resourceGroup.name,
 sku: {
 tier: "Premium",
 size: "EP1",
 },
 maximumElasticWorkerCount: 20,
});

const javaApp = new azure.appservice.ArchiveFunctionApp("http-java", {
 resourceGroupName: resourceGroup.name,
 plan: premiumPlan,
 archive: new pulumi.asset.FileArchive("./java/target/functions/fabrikam"),
 appSettings: { runtime: "java" },
});

In this example, I deployed a Java application. The Premium Plan has a couple of configuration knobs: the instance size and the maximum scale-out limit.

HTTP Functions as Callbacks

Node.js is another runtime supported by Azure Functions. Like before, you can use the ArchiveFunctionApp class to deploy the application from an external folder. However, Pulumi’s Node.js SDK provides a way to mix the code of your functions directly into your infrastructure definition.

3. Define Node.js Functions as Inline Callbacks

[ Runnable Example ]

A TypeScript or a JavaScript function becomes an Azure Function deployed to the cloud:

const greeting = new azure.appservice.HttpEventSubscription('greeting', {
 resourceGroupName: resourceGroup.name,
 callback: async (context, req) => {
 return {
 status: 200,
 body: `Hello ${req.query['name'] || 'World'}!`,
 };
 }
});

export const url = greeting.url;

You are free to use any NPM module inside the callback, and the dependencies will be automatically packaged into the deployment artifact. Alternatively, you can extract the callback function into a separate file or package and import it from your infrastructure code.

While mixing infrastructure and application code in the same file may seem counterintuitive, it provides many benefits:

Combined code binaries (data plane) and infrastructure (control plane) as a single unit of deployment
Eliminate the need for boilerplate configuration like host.json and function.json files
Robust typing out-of-the-box: For instance, you can flawlessly “dot into” the content and req object above.

You can read more about the motivation in Serverless as Simple Callbacks with Pulumi and Azure Functions.

The previous example deployed a Function App with a single Function. However, Azure supports applications with multiple Functions bundled together.

4. REST APIs as Multiple Functions

It’s common to use Azure Functions for implementing RESTful APIs. As an API may consist of multiple endpoints, we may need to combine several HTTP Functions into a single deployment unit.

To achieve that, we define an HttpFunction object per Azure Function, each with its callback and settings. Then, we pass an array of these objects into a MultiCallbackFunctionApp constructor. Each Function definition may have a specific route and HTTP methods to handle:

const get = new azure.appservice.HttpFunction("Read", {
 route: "items",
 methods: ["GET"],
 callback: async (context, request) => {
 const items = await repository.list();
 return { status: 200, body: items };
 },
});

const post = new azure.appservice.HttpFunction("Add", {
 route: "items",
 methods: ["POST"],
 callback: async (context, request) => {
 const id = await repository.add(request.body);
 return { status: 201, body: { id } };
 },
});

const app = new azure.appservice.MultiCallbackFunctionApp("multi-app", {
 resourceGroupName: resourceGroup.name,
 functions: [get, post],
});

You can combine as many Functions as you want, including functions of different types, as shown in the next example.

5. Function “Warming” with a Timer Function

[ Runnable Example ]

Scheduled jobs are another frequent use case for serverless functions. It’s possible to define a cron expression and get the code executed at designated intervals.

The Consumption Plan disposes a worker if no Function runs in about 20 minutes. After disposal, the next execution will cause a cold start, and the response latency, will increase.

In order to avoid this disposal, we can combine a target HTTP Function with a Timer Function. The body of the Timer Function is empty, since its sole purpose is to trigger the Function App and keep the worker “warm”:

const warmer = new azure.appservice.TimerFunction("warmer", {
 schedule: "0 */5 * * * *",
 callback: async () => {},
});

const http = new azure.appservice.HttpFunction("hello", {
 callback: async (context, req) => {
 return {
 status: 200,
 body: "Hello World!",
 };
 }
});

const app = new azure.appservice.MultiCallbackFunctionApp("always-warm-app", {
 resourceGroupName: resourceGroup.name,
 functions: [http, warmer],
});

It’s easy to imagine a custom component WarmedFunctionApp which appends a standard Timer Function to an array of Functions passed to its constructor.

Cloud Event Handling

While HTTP is a widespread use case, Azure Functions support many other trigger types too. The previous post featured Storage Queues and ServiceBus Topics. Pulumi supports Timers, Events Hubs, Event Grid, Storage Blobs, Service Bus Queues, and Cosmos DB Change Feed events, too! Let’s see how that looks using Azure Event Hubs.

6. Process Events from Azure Event Hubs

[ Runnable Example ]

Azure Event Hubs is a fully-managed log-based messaging service comparable to Apache Kafka. In contrast to a self-hosted Kafka cluster, it only takes a few lines of JavaScript to create an Event Hub and start processing events:

const eventHub = new eventhub.EventHub("my-hub", {
 resourceGroupName: resourceGroup.name,
 namespaceName: namespace.name,
 partitionCount: 2,
 messageRetention: 7,
});

eventHub.onEvent("MyHubEvent", async (context, msg) => {
 console.log("Event Hub message: " + JSON.stringify(msg));
});

Every time a new event comes in, be it once per hour or a thousand times a second, the Function is executed. Azure manages the scale-out for you.

[ Runnable Example ]

Azure Event Grid is another trigger type for Azure Functions. It is a dispatcher service for distributing events from many other Azure services as well as external data sources.

A classic example is subscribing to events from Azure Blob Storage. The Event Grid subscription connects to a given Storage Account and provides several handy options to filter the event stream. The following example subscribes to all JPG files created in any container of the account:

const storageAccount = new azure.storage.Account("eventgridsa", {
 resourceGroupName: resourceGroup.name,
 accountReplicationType: "LRS",
 accountTier: "Standard",
 accountKind: "StorageV2",
});

azure.eventgrid.events.onGridBlobCreated("OnNewBlob", {
 storageAccount,
 subjectFilter: {
 caseSensitive: false,
 subjectEndsWith: ".jpg",
 },
 callback: async (context, event) => {
 context.log(`Subject: ${event.subject}`);
 context.log(`File size: ${event.data.contentLength}`);
 },
});

The event object is strongly typed: The snippet above logs the file size, but there are many other properties at your disposal. Code completion makes the discovery process painless.

It’s worth mentioning that Pulumi does much of the work behind the scenes here, allowing you to focus on the important parts of the task at hand. In particular, Pulumi retrieves the appropriate secret key from Azure Functions ARM API and creates an Event Grid subscription which points to a specific webhook containing that key.

8. Respond to Resource-level Events

[ Runnable Example ]

Here is a good illustration of the power of Event Grid. A callback function gets triggered for each change to any resource belonging to the target Resource Group:

const resourceGroup = new azure.core.ResourceGroup("eventgrid-rg");

eventgrid.events.onResourceGroupEvent("OnResourceChange", {
 resourceGroupName: resourceGroup.name,
 callback: async (context, event) => {
 context.log(`Subject: ${event.subject}`);
 context.log(`Event Type: ${event.eventType}`);
 context.log(`Data: ${JSON.stringify(event.data)}`);
 },
});

This simple piece of code is an easy launching point for many automation, auditing, and governance scenarios.

Data Flows with Function Bindings

Azure Functions come with a powerful system of bindings. So far, we only saw examples of using event sources as trigger bindings. However, Azure also supports a powerful set of so called “input” and “output” bindings.

9. Output Bindings

[ Runnable Example ]

Output bindings enable developers to easily forward the data from an Azure Function to an arbitrary destination in a declarative manner. For instance, if a queue handler needs to send a message to another queue, we don’t have to use cloud SDKs. Instead, we can return the message-to-be-sent from the callback and wire it to the output queue. Here is a quick example:

const incoming = new azure.storage.Queue("queue-in", {
 storageAccountName: storageAccount.name,
});

const outgoing = new azure.storage.Queue("queue-out", {
 storageAccountName: storageAccount.name,
});

incoming.onEvent("NewMessage", {
 outputs: [outgoing.output("queueOut")],
 callback: async (context, person) => {
 return {
 queueOut: `${person.name} logged into the system`,
 };
 },
});

Two elements play together here:

The outputs property defines an output binding with the name queueOut and the destination to outgoing
The queueOut property of the resulting object contains the output message

Note that the binding name must match the output property.

10. Input Bindings

[ Runnable Example ]

Input bindings pull extra bits of information and pass them as input parameters to the callback. The exact usage depends on the trigger and binding types and might be tricky to get right with JSON configuration files. Here is one example of wiring done in a Pulumi program:

const values = new azure.storage.Table("values", {
 storageAccountName: storageAccount.name,
});

const getFunc = new azure.appservice.HttpEventSubscription('get-value', {
 resourceGroupName: resourceGroup.name,
 route: "{key}",
 inputs: [
 values.input("entry", { partitionKey: "lookup", rowKey: "{key}" }),
 ],
 callback: async (context, request, entry) => {
 return {
 status: 200,
 body: entry.value,
 };
 },
});

There are three crucial bits here:

The route property of this HTTP Function contains a template parameter key. The runtime extracts the actual key value from the HTTP request.
The inputs option contains a reference to a Storage Table with a hardcoded partitionKey and a rowKey bound to the key template parameter. At execution time, a row is retrieved based on the combination of the keys. The entry, if found, is passed to the callback.
The callback has three input parameters, while all previous examples had two. The third parameter contains the retrieved row.

It’s possible to have multiple input and output bindings, and any combination of those.

Wrapping Up

In this post, we’ve seen some of the exciting things you can do with Azure Functions in Pulumi. Developers use serverless functions as a glue between managed cloud services. Pulumi offers a compelling way to define the links between these pieces of cloud infrastructure in a simple and expressive way.

For a streamlined Pulumi walkthrough, including language runtime installation and cloud configuration, see the Azure Quickstart, check more examples at Pulumi Azure GitHub, and join the Pulumi Community Slack.

7 Ways to Deal with Application Secrets in Azure

Mikhail Shilkov — Fri, 26 Jul 2019 00:00:00 +0000

Every non-trivial application relies on configuration values that may depend on the current execution environment. Some of these values contain sensitive information that shouldn’t be shared publicly. In general, the fewer parties that have access to those secret values, the safer the application will be—in fact, in an ideal world, no one would be granted direct access to those secrets.

Examples of secret configuration values include:

A connection string to a message bus or a database
A SAS Token to an Azure Storage account
An access key for a third-party service

There’s no one universal way to manage secrets, as a lot depends on the context in which they are used. In this article, I go through seven ways to use secret values in a .NET Core application running in Azure. I start with naively hard-coded strings and build up from there to more secure options.

While the concepts are universally applicable, my code samples focus on a .NET application running in Azure App Service and configured with Pulumi. Consider them best practices for managing client secrets in Azure.

1. Hard-Coded Application Secrets

Whenever you want to try a new API requiring a secret access token, it’s natural to copy-paste that secret into your code and run it—simply to make sure that the setup works.

For example, you are testing integration with a payment service:

var apiKey = "payment-service-key";
var paymentService = new PaymentService(apiKey);

or you want to start sending messages to an Azure Storage Queue:

var connectionString = "DefaultEndpointsProtocol=https;AccountName=account-name;AccountKey=account-key";
var storageAccount = CloudStorageAccount.Parse(connectionString);
var queueClient = storageAccount.CreateCloudQueueClient();

Both snippets are fine for a Hello-World application with a lifespan of two hours. However, I would strongly discourage doing so in any code that can potentially be checked into a source control system, even for a 10-minute experiment. Don’t copy-paste secrets into files which are part of a git repository. One accidental git commit & git push—and the secrets are compromised.

Avoid the “Secrets as Code” practice: there are bots scanning your GitHub repository for those.

2. Configuration Files

Traditionally, putting secrets in a configuration file is considered more secure. For classic .NET applications, this would be an app.config or a web.config file. The idea is that the values on a developer machine are different from the values in the production environment. Non-sensitive development settings are kept on disk and maybe in source control, while the real secrets are only injected by a deployment script or a CI/CD system, so they are not exposed publicly.

In the .NET Core world, a configuration is usually stored in an appsettings.json file:

{
 "MyConfig": {
 "PaymentApiKey": "payment-service-key",
 "StorageConnectionString": "DefaultEndpointsProtocol=https;AccountName=account-name;AccountKey=account-key"
 }
}

which is then mapped to a plain C# object in code:

public class MyConfig
{
 public string PaymentApiKey { get; set; }
 public string StorageConnectionString { get; set; }
}

With some configuration not shown here, the properties are filled at startup time and can be used in the code as a plain object:

var apiKey = config.PaymentApiKey;
var paymentService = new PaymentService(apiKey);
// ...

var connectionString = config.StorageConnectionString;
var storageAccount = CloudStorageAccount.Parse(connectionString);
var queueClient = storageAccount.CreateCloudQueueClient();

In my experience, such setup still poses substantial risks. appsettings.production.json may still be accidentally checked into the source control. Developers tend to use real cloud resources for their local and test environments, which do contain sensitive information that can be exploited when leaked.

On top of that, it doesn’t feel right to mix configuration—how many threads to run, or how big the message batches should be—with secret connection strings and API keys in the same file. These are separate kinds of configuration, and they warrant different workflows.

3. Environment Variables and Application Settings

One alternative approach is to read the secret values from environment variables. .NET Core configuration system can parse environment variables instead of, or in addition to the settings files. Likewise, one could read such values with a one-liner in C#:

var apiKey = Environment.GetEnvironmentVariable("PAYMENT_API_KEY");
// ...

var connectionString = Environment.GetEnvironmentVariable("STORAGE_CONNECTION_STRING");
// ...

Your CI/CD system should inject those values as part of the deployment pipeline.

The App Service gives you the ability to set environment variables via Application Settings. Here is a snippet of a Pulumi program that passes both secret values to appSettings:

const cfg = new pulumi.Config();
const paymentApiKey = cfg.requireSecret("paymentApiKey");

const storageAccount = new azure.storage.Account("storage", {
 resourceGroupName: resourceGroup.name,
 accountReplicationType: "LRS",
 accountTier: "Standard",
});

const app = new azure.appservice.AppService("app", {
 resourceGroupName: resourceGroup.name,
 appServicePlanId: appServicePlan.id,
 appSettings: {
 "PAYMENT_API_KEY": paymentApiKey,
 "STORAGE_CONNECTION_STRING": storageAccount.primaryConnectionString,
 },
});

The Storage connection string is produced by the Pulumi program directly, so it doesn’t have to be placed anywhere outside the program itself.

The payment service key is provided by a third party, so its encrypted value is stored in Pulumi configuration. Read Managing Secrets with Pulumi to learn about security options available for secrets in Pulumi config.

4. Azure Key Vault

In the previous example, both secrets end up in Application Settings. Every person with sufficient permissions may go to the App Service and see them in clear text. While this can be restricted, it’s a good idea to grant full read access to the application developers and operators of the App Service to give them the full picture when troubleshooting issues.

In some cases, compliance to a certain standard may require the use of a certified key management service offering enhanced security for sensitive secrets.

Azure has a dedicated service for storing secrets, Azure Key Vault. You can create and populate a Key Vault with all the secrets from the same Pulumi program:

const vault = new azure.keyvault.KeyVault("vault", {
 resourceGroupName: resourceGroup.name,
 sku: {
 name: "standard",
 },
 tenantId: tenantId,
 accessPolicies: [{
 tenantId,
 // The current principal has to be granted permissions to Key Vault so that it can actually add and then remove
 // secrets to/from the Key Vault. Otherwise, 'pulumi up' and 'pulumi destroy' operations will fail.
 objectId: currentPrincipal,
 secretPermissions: ["delete", "get", "list", "set"],
 }],
});

const secret = new azure.keyvault.Secret("paymentApiKey", {
 keyVaultId: vault.id,
 value: paymentApiKey,
});

Additionally, one or many Service Principals (SP) should be configured to access the Key Vault. One SP which has been granted secret management access deploys the infrastructure by running the Pulumi program. That’s why the snippet above assigns an access policy to currentPrincipal.

Another SP is used by the application itself to read the secret values. The Client ID and Client Secret parameters are placed in the application configuration. Finally, .NET Core configuration gets hooked to the Key Vault at startup:

builder.AddAzureKeyVault(
 $"https://{config["azureKeyVault:vault"]}.vault.azure.net/",
 config["azureKeyVault:clientId"],
 config["azureKeyVault:clientSecret"]
);

This solution is not entirely satisfying though, since we’ve traded storing the secrets for storing SP credentials. Is that a big enough win? Luckily, it’s easy to get rid of those credentials with Managed identities.

5. Accessing Key Vault with Managed Identities

With Managed identities, Azure takes care of creating a Service Principal, passing the credentials, rotating secrets, and so on. Enabling a managed identity on App Service is just an extra option:

const app = new azure.appservice.AppService("app", {
 resourceGroupName: resourceGroup.name,
 appServicePlanId: appServicePlan.id,
 // A system-assigned managed identity
 identity: {
 type: "SystemAssigned",
 },
});

On top of that, the managed principal must be granted access to the Key Vault:

const principalId = app.identity.apply(id => id.principalId);

// Grant App Service access to KV secrets
new azure.keyvault.AccessPolicy("app-policy", {
 keyVaultId: vault.id,
 tenantId: tenantId,
 objectId: principalId,
 secretPermissions: ["get"],
});

Now, the configuration block in the .NET Core app doesn’t need to retrieve any secrets. AzureServiceTokenProvider helps with the authentication process:

var azureServiceTokenProvider = new AzureServiceTokenProvider();
var keyVaultClient = new KeyVaultClient(
 new KeyVaultClient.AuthenticationCallback(
 azureServiceTokenProvider.KeyVaultTokenCallback));
builder.AddAzureKeyVault(
 keyVaultEndpoint, keyVaultClient, new DefaultKeyVaultSecretManager());

That’s quite a bit of a boilerplate, but there is a way to get rid of it.

6. Accessing Key Vault from Application Settings

App Service has a neat feature of integrating its Application Settings with Key Vault. It allows us to combine #3 and #5’s approaches and get the best of both:

// Produce a URI of the KV secret defined above
const secretUri = pulumi.interpolate`${secret.vaultUri}secrets/${secret.name}/${secret.version}`;

const app = new azure.appservice.AppService("app", {
 resourceGroupName: resourceGroup.name,
 appServicePlanId: appServicePlan.id,
 // A system-assigned managed identity
 identity: {
 type: "SystemAssigned",
 },
 appSettings: {
 // The setting points directly to the KV setting
 "PAYMENT_API_KEY": pulumi.interpolate`@Microsoft.KeyVault(SecretUri=${secretUri})`,
 }
});

With that, the API key is loaded into the App Service environment variable without its value being publicly exposed anywhere!

7. Role-Based Access Control

What is the most secure way to deal with secrets? Have no secrets. The Storage Account connection string is a great example when it’s possible to avoid storing and reading the sensitive value altogether.

Role-Based Access Control (RBAC) of Azure Active Directory (AAD) is a great tool to manage permissions in a declarative way. Let’s assume our application only needs to send messages to one Storage Queue. Instead of storing a full connection string with an access token, the connection string should point to the account, and the identity behind the App Service should be granted write permissions to the required Storage Queue:

const permission = new azure.role.Assignment("send", {
 principalId,
 scope: pulumi.interpolate`${storageAccount.id}/queueServices/default/queues/${queue.name}`,
 roleDefinitionName: "Storage Queue Data Message Sender",
});

We can put the queue URL into Application Settings because there’s nothing secret in it:

const queueUri = pulumi.interpolate`${storageAccount.primaryQueueEndpoint}${queue.name}`;

const app = new azure.appservice.AppService("app", {
 resourceGroupName: resourceGroup.name,
 appServicePlanId: appServicePlan.id,
 identity: {
 type: "SystemAssigned"
 },
 appSettings: {
 "STORAGE_QUEUE_URL": queueUri,
 },

It takes a bit of C# boilerplate to send a message with role-based authorization:

var provider = new AzureServiceTokenProvider();
string accessToken = await provider.GetAccessTokenAsync("https://storage.azure.com/");
var tokenCredential = new TokenCredential(accessToken);
var storageCredentials = new StorageCredentials(tokenCredential);
var url = Environment.GetEnvironmentVariable("StorageBlobUrl");
var queue = new CloudQueue(new Uri(url), storageCredentials);
queue.AddMessage(new CloudQueueMessage("Hello"));

Let’s hope another quality-of-life improvement is on the way.

Role of Infrastructure as Code

While security practices may vary depending on application requirements, Pulumi plays an essential role in the appropriate setup of service configuration and environment:

It links an output of one resource to another one’s input, avoiding the need to store the values.
It provides a built-in mechanism to manage external secrets.
It is a great way to take advantage of Azure features like Managed identities and RBAC in a cohesive way.

Infrastructure as Code helps make your applications secure and reliable. Refer to the full example of using Key Vault, Managed identities, RBAC with App Service and Pulumi.

Globally-distributed Serverless App in 100 Lines of Code

Mikhail Shilkov — Tue, 02 Jul 2019 00:00:00 +0000

Pulumi is excellent at connecting multiple cloud components into a cohesive application. In my previous post, I introduced the way to mix JavaScript or TypeScript serverless functions directly into the cloud infrastructure programs.

Today, I will build a serverless application with both the data store and the HTTP endpoint located close to end users to ensure prompt response time. The entire application runs on top of managed Azure services and is defined as a single Pulumi program in TypeScript.

Baseline

I’m going to build a URL shortener: a simple HTTP endpoint which accepts a shortcode in the URL and then redirects a user to the full URL associated with the given short code.

I start simple and make a non-distributed version of the application first. It consists of two main components: a Cosmos DB container to store URL mappings and two Azure Functions to handle HTTP requests:

Let’s define this infrastructure in a Pulumi program.

Cosmos DB Collection

Arguably, Cosmos DB is overkill in such a simple scenario: we just need a key-value store, so Table Storage would suffice. However, Cosmos DB comes handy at the multi-region setup later on.

Next to a standard resource group definition, I create a Cosmos DB account with location set to a single region and Session consistency level:

import * as pulumi from "@pulumi/pulumi";
import * as azure from "@pulumi/azure";
import { CosmosClient } from "@azure/cosmos";

const location = "westus"; // To be changed to multiple configurable locations

let resourceGroup = new azure.core.ResourceGroup("UrlShorterner", {
 location,
});

let cosmosdb = new azure.cosmosdb.Account("UrlStore", {
 resourceGroupName: resourceGroup.name,
 location,
 geoLocations: [{ location, failoverPriority: 0 }],
 offerType: "Standard",
 consistencyPolicy: {
 consistencyLevel: "Session",
 maxIntervalInSeconds: 5,
 maxStalenessPrefix: 100,
 },
});

Cosmos DB has a hierarchical structure of Accounts, Databases, and Containers. Let’s define a database and a SQL collection to store our URL mappings:

const database = new azure.cosmosdb.SqlDatabase("Database", {
 resourceGroupName: resourceGroup.name,
 accountName: account.name,
});

const collection = new azure.cosmosdb.SqlContainer("Urls", {
 resourceGroupName: resourceGroup.name,
 accountName: account.name,
 databaseName: database.name,
});

Function App

Serverless Azure Functions are going to handle the HTTP layer in my application. I use the technique of serverless functions as callbacks to define Azure Functions inline inside my Pulumi program:

const fn = new azure.appservice.HttpEventSubscription("GetUrl", {
 resourceGroup,
 location,
 route: "{key}",
 callback: async (_, request: azure.appservice.HttpRequest) => {
 const endpoint = cosmosdb.endpoint.get();
 const masterKey = cosmosdb.primaryMasterKey.get();

 const client = new CosmosClient(
 { endpoint, key, connectionPolicy: { preferredLocations: [location] } });
 const container = client.database(database.name.get()).container(collection.name.get());

 const key = request.params['key'];
 try {
 const response = await container.item(key.toString()).read();
 return {
 status: 301,
 headers: { "location": response.body.url },
 body: '',
 };
 } catch (e) {
 return { status: 404, body: 'Short URL not found' }


});

export const url = fn.url;

I supply the template {key} as route parameter so that the function accepts wildcard URLs and extract the wildcard value as a key parameter available inside the request object. Then, I use the received key to look up the full URL. The URL is returned to the client as a header of the 301 Redirect response. 404 Not Found is returned in case the requested document does not exist—the naughty Cosmos SDK throws an error in this scenario.

Note how the Cosmos DB parameters endpoint and primaryMasterKey are used directly inside the function. There is no need to manage the keys manually or explicitly put them into application settings: the generated keys are injected into the code at the time of deployment. In practice, a better idea is to store the key in a KeyVault and read it via application settings, but I’ll leave this for a separate article.

A function to add a URL to the database looks quite similar to the above:

const fn = new azure.appservice.HttpEventSubscription("AddUrl", {
 resourceGroup,
 methods: ["POST"],
 callback: async (_, request: azure.appservice.HttpRequest) => {
 const endpoint = cosmosdb.endpoint.get();
 const masterKey = cosmosdb.primaryMasterKey.get();

 const client = new CosmosClient(
 { endpoint, key, connectionPolicy: { preferredLocations: [primaryLocation] } });
 const container = client.database(database.name.get()).container(collection.name.get());

 await container.items.create(request.body);
 return { status: 200, body: 'Short URL saved' };

});
export const addEndpoint = fn.url;

The key differences are:

Using POST as the accepted HTTP method
Creating a new item in Cosmos DB
Using request.body as the payload to save

I might need to add some validation in a real application—skipped for now.

Trying It Out

Now, when a user navigates to an address like https://geturl-xyz.azurewebsites.net/api/URLKEY, they get redirected to the full URL associated with the key. The URL is not quite short yet, but we could make it so with a custom domain configuration.

Presumably, we expect our URL shortener to be popular around the world. Therefore, one key aspect is to make sure that the service responds fast and enables a smooth user experience. Let’s measure the response time of our first version from different spots around the world:

Location	Response time
San Francisco	133 ms
New York	272 ms
London	383 ms
Frankfurt	420 ms
Tel-Aviv	470 ms
Hong-Kong	437 ms
Brisbane	449 ms

The further we get from the West US region, the slower the responses become.

How can we do better?

Bring Compute and Data Closer to Users

It’s hard to beat the speed of light, so if we want to respond fast, we need to bring both code and data close to any target user.

Luckily, the most involved aspect of that—the data distribution—can be handled by Cosmos DB multi-region accounts. We need to take care of deploying the code in multiple locations and routing the traffic to the nearest one. Here is the plan:

I don’t care too much about the latency of Add URL function, so I left it out of the picture.

Configuring the locations

The above picture shows three Azure regions, but since I’m using a general-purpose programming language, I can handle any number of locations in the same way. Actually, I put the list of target regions in Pulumi config file and read it at execution time:

const config = new pulumi.Config();
const locations = config.require("locations").split(',');
const primaryLocation = locations[0];

The order of regions defines the priority.

Multi-region Cosmos DB Account

Since locations is a simple array, I can map this array to produce the array of geoLocations to be set for my Cosmos DB:

let cosmosdb = new azure.cosmosdb.Account("url-cosmos", {
 location: primaryLocation,
 geoLocations: locations.map((location, failoverPriority) => ({
 location,
 failoverPriority
 })),
 /* ... other properties stay the same ... */

Right here, the full power of a programming language at my fingertips!

Multi-region Serverless App

Azure Traffic Manager is a DNS-based traffic load balancer that enables you to distribute traffic optimally to services across global Azure regions while providing high availability and responsiveness.

Sounds like what I need for my global application! Let’s define a Traffic Manager profile:

const profile = new azure.trafficmanager.Profile("UrlShortEndpoint", {
 resourceGroupName: resourceGroup.name,
 trafficRoutingMethod: 'Performance',
 dnsConfigs: [{
 relativeName: "shorturls",
 ttl: 60,
 }],
 monitorConfigs: [{
 protocol: 'HTTP',
 port: 80,
 path: '/api/ping',
 }]
});

I set the routing method to Performance:

Select Performance when you have endpoints in different geographic locations and you want end users to use the “closest” endpoint in terms of the lowest network latency.

Now, I need to create a Function App in each of the target regions. That’s easy to achieve with a for..of loop:

for (const location of locations) {
 const fn = new azure.appservice.HttpEventSubscription(`GetUrl-${location}`, {
 resourceGroup,
 location,
 route: "{key}",
 callback: /* ... the function stays the same ... */
 });
}

Finally, I set up an Endpoint for each Function App to bind them to the Traffic Manager. I do so in the same loop as above:

for (const location of locations) {
 const fn = new azure.appservice.HttpEventSubscription(`GetUrl-${location}`, {
 /* ... function definition as shown above ... */
 });

 const app = fn.functionApp;

 new azure.trafficmanager.Endpoint(`tme${location}`, {
 resourceGroupName: resourceGroup.name,
 profileName: profile.name,
 type: 'azureEndpoints',
 targetResourceId: app.id,
 target: app.defaultHostname,
 endpointLocation: app.location,
 });
}

I assign the properties of each Function App to the corresponding Endpoint with a simple object initializer expression.

Everything is in place. I can export the Traffic Manager endpoint:

export const endpoint = pulumi.interpolate `http://${profile.fqdn}/api/{key}`;

Time to re-run the latency tests.

Results

I deployed the distributed infrastructure to five Azure regions: West US, East US, West Europe, East Asia, and Australia East. The results are shown in the table below, next to the values from the initial run:

Location	Single region	Multiple regions
San Francisco	133 ms	140 ms
New York	272 ms	152 ms
London	383 ms	150 ms
Frankfurt	420 ms	130 ms
Tel-Aviv	470 ms	307 ms
Hong-Kong	437 ms	149 ms
Brisbane	449 ms	529 ms

We got a much smoother distribution of response time. There’s no region close enough to Tel-Aviv, so its latency is still high-ish. I checked Brisbane, and the traffic was somehow directed to the East US region, so the latency hasn’t improved at all. For today, I guess I’ll stick to the hypothesis “Australian internet is slow” and advise to always test your target scenarios without trusting the providers blindly.

More importantly, I walked you through a scenario of developing an end-to-end application with Pulumi. I highlighted the power of leveraging familiar techniques coming from the TypeScript realm. You can find the full code in Pulumi examples and more on Cosmos DB in the Azure Provider.

Cloud brings superpowers to developer’s hands. You just need to use those efficiently.

Hosting a Static Website on Azure with Pulumi

Mikhail Shilkov — Thu, 27 Jun 2019 00:00:00 +0000

Static websites are back in the mainstream these days. Website generators like Jekyll, Hugo, or Gatsby, make it fairly easy to combine templates and markdown pages to produce static HTML files. Static assets are the simplest thing to serve and cache, so the whole setup ends up being fast and cost-efficient.

Many platforms offer services to host such static websites. This post explains the steps to create the infrastructure to do so on Microsoft Azure.

Setting up the infrastructure to serve a static website doesn’t sound like it would be all that difficult, but when you consider HTTPS certificates, content distribution networks, and attaching it to a custom domain, integrating all the components can be quite daunting.

Fortunately, this is a task where Pulumi shines. Pulumi’s code-centric approach not only makes configuring cloud resources easier to do and maintain, but it also eliminates the pain of integrating multiple services.

Overview

Our goal is to create a static website with a custom domain – I’ll use an imaginary demo.pulumi.com for this article. In 2019, my website has to support HTTPS, so we need to create a custom TLS certificate too.

The final solution consists of several Azure services:

Static files will be stored in a Blob Container inside a Storage Account
The Storage Account will have Static Website feature enabled to have some basic URL rewrite rules
We’ll put an Azure CDN Endpoint in front of the container to support the custom domain over TLS
Azure CDN will self-manage the TLS certificate
Our custom DNS provider will have the rule to point to the CDN endpoint (that’s a manual step)

The diagram below outlines the interaction of these components:

Let’s break down how to configure each component using Pulumi.

Resource Group

Let’s start a new Pulumi program, import the Pulumi packages, and define a new resource group:

import * as pulumi from "@pulumi/pulumi";
import * as azure from "@pulumi/azure";

const resourceGroup = new azure.core.ResourceGroup("demo-rg", {
 location: azure.WestEurope,
});

Storage Account

The Storage Account will contain our static website’s assets:

const storageAccount = new azure.storage.Account("demopulumi", {
 resourceGroupName: resourceGroup.name,
 accountReplicationType: "LRS",
 accountTier: "Standard",
 accountKind: "StorageV2",
});

The only trick here is to make sure that the account kind is V2; otherwise, it won’t support the static website feature.

Static Website Hosting in Azure Storage

Any storage container could be exposed as a web endpoint. However, that’s not flexible enough. The URL would always include the container name and the exact file name, so the user would have to ask for https://demo.pulumi.com/containername/index.html instead of simply https://demo.pulumi.com/.

Enabling Static Website Hosting in Azure Storage improves this experience. A dedicated container $web gets automatically created, which also has special treatment for index and 404 documents.

The bad news is that Static Website Hosting is not a part of Azure Resource Manager API, and therefore, it’s not available out-of-the-box in ARM templates, Terraform, or Pulumi. We can enable this feature with Azure CLI, so the solution is to create a dynamic Pulumi resource which enables Pulumi experience while delegating the work to the CLI. You can find the full source code for the dynamic resource in this example, but the usage is quite trivial:

const staticWebsite = new StorageStaticWebsite("demopulumi-static",
 { accountName: storageAccount.name },
 { parent: storageAccount });

// Web endpoint to the website
export const staticEndpoint = staticWebsite.endpoint;

The last line exports the static website endpoint. It will look something like https://demopulumi01234abc.z6.web.core.windows.net/—no custom domain yet, but already functional—once we deploy some static files in there.

Static Files

Now, it’s time to upload the static files to the $web Blob Container.

For this demo, I’ve created a folder wwwroot with two files in it: index.html and 404.html. I can upload those files with the following Pulumi snippet:

["index.html", "404.html"].map(name =>
 new azure.storage.Blob(name, {
 name,
 resourceGroupName: resourceGroup.name,
 storageAccountName: storageAccount.name,
 storageContainerName: staticWebsite.webContainerName,
 type: "block",
 source: `./wwwroot/${name}`,
 contentType: "text/html",
 })
);

In practice, your static website might contain hundreds or thousands of files. At that point, you might want to split the file upload operation from Pulumi and do it as a separate step in your CI/CD pipeline.

Azure CDN

To make the static website files available over our custom domain and HTTPS, we need to create an Azure CDN Endpoint and to point it to the storage account:

const cdn = new azure.cdn.Profile("demo-cdn", {
 resourceGroupName: resourceGroup.name,
 sku: "Standard_Microsoft",
});

const endpoint = new azure.cdn.Endpoint("demo-cdn-ep", {
 name: "demopulumi",
 resourceGroupName: resourceGroup.name,
 profileName: cdn.name,
 isHttpAllowed: true,
 isHttpsAllowed: true,
 originHostHeader: staticWebsite.hostName,
 origins: [{
 name: "blobstorage",
 hostName: staticWebsite.hostName,
 }],
});

// CDN endpoint to the website. Allow it some time after the deployment to get ready.
export const cdnEndpoint = pulumi.interpolate`https://${endpoint.hostName}/`;

I specified an explicit name for the CDN Endpoint: demopulumi. This name has to be globally unique because it is a part of the endpoint URL https://demopulumi.azureedge.net. Please pick a custom name before running the program.

I pointed the CDN origin to the static website name with staticWebsite.hostName. As usual, it’s easy to link resources in Pulumi code!

You might need to wait a few minutes before your content is visible as the CDN configuration is not immediately executed. I’ve set isHttpAllowed to true because HTTP is available sooner than HTTPS; feel free to switch it off for your production configuration.

Configure a Domain DNS Rule

You’ve probably registered your domain with some third-party provider. Follow the instructions of your provider to configure a CNAME rule for the website’s DNS. For a custom domain demo.pulumi.com, the CNAME entry demo would be linked to the endpoint demopulumi.azureedge.net.

CNAME entries don’t support “naked” domains like pulumi.com. If you want to set up a top-level domain to be served from the static Azure website, you’d have to use an Alias DNS record, which isn’t supported by some DNS providers. Please check with your provider for available options.

The following step assumes that a CNAME record is configured; otherwise, it would fail with a validation error.

Custom Domain and TLS

The final step is to point our custom domain to the CDN endpoint and provision a TLS certificate to enable HTTPS support. Once again, these operations are not parts of the ARM API surface, so another dynamic resource was created to support them.

The usage is quite straightforward, just make sure to use your own domain in the following snippet:

const customDomain = new CDNCustomDomainResource("cdn-custom-domain", {
 resourceGroupName: resourceGroup.name,
 // Ensure that there is a CNAME record for demo pointing.pulumi.com to demopulumi.azureedge.net.
 // You would do that in your domain registrar's portal.
 customDomainHostName: "demo.pulumi.com",
 profileName: cdn.name,
 endpointName: endpoint.name,
 // This will enable HTTPS through Azure's one-click automated certificate deployment.
 // The certificate is fully managed by Azure from provisioning to automatic renewal
 // at no additional cost to you.
 httpsEnabled: true,
}, { parent: endpoint });

Bring It Live!

And we are done! Run pulumi up and make sure that all resources get created successfully.

Start testing with staticEndpoint—it’s the first one to become available. cdnEndpoint might return some 404’s at first, be patient. Then, try your custom domain with HTTP. Finally, the TLS certificate takes quite a while to be registered, so come back in an hour or so to test HTTPS.

While a “static website” may sound simple, we went through a rather complicated process to wire all the components together. Some Azure services and features are more straightforward to automate than others, but in the end, we combined all of them into one cohesive Pulumi program to host a static website, served over HTTPS and from a worldwide CDN. That’s the power of a general-purpose programming language applied to the task of sophisticated infrastructure automation. Once the reusable components are in place, reliable and reproducible deployments become a reality.

Serverless as Simple Callbacks with Azure Functions

Mikhail Shilkov — Tue, 07 May 2019 00:00:00 +0000

Today’s guest post is from Mikhail Shilkov, a Microsoft Azure MVP and early Pulumi user and contributor - enjoy!

Serverless compute services, like Azure Functions, offer an amazing power to application developers to leverage: highly available, automatically scaled, low-ceremony, pay-per-value functions created in several lines of code.

So, what’s the simplest way to take a Node.js function and deploy it to Azure cloud as an HTTP endpoint? How about this little tutorial:

1. Create a new Pulumi program:

$ pulumi new azure-typescript

2. Define an HTTP endpoint in `index.ts`:

import * as azure from '@pulumi/azure';

const resourceGroup = new azure.core.ResourceGroup('example', { location: 'West US' });

const greeting = new azure.appservice.HttpEventSubscription('greeting', {
 resourceGroup,
 callback: async (context, req) => {
 return {
 status: 200,
 body: `Hello ${req.query['name'] || 'World'}!`
 };
 }
});

export const url = greeting.url;

3. Deploy:

$ pulumi up
 
Updating:
   Type
 + pulumi:pulumi:Stack
 + ├─ azure:appservice:HttpEventSubscription
 + │ ├─ azure:storage:Account
 + │ ├─ azure:appservice:Plan
 + │ ├─ azure:storage:Container
 + │ ├─ azure:storage:ZipBlob
 + │ └─ azure:appservice:FunctionApp
 + └─ azure:core:ResourceGroup
 
Outputs:
   url: "https://greetingc21a23fe.azurewebsites.net/api/greeting"
Resources:
   + 8 created

4. Access your function via HTTP:

$ curl https://greetingc21a23fe.azurewebsites.net/api/greeting?name=Pulumi
Hello Pulumi!

With 12 lines of code and two CLI commands, I’ve created all Azure resources required to host a serverless function without an explicit configuration of Azure services. Okay, I had to define a location for my resource group, but that could also be accomplished via pulumi config.

Pulumi compiled my TypeScript function, serialized it to a JavaScript file, created the bindings in a function.json file, hosting configuration in a host.json file, uploaded all these assets to Blob Storage, and configured a Consumption Plan and a Function App to run my function. An automated and reproducible deployment in less than two minutes.

Beyond Hello-World

The power of Node.js comes from the richness of its library ecosystem. There’s an NPM package for everything, so you definitely want to use those.

Serverless-function-as-callback imports dependencies in a transparent way. Install the NPM packages of your choice and use them inside the callback:

import * as moment from 'moment';

const greeting = new azure.appservice.HttpEventSubscription('greeting', {
 resourceGroup,
  callback: async (context, req) => {
 return {
 status: 200,
 body: `Hello ${req.query['name'] || 'World'} at ${moment().format('LLLL')}!`
 };
 }
});

The packages get zipped up inside the deployment artifact automatically so that the Function App can find them at the startup. So there’s no need to manually figure out how to produce the archive, get it uploaded, and maintain it as your libraries get updated.

Stay tuned for another blog post with a full implementation of a serverless URL shortener application deployed into multiple Azure regions for fast response time and improved resiliency.

Not Only HTTP

Your application might not be a bunch of HTTP functions. You probably want to leverage queues for asynchronous message passing. How about defining a callback on the queue object itself:

const storageAccount = new azure.storage.Account("storage", {
   resourceGroupName: resourceGroup.name,
   location: resourceGroup.location,
   accountReplicationType: "LRS",
   accountTier: "Standard",
});

const queue = new azure.storage.Queue("myqueue", {
  resourceGroupName: resourceGroup.name,
  storageAccountName: storageAccount.name
});

queue.onEvent("newMessage", async (context, msg) => {
   // code to process 'msg' however you want here
   console.log("Message received: " + msg.toString());
});

Alternatively, define a ServiceBus topic and immediately subscribe to the messages:

import * as servicebus from "@pulumi/azure/eventhub";

const namespace = new servicebus.Namespace("test", {
   resourceGroupName: resourceGroup.name,
   sku: "standard",
});

const topic = new servicebus.Topic("mytopic", {
   resourceGroupName: resourceGroup.name,
   namespaceName: namespace.name,
});

export const subscription = topic.onEvent("mysubscription", async (context, msg) => {
   console.log("Received: " + msg.toString());
});

In addition, get notified when a new PNG image is uploaded to a Blob Container:

const storageContainer = new azure.storage.Container("images-container", {
  resourceGroupName: resourceGroup.name,
  storageAccountName: storageAccount.name,
  name: "images",
});

storageContainer.onBlobEvent("newImage", {
   callback: async (context, blob) => {
       console.log("File size: " + context.bindingData.properties.length);
   },
   filterSuffix: ".png",
});

In all these examples, you get a fully configured Function App on Consumption Plan ramped up and bound to the trigger of choice. Your callback runs in the cloud handling every event with no manual work of hooking these different components together.

Using a general-purpose language like TypeScript provides one consistent approach to defining and delivering serverless applications and infrastructure as one cohesive application.

We strive to make cloud development as simple as everyday JavaScript development that made the language so successful. A lot is happening behind the scenes, but the code still looks like “normal code”. Composition of cloud resources should be as straightforward as hooking up components in any traditional application.

Looking Ahead

Pulumi serverless programming model for Azure Functions is just ramping up. There is only a handful of trigger types supported right now, and some configuration parameters are not exposed yet.

So, today is the perfect time to chime in and join the discussion! Help us answer the questions:

Is this programming model beneficial to your scenarios?
Which trigger types do you want to be supported?
How should we package multiple functions into Function App(s)?
Do you need input and output bindings to be supported, and if yes, in which shape?

Feel free to create an issue on GitHub, tag us on Twitter, or join our community Slack channel.

Happy serverless programming!

Level up your Azure Platform as a Service Applications

Mikhail Shilkov — Mon, 06 May 2019 00:00:00 +0000

Today’s guest post is from Mikhail Shilkov, a Microsoft Azure MVP and early Pulumi user and contributor - enjoy!

Today I want to guide you through the process of developing Pulumi programs to leverage Azure Platform-as-a-Service (PaaS) services. My language of choice is TypeScript—a powerful and expressive typed language, which is very familiar to many Azure users.

Azure Platform as a Service

Azure consists of dozens of cloud services, from VMs to Kubernetes to Serverless. In my experience, a lot of customers choose Azure for its strong portfolio of PaaS-level services.

Azure App Service is a well-established managed compute offering to run web applications, RESTful APIs, or background workers. Azure SQL Database is a fully managed service to run relational databases with features like high availability and backups available out-of-the-box. Enriched by services like Azure DevOps for CI/CD and Application Insights for APM, PaaS is a powerful way to get the benefits of the cloud without the need to fully re-architect software solutions.

The power of relying on PaaS is evidenced by significant customer adoption. App Service is among the most popular compute services in Azure:

If you use automation (ARM, scripts, TF, …) to define and deploy Azure infrastructure, which services are your primary target? Vote & RT!

– Mikhail Shilkov (@MikhailShilkov) April 23, 2019

Nonetheless, PaaS services pose different challenges to application developers. In particular, the usage of multiple cloud services demands an investment in infrastructure automation. That’s where Pulumi comes to the rescue.

A Sample Application

For this walkthrough, I took an existing application from Azure Samples GitHub: .NET Core MVC sample for Azure App Service. Predictably enough, it’s a Todo List application, and this time it is a web app built with ASP.NET Core, Entity Framework Core and a SQL database. Build an ASP.NET Core and SQL Database app in Azure App Service describes how to deploy such application to Azure App Service by means of clicking buttons in the Azure portal.

“Friends don’t let friends right-click publish”

Instead, I suggest relying on infrastructure as code. I built a Pulumi program and integrated it into a fully automated build and deployment pipeline in Azure DevOps.

Here is a sketch of the solution:

Let’s get started building together!

Solution Structure

The following snippet shows the essential elements of the solution:

infra # Cloud infrastructure definition goes here
   index.ts # Pulumi program in TypeScript
src
   Controllers #
   Models # ASP.NET Core web app
   Views # /
   Data # EF Core Data Context
   wwwroot # Static assets (JavaScript/CSS/Images)
azure-pipelines.yml # Azure DevOps pipeline definition

As a first step, I cloned the Todo List app into the src folder. There’s nothing specific to Pulumi here: it’s just an ASP.NET Core app. It could be your application instead.

Note: You can find the full code of the application, infrastructure definition, and deployment pipeline in Pulumi Examples repository.

Bootstrapping a Pulumi Program

The Pulumi development experience is powered by the Pulumi CLI. After installing the CLI, I jump into an empty infra folder and run pulumi new azure-typescript accepting all the default answers. The CLI bootstraps a skeleton of a TypeScript NodeJS application. The code looks like this:

import * as pulumi from "@pulumi/pulumi";
import * as azure from "@pulumi/azure";

// Create an Azure Resource Group
const resourceGroup = new azure.core.ResourceGroup("resourceGroup", {
    location: "WestUS",
});

// Create an Azure resource (Storage Account)
const account = new azure.storage.Account("storage", {
    resourceGroupName: resourceGroup.name,
    location: resourceGroup.location,
    accountTier: "Standard",
    accountReplicationType: "LRS",
});

// Export the connection string for the storage account
export const connectionString = account.primaryConnectionString;

The infrastructure pieces are defined by instantiating objects of appropriate types: ResourceGroup and Account in this example.

Stacks

If an application is developed to run and evolve for months and years, it’s smart to invest in practices like Continuous Integration and Deployment (CI/CD) and Infrastructure as Code (IaC). It’s quite likely that such an application will run in multiple environments: production, staging, development, and so on.

Pulumi comes with a handy concept of stacks— isolated, independently configurable instances of a Pulumi program. A separate stack can be designated for each deployment environment.

We can take the notion of the stack into the program and apply the stack name to definitions of infrastructure resources:

// Use first 10 characters of the stackname as prefix for resource names
const prefix = pulumi.getStack().substring(0, 9);
 
const resourceGroup = new azure.core.ResourceGroup(`${prefix}-rg`, {
    location: "WestUS",
});
 
const resourceGroupArgs = {
    resourceGroupName: resourceGroup.name,
    location: resourceGroup.location,
};
 
// Storage Account name must be lowercase and cannot have any dash characters
const storageAccountName = `${prefix.toLowerCase().replace(/-/g, "")}sa`;
const storageAccount = new azure.storage.Account(storageAccountName, {
    ...resourceGroupArgs,
 
    accountKind: "StorageV2",
    accountTier: "Standard",
    accountReplicationType: "LRS",
});

Note how I use the power of the general purpose programming language to

Interact with the environment by reading the stack name
Encode custom rules for resource naming
Work around the shortcomings of the cloud, namely, the restricted set of characters to use in Storage Accounts
Extract the value resourceGroupArgs to reuse the same definition for upcoming resources.

As a result, the stacks production and dev will be deployed to separate resource groups with clean and consistent naming throughout the resources.

Deploying the Application to App Service

Now, it’s time to define the infrastructure to host my ASP.NET Core app. There are three pieces of the puzzle to fit together.

1. App Service Plan

An App Service Plan defines the pricing tier, instance size and other parameters related to performance, scalability, and cost of the hosted application. My definition is relatively straightforward:

const appServicePlan = new azure.appservice.Plan(`${prefix}-asp`, {
    ...resourceGroupArgs,
 
    kind: "App",
 
    sku: {
        tier: "Basic",
        size: "B1",
    },
});

In a more advanced scenario, I could provision different performance tiers based on the target environment identified by the stack name.

2. Deployment Artifact

App Service is a mature Azure service with a long history, so it has numerous options for deployment methods. Arguably, the newest Run from Package is the most friendly way to practice Infrastructure as Code automation.

Essentially, I prepare a zip file with the published .NET Core assemblies and upload it to Azure Blob Storage:

const storageContainer = new azure.storage.Container(`${prefix}-c`, {
    resourceGroupName: resourceGroup.name,
    storageAccountName: storageAccount.name,
    containerAccessType: "private",
});
 
const blob = new azure.storage.ZipBlob(`${prefix}-b`, {
    resourceGroupName: resourceGroup.name,
    storageAccountName: storageAccount.name,
    storageContainerName: storageContainer.name,
    type: "block",
 
    content: new pulumi.asset.FileArchive("../src/bin/Debug/netcoreapp2.1/publish.zip")
});

3. App Service

Now, I can define an App Service and instruct it to use this package to run the website. I do so by linking its application settings to the shared access signature of the blob:

const codeBlobUrl = signedBlobReadUrl(blob, storageAccount, storageContainer);
 
const app = new azure.appservice.AppService(`${prefix}-as`, {
    ...resourceGroupArgs,
    appServicePlanId: appServicePlan.id,
 
    appSettings: {
        "WEBSITE_RUN_FROM_PACKAGE": codeBlobUrl,
    },
});

At startup, App Service downloads the zip and mounts it as a local read-only disk to store the application binaries.

Adding a SQL Database

The application host is now defined, but I also need a relational database to store and query Todo Items. Staying true to the PaaS path, I’m using Azure SQL Database service.

Reading Configuration Parameters

Setting up a SQL Server requires a couple of parameter values that might change between execution environments, for instance, a username and a password for the connection string. Pulumi provides a way to configure the program’s parameters per stack.

The configuration itself will happen in my CI/CD pipeline. For now, I can query the values with pulumi.Config helper tool:

// Get the username and the password to use for SQL from config.
const config = new pulumi.Config();
const username = config.require("sqlUsername");
const pwd = config.require("sqlPassword");

Azure SQL Server and Database

I’m all set to code the SQL infrastructure. Two resources need to be defined:

const sqlServer = new azure.sql.SqlServer(`${prefix}-sql`, {
    ...resourceGroupArgs,
 
    administratorLogin: username,
    administratorLoginPassword: pwd,
    version: "12.0",
});
 
const database = new azure.sql.Database(`${prefix}-db`, {
    ...resourceGroupArgs,
    serverName: sqlServer.name,
    requestedServiceObjectiveName: "S0"
});

Note that requestedServiceObjectiveName defines the performance tier and the price of the Azure SQL Database.

Wiring App Service to the Database

By default, Azure SQL Database is configured to deny any incoming connections for security reasons. One approach is to allow access to all Azure services. However, a more secure method is to white-list the Outbound IPs of the App Service:

const firewallRules = app.outboundIpAddresses.apply(
    ips => ips.split(',').map(
        ip => new azure.sql.FirewallRule(`FR${ip}`, {
            endIpAddress: ip,
            resourceGroupName: resourceGroup.name,
            serverName: sqlServer.name,
            startIpAddress: ip,
        })
    ));

Note that the actual IPs of the service are unknown at the time of writing the program. Nonetheless, the combination of apply, split, and map functions enables me to wire the runtime value of IPs to the proper set of firewall rules.

Additional firewall rules may be implemented to allow administrative access from outside the App Service.

Finally, our ASP.NET Core application expects a connection string with MyDbConnection. To create one, I join the database server and the database name to produce the connection string and add it to the App Service configuration:

const app = new azure.appservice.AppService(`${prefix}-as`, {
    // ... other parameters as defined above
 
    connectionStrings: [{
        name: "MyDbConnection",
        value:
            pulumi.all([sqlServer.name, database.name]).apply(([server, db]) =>
                `Server=tcp:${server}.database.windows.net;initial catalog=${db};user ID=${username};password=${pwd};Min Pool Size=0;Max Pool Size=30;Persist Security Info=true;`),
        type: "SQLAzure"
    }]
});

Collecting Metrics with Application Insights

Application Insights is an Application Performance Management (APM) service to be used for collecting metrics from cloud applications.

After adding Application Insights NuGet packages into my ASP.NET solution, I can go ahead and define the App Insights resource and link it to the App Service with an instrumentation key:

const appInsights = new azure.appinsights.Insights(`${prefix}-ai`, {
    ...resourceGroupArgs,
 
    applicationType: "Web"
});
 
const app = new azure.appservice.AppService(`${prefix}-as`, {
    // ... other parameters as defined above
 
    appSettings: {
        "ApplicationInsights:InstrumentationKey": appInsights.instrumentationKey,
        // ... other settings as defined above
    },
});

Continuous Deployment with Azure DevOps

While Pulumi CLI works great during development; a production deployment should rather be executed in a CI/CD pipeline. This time I’m using Azure DevOps – a managed CI/CD service to build, test, and deploy cloud applications.

In particular, I defined an Azure Pipeline consisting of three steps:

Build & Publish the .NET Core application;
Restore NPM packages for the infrastructure program;
Provision the cloud infrastructure with Pulumi.

Build & Publish the .NET Application

The first step utilizes a built-in task which triggers the .NET Core CLI to build the source code in the src folder and publish the assemblies as a zip file.

steps:
- task: DotNetCoreCLI@2
 inputs:
   command: 'publish'
   projects: 'src'
 displayName: 'Build and publish the ASP.NET Core app'

Restore NPM packages for the infrastructure program

The second step is a simple npm install step to restore the NodeJS dependencies:

 - task: Npm@1
   inputs:
      command: 'install'
      workingDir: 'infra'
   displayName: 'NPM install packages'

Install Pulumi and Run Infrastructure Code

There is a Pulumi task available in Azure Marketplace: Pulumi Azure Pipelines Task. After installing it to your organization, you should be able to utilize a simple task like this:

 - task: Pulumi@0
   inputs:
      azureSubscription: 'Your Azure Subscription(aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee)'
      command: 'up'
      args: '--yes'
      cwd: 'infra'
      stack: 'dev'
   displayName: 'Install pulumi and run infra code'
   name: pulumi

Alternatively, if you can’t install third-party tasks from Azure Marketplace, you can call the Pulumi CLI from a custom shell script, see this pipeline definition.

When everything is wired correctly, I see this screen in Azure DevOps:

The newly created resource group contains six resources:

The application is up and running:

The telemetry is flowing into Application Insights:

You can find the full code of the application, infrastructure definition, and deployment pipeline in Pulumi Examples repository.

Pulumi ❤️ Azure PaaS

Azure App Service and friends are a great way to deploy web applications and APIs without worrying about the nitty-gritty details of the underlying hardware.

Writing a TypeScript program to compose an application out of the cloud building blocks feels like a breeze to me. I can reuse all my skills and stay productive by defining cloud resources as code in a familiar language.

You can get going with these resources:

Pulumi enables developers to define cloud infrastructure using general purpose programming languages. Pulumi works with multiple cloud providers and has first-class support for all services in Microsoft Azure.

Pulumi Blog: Mikhail Shilkov

Pulumi Updates, Explained: AI-Powered Features in Pulumi CLI

Update Explanation

Diagnostics Summarization

Getting Started

Share Your Feedback

AI-Assisted Infrastructure as Code with Pulumi's Model Context Protocol Server

Setting up the Pulumi MCP Integration in Cursor

The Goal: Provisioning an AKS Cluster

The Traditional Approach

The AI Assistant + Pulumi MCP Approach

Understanding the AI Assistant + Tool Interaction

Step-by-Step Walkthrough

The Request

AI-Powered Resource Discovery

AI Code Generation & Editing

Integrated Validation

AI-Assisted Debugging & Iteration

Collaborative Problem Solving & Tooling

Successful Deployment

Accessing Outputs

Why This Changes the Game

Conclusion

Announcing the Next-Generation of Pulumi Components: Enabling Infrastructure Abstractions

Understanding Pulumi Components

What’s New with Pulumi Components

How It Works

Creating Components

The PulumiPlugin.yaml File

Define an Entry Point

Sharing Components

Publishing to Git

Referencing Components Locally

Consuming Components Across Languages

See It In Action

Pro Tips

The Spectrum of Pulumi Components You Can Build

Conclusion

Introducing Customizable Resource Auto-naming in Pulumi

The Road to Better Resource Naming

Introducing Auto-naming Configuration

Disable Auto-naming

Use Logical Names As-Is

Custom Naming Patterns

See It In Action

Configuration Syntax

Getting Started

General Availability

Pulumi + Azure Deployment Environments: Better Together for Enterprise Developers

How It Works

Getting Started

Bring a Custom Container Image

Conclusion

Announcing 6.0 of the Pulumi AWS Provider

Terraform Plugin Framework Support

New Resources and Functions

SDK Size Optimization

Upstream Breaking Changes

Fixes for Highly Upvoted Issues

MaxItemsOne and Deprecations

Migration Guide

Get Started Today

Pulumi Release Notes: Resource Search, Deployment Actions, Projects in Self-Managed Backends, and more

Pulumi CLI

Aligning Projects between Service and Self-Managed Backends

Improved Interactive Scrolling Experience

Shortcut to See the Current Update in Web Browser

New Flag to Specify Teams for a Newly Created Stack

Visual Indication of Retained Resources

View More Details in Stack History

Pulumi Automation API

Manage Stack Tags

Full “Who Am I” command support

Cloud Providers and Packages

Pulumi Cloud & Pulumi.com

Resource Search Improvements

Events Dashboard card

Stacks Page Icons

Deployment Actions

SAML Admin

New –shell option for `pulumi stack output`

`pulumi new` in directories with source control