Pulumi Blog: Thomas Kappler

Azure Native V3: A Leaner, Faster SDK for Microsoft Azure

Thomas Kappler — Thu, 03 Apr 2025 00:00:00 +0000

Pulumi Azure Native V3 is the most comprehensive infrastructure as code (IaC) solution for Microsoft Azure, combining full resource coverage with dramatic improvements in performance and developer experience.

With V3, the Pulumi Azure Native SDK is 75% smaller while maintaining 100% coverage of the Azure Resource Manager (ARM) API. This makes it the most efficient and complete way to define and manage Azure infrastructure with real programming languages.

Pulumi Azure Native provides direct access to the Azure Resource Manager API, making it hands-down the best infrastructure as code solution for Microsoft Azure. Unlike traditional providers that abstract away cloud APIs, Azure Native gives you more resource and property coverage compared to Terraform by directly mapping to Azure’s native REST API. This means:

Complete API coverage: Access every Azure resource available in ARM
Same-day updates: New Azure services and properties are available immediately
Advanced property support: Work with every resource property, not just a subset
Latest API versions: Always up to date with the newest stable Azure APIs

And with V3’s slimmer SDK footprint, developers benefit from faster IDE performance, reduced memory usage, quicker installs, and improved CI/CD pipeline efficiency.

These improvements came about through our RFC process, where we explored various options to balance SDK size with the flexibility needed by advanced users.

A Slimmer, More Efficient SDK

The most significant improvement in Azure Native V3 is the dramatic 75% reduction in SDK size across all languages. By optimizing the SDK to include only the default API versions, we’ve solved one of the main pain points reported by our users, the size of the package:

Language	Before	After	Reduction
.NET	77 MB	15 MB	80.5%
JS/TS	494 MB	87 MB	82.4%
Python	104.6 MB	25 MB	76.3%
Go (source)	910 MB	167 MB	81.6%
Java (explicit versions were not supported in v2)	68 MB	71 MB	+4.4%
YAML	no SDK	no SDK	-

This transformation delivers immediate benefits that every developer will notice: faster IDE performance with reduced load times, more responsive language servers for better autocompletion, reduced memory consumption, significantly quicker package installation, and improved CI/CD pipeline performance.

Additional Enhancements in Azure Native V3

Azure Native V3 introduces more than just SDK size optimization. Other improvements include:

Reorganized module structure better aligned with Azure SDK conventions
Improved resource naming for clearer distinction between similar resources
Enhanced handling of nested properties to avoid accidental overwriting
Better state management with the removal of unnecessary __inputs fields (most noticeable with large stacks)

Understanding Azure Versioning

The Pulumi Azure Native provider needs to manage two different versioning systems:

Pulumi’s Semantic Versioning: Our SDKs follow the traditional major.minor.patch format (e.g., 3.0.0)
- Patch versions (3.0.1) contain bug fixes only
- Minor versions (3.1.0) add new resources and non-breaking features
- Major versions (3.0.0) may contain breaking changes
Azure’s Date-based API Versions: Azure resources use date-based versions like “2024-01-01” or “2023-05-15-preview”
- Each Azure service (resource provider) has its own set of API versions
- Breaking changes can occur between different API versions
- Preview versions (with “-preview” suffix) contain newer features that may not be stable yet

Default API Versions

Azure Native V3 includes refreshed default API versions across nearly all resources, ensuring you’re automatically working with the latest stable Azure features and capabilities without any extra configuration. The new default versions unlock powerful new Azure functionality out of the box, such as:

EventGrid (now 2025-02-15): Support for advanced filtering, dead-lettering, and custom event schemas
MachineLearningServices (now 2024-10-01): Enhanced compute instance capabilities and new model deployment options
Storage (now 2024-01-01): Improved performance tiers and expanded security features
KeyVault (now 2024-03-01): Enhanced certificate management and expanded access policy options

Using Non-Default API Versions

Most users (approximately 95% based on our Pulumi Cloud data) will be perfectly served by these default API versions. However, there are situations where you might need to use a specific API version.

For these cases, Azure Native V3 provides two flexible approaches:

Option 1: Accessing Explicit API Versions via Local Packages

The most type-safe way to use explicit API versions is through local packages. This approach gives you the full SDK experience with strong typing, auto-completion, and documentation.

Add the specific API version package

Use the Pulumi CLI to add just the packages and versions you need:

pulumi package add azure-native storage v20240101

For more details on the package add command, see the CLI documentation.

Step 2: Import and use the versioned package

Once added, you can import and use the specific API version in your code:

import * as resources from "@pulumi/azure-native/resources";
import * as storage_v20240101 from "@pulumi/azure-native_storage_v20240101";

const resourceGroup = new resources.ResourceGroup("resourceGroup");

const storageAccount = new storage_v20240101.storage.StorageAccount("sa", {
 resourceGroupName: resourceGroup.name,
 sku: {
 name: storage_v20240101.storage.SkuName.Standard_LRS,
 },
 kind: storage_v20240101.storage.Kind.StorageV2,
});

import pulumi
from pulumi_azure_native import resources
import pulumi_azure_native_storage_v20240101 as storage_v20240101

resource_group = resources.ResourceGroup("resource_group")

storage_account = storage_v20240101.storage.StorageAccount("sa",
 resource_group_name=resource_group.name,
 sku=storage_v20240101.storage.SkuArgs(
 name=storage_v20240101.storage.SkuName.STANDARD_LRS,
 ),
 kind=storage_v20240101.storage.Kind.STORAGE_V2)

using Pulumi;
using Pulumi.AzureNative.Resources;
using Storage_v20240101 = Pulumi.AzureNative_Storage_v20240101;

class MyStack : Stack
{
 public MyStack()
 {
 var resourceGroup = new ResourceGroup("resourceGroup");

 var storageAccount = new Storage_v20240101.Storage.StorageAccount("sa", new()
 {
 ResourceGroupName = resourceGroup.Name,
 Sku = new Storage_v20240101.Storage.SkuArgs
 {
 Name = Storage_v20240101.Storage.SkuName.Standard_LRS,
 },
 Kind = Storage_v20240101.Storage.Kind.StorageV2,
 });
 }
}

package main

import (
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
 "github.com/pulumi/pulumi-azure-native/sdk/go/azure/resources"
 storage_v20240101 "github.com/pulumi-azure-native-storage-v20240101/sdk/go/azure/storage"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 resourceGroup, err := resources.NewResourceGroup(ctx, "resourceGroup", nil)
 if err != nil {
 return err
 }

 storageAccount, err := storage_v20240101.NewStorageAccount(ctx, "sa", &storage_v20240101.StorageAccountArgs{
 ResourceGroupName: resourceGroup.Name,
 Sku: &storage_v20240101.SkuArgs{
 Name: storage_v20240101.SkuNameStandard_LRS,
 },
 Kind: storage_v20240101.KindStorageV2,
 })
 if err != nil {
 return err
 }

 return nil
 })
}

name: azure-native-example
runtime: yaml
description: A minimal Azure Native Pulumi YAML program
resources:
 resourceGroup:
 type: azure-native:resources:ResourceGroup
 properties:
 location: westus2

 # Using specific API version package
 storageAccount:
 type: azure-native_storage_v20240101:storage:StorageAccount
 properties:
 resourceGroupName: ${resourceGroup.name}
 sku:
 name: Standard_LRS
 kind: StorageV2

package myproject;

import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.azurenative.resources.ResourceGroup;
import com.pulumi.azurenative.resources.ResourceGroupArgs;
import com.pulumi.azurenative_storage_v20240101.storage.StorageAccount;
import com.pulumi.azurenative_storage_v20240101.storage.StorageAccountArgs;
import com.pulumi.azurenative_storage_v20240101.storage.enums.Kind;
import com.pulumi.azurenative_storage_v20240101.storage.enums.SkuName;
import com.pulumi.azurenative_storage_v20240101.storage.inputs.SkuArgs;

public class App {
 public static void main(String[] args) {
 Pulumi.run(App::stack);
 }

 private static void stack(Context ctx) {
 var resourceGroup = new ResourceGroup("resourceGroup");

 var storageAccount = new StorageAccount("sa", StorageAccountArgs.builder()
 .resourceGroupName(resourceGroup.name())
 .sku(SkuArgs.builder()
 .name(SkuName.Standard_LRS)
 .build())
 .kind(Kind.StorageV2)
 .build());
 }
}

Option 2: Using the Generic Resource

An alternative approach is to use the generic azure-native.resources.Resource. This special resource allows you to access any Azure resource at any API version, including older versions that might not be available through local packages.

While not as type-safe as the local package approach, the generic resource offers maximum flexibility and is useful when you need to access a resource that is not yet supported by the Pulumi provider or when an issue prevents you from using a specific resource.

In most cases, we recommend using the local packages approach when you need a specific API version, and only falling back to the generic resource for special cases or resources not yet supported by the Pulumi provider.

For more details, see the Azure Native version guide.

Getting Started with Azure Native V3

If You’re Already Using Pulumi Azure Native

Upgrading ensures you benefit from the smaller SDK, refreshed defaults, and the best performance when managing Azure at scale.

Upgrading from v2: Follow the v2 to v3 migration guide.
Upgrading from v1: First move to v2 using our v1 to v2 upgrade guide, the follow the v2 to v3 migration guide. This two-step process resolves deprecation warnings and ensures a smoother migration.

For language-specific installation instructions, see our Azure Native setup guide.

If You’re New to Pulumi

Pulumi makes it easy to define, deploy, and manage Azure infrastructure using your favorite programming languages. To get started:

Sign up for Pulumi Cloud - free for individuals and teams.
Explore the Azure Native installation guide.
Try a quickstart project to provision real Azure resources in minutes.

Conclusion

Pulumi Azure Native V3 is the most complete and efficient solution for managing Azure infrastructure as code. With a dramatically smaller SDK, updated defaults, flexible API versioning, and enhanced developer experience, it offers the best combination of performance, coverage, and control for Microsoft Azure users.

We’d like to thank our community for their valuable feedback throughout the development of V3. Your input helped shape this release, and we’re excited to see what you’ll build with it.

Have questions or suggestions? Open an issue on GitHub, join our Community Slack.

Pulumi Command Provider 1.0 Release

Thomas Kappler — Mon, 01 Jul 2024 10:00:00 -0700

Today, we’re happy to announce the 1.0 release of the Pulumi Command provider. This release marks the provider’s official transition from preview status to general availability (GA). Since its first preview release in late 2021, thousands of Pulumi users incorporated the Command package into their cloud infrastructure projects to help manage local and remote command execution and filesystem operations. In fact, the Command provider is already our ninth-most popular provider, so it’s time to make things official!

The Pulumi Command Provider enables you to execute commands and scripts either locally or remotely as part of the Pulumi resource model, enabling stateful command execution. It also has convenient support for copying assets via SSH. Being able to run arbitrary commands opens up a wide variety of scenarios and integrations with other tools and systems.

Some of the provider’s popular uses include:

Running a command locally after creating a resource, to register it with an external service
Running a command locally before deleting a resource, to deregister it with an external service
Running a command on a remote host immediately after creating it
Copying a file to a remote host after creating it (potentially as a script to be executed afterwards)
As a simple alternative to some use cases for Dynamic Providers, by running appropriate commands in the different stages of the Pulumi life cycle.

Here’s a simple example of running an arbitrary command when another resource changes: once an AWS Lambda function is deployed, we call it and it responds with a message that is custom to the current Pulumi stack. In a real-world scenario, we could wait for a number of other resources to be deployed as well, and have the Lambda perform registration or initialization of services.

import * as aws from "@pulumi/aws";
import { local } from "@pulumi/command";
import { getStack } from "@pulumi/pulumi";

const f = new aws.lambda.CallbackFunction("f", {
 publish: true,
 callback: async (ev: any) => {
 return `Stack ${ev.stackName} is deployed!`;
 }
});

const invoke = new local.Command("execf", {
 create: `aws lambda invoke --function-name "$FN" --payload '{"stackName": "${getStack()}"}' --cli-binary-format raw-in-base64-out out.txt >/dev/null && cat out.txt | tr -d '"' && rm out.txt`,
 environment: {
 FN: f.qualifiedArn,
 AWS_REGION: aws.config.region!,
 AWS_PAGER: "",
 },
}, { dependsOn: f })

export const output = invoke.stdout;

import pulumi
import json
import pulumi_aws as aws
import pulumi_command as command

lambda_role = aws.iam.Role("lambdaRole", assume_role_policy=json.dumps({
 "Version": "2012-10-17",
 "Statement": [{
 "Action": "sts:AssumeRole",
 "Effect": "Allow",
 "Principal": {
 "Service": "lambda.amazonaws.com",
 },
 }],
}))

lambda_function = aws.lambda_.Function("lambdaFunction",
 name="f",
 publish=True,
 role=lambda_role.arn,
 handler="index.handler",
 runtime=aws.lambda_.Runtime.NODE_JS20D_X,
 code=pulumi.FileArchive("./handler"))

aws_config = pulumi.Config("aws")
aws_region = aws_config.require("region")

invoke_command = command.local.Command("invokeCommand",
 create=f"aws lambda invoke --function-name \"$FN\" --payload '{{\"stackName\": \"{pulumi.get_stack()}\"}}' --cli-binary-format raw-in-base64-out out.txt >/dev/null && cat out.txt | tr -d '\"' && rm out.txt",
 environment={
 "FN": lambda_function.arn,
 "AWS_REGION": aws_region,
 "AWS_PAGER": "",
 },
 opts = pulumi.ResourceOptions(depends_on=[lambda_function]))

pulumi.export("output", invoke_command.stdout)

package main

import (
 "encoding/json"
 "fmt"

 "github.com/pulumi/pulumi-aws/sdk/v6/go/aws/iam"
 "github.com/pulumi/pulumi-aws/sdk/v6/go/aws/lambda"
 "github.com/pulumi/pulumi-command/sdk/go/command/local"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi/config"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 awsConfig := config.New(ctx, "aws")
 awsRegion := awsConfig.Require("region")

 tmpJSON0, err := json.Marshal(map[string]interface{}{
 "Version": "2012-10-17",
 "Statement": []map[string]interface{}{
 {
 "Action": "sts:AssumeRole",
 "Effect": "Allow",
 "Principal": map[string]interface{}{
 "Service": "lambda.amazonaws.com",
 },
 },
 },
 })
 if err != nil {
 return err
 }
 json0 := string(tmpJSON0)
 lambdaRole, err := iam.NewRole(ctx, "lambdaRole", &iam.RoleArgs{
 AssumeRolePolicy: pulumi.String(json0),
 })
 if err != nil {
 return err
 }
 lambdaFunction, err := lambda.NewFunction(ctx, "lambdaFunction", &lambda.FunctionArgs{
 Name: pulumi.String("f"),
 Publish: pulumi.Bool(true),
 Role: lambdaRole.Arn,
 Handler: pulumi.String("index.handler"),
 Runtime: pulumi.String(lambda.RuntimeNodeJS20dX),
 Code: pulumi.NewFileArchive("./handler"),
 })
 if err != nil {
 return err
 }
 invokeCommand, err := local.NewCommand(ctx, "invokeCommand", &local.CommandArgs{
 Create: pulumi.String(fmt.Sprintf("aws lambda invoke --function-name \"$FN\" --payload '{\"stackName\": \"%v\"}' --cli-binary-format raw-in-base64-out out.txt >/dev/null && cat out.txt | tr -d '\"' && rm out.txt", ctx.Stack())),
 Environment: pulumi.StringMap{
 "FN": lambdaFunction.Arn,
 "AWS_REGION": pulumi.String(awsRegion),
 "AWS_PAGER": pulumi.String(""),
 },
 }, pulumi.DependsOn([]pulumi.Resource{
 lambdaFunction,
 }))
 if err != nil {
 return err
 }
 ctx.Export("output", invokeCommand.Stdout)
 return nil
 })
}

using System.Collections.Generic;
using System.Text.Json;
using Pulumi;
using Aws = Pulumi.Aws;
using Command = Pulumi.Command;

return await Deployment.RunAsync(() =>
{
 var awsConfig = new Config("aws");

 var lambdaRole = new Aws.Iam.Role("lambdaRole", new()
 {
 AssumeRolePolicy = JsonSerializer.Serialize(new Dictionary<string, object?>
 {
 ["Version"] = "2012-10-17",
 ["Statement"] = new[]
 {
 new Dictionary<string, object?>
 {
 ["Action"] = "sts:AssumeRole",
 ["Effect"] = "Allow",
 ["Principal"] = new Dictionary<string, object?>
 {
 ["Service"] = "lambda.amazonaws.com",
 },
 },
 },
 }),
 });

 var lambdaFunction = new Aws.Lambda.Function("lambdaFunction", new()
 {
 Name = "f",
 Publish = true,
 Role = lambdaRole.Arn,
 Handler = "index.handler",
 Runtime = Aws.Lambda.Runtime.NodeJS20dX,
 Code = new FileArchive("./handler"),
 });

 var invokeCommand = new Command.Local.Command("invokeCommand", new()
 {
 Create = $"aws lambda invoke --function-name \"$FN\" --payload '{{\"stackName\": \"{Deployment.Instance.StackName}\"}}' --cli-binary-format raw-in-base64-out out.txt >/dev/null && cat out.txt | tr -d '\"' && rm out.txt",
 Environment =
 {
 { "FN", lambdaFunction.Arn },
 { "AWS_REGION", awsConfig.Require("region") },
 { "AWS_PAGER", "" },
 },
 }, new CustomResourceOptions
 {
 DependsOn =
 {
 lambdaFunction,
 },
 });

 return new Dictionary<string, object?>
 {
 ["output"] = invokeCommand.Stdout,
 };
});

package generated_program;

import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.aws.iam.Role;
import com.pulumi.aws.iam.RoleArgs;
import com.pulumi.aws.lambda.Function;
import com.pulumi.aws.lambda.FunctionArgs;
import com.pulumi.command.local.Command;
import com.pulumi.command.local.CommandArgs;
import static com.pulumi.codegen.internal.Serialization.*;
import com.pulumi.resources.CustomResourceOptions;
import com.pulumi.asset.FileArchive;
import java.util.Map;

public class App {
 public static void main(String[] args) {
 Pulumi.run(App::stack);
 }

 public static void stack(Context ctx) {
 var awsConfig = ctx.config("aws");
 var awsRegion = awsConfig.require("region");

 var lambdaRole = new Role("lambdaRole", RoleArgs.builder()
 .assumeRolePolicy(serializeJson(
 jsonObject(
 jsonProperty("Version", "2012-10-17"),
 jsonProperty("Statement", jsonArray(jsonObject(
 jsonProperty("Action", "sts:AssumeRole"),
 jsonProperty("Effect", "Allow"),
 jsonProperty("Principal", jsonObject(
 jsonProperty("Service", "lambda.amazonaws.com")))))))))
 .build());

 var lambdaFunction = new Function("lambdaFunction", FunctionArgs.builder()
 .name("f")
 .publish(true)
 .role(lambdaRole.arn())
 .handler("index.handler")
 .runtime("nodejs20.x")
 .code(new FileArchive("./handler"))
 .build());

 var lambdaInvokeOut = lambdaFunction.arn().applyValue(arn -> {
 var invokeCommand = new Command("invokeCommand", CommandArgs.builder()
 .create(String.format(
 "aws lambda invoke --function-name \"$FN\" --payload '{\"stackName\": \"%s\"}' --cli-binary-format raw-in-base64-out out.txt >/dev/null && cat out.txt | tr -d '\"' && rm out.txt",
 ctx.stackName()))
 .environment(Map.ofEntries(
 Map.entry("FN", arn),
 Map.entry("AWS_REGION", awsRegion),
 Map.entry("AWS_PAGER", "")))
 .build(),
 CustomResourceOptions.builder()
 .dependsOn(lambdaFunction)
 .build());

 return invokeCommand.stdout();
 });

 ctx.export("output", lambdaInvokeOut);
 }
}


resources:
 lambdaRole:
 type: aws:iam:Role
 properties:
 assumeRolePolicy:
 fn::toJSON:
 Version: "2012-10-17"
 Statement:
 - Action: sts:AssumeRole
 Effect: Allow
 Principal:
 Service: lambda.amazonaws.com

 lambdaFunction:
 type: aws:lambda:Function
 properties:
 name: f
 publish: true
 role: ${lambdaRole.arn}
 handler: index.handler
 runtime: "nodejs20.x"
 code:
 fn::fileArchive: ./handler

 invokeCommand:
 type: command:local:Command
 properties:
 create: 'aws lambda invoke --function-name "$FN" --payload ''{"stackName": "${pulumi.stack}"}'' --cli-binary-format raw-in-base64-out out.txt >/dev/null && cat out.txt | tr -d ''"'' && rm out.txt'
 environment:
 FN: ${lambdaFunction.arn}
 AWS_REGION: ${aws:region}
 AWS_PAGER: ""
 options:
 dependsOn:
 - ${lambdaFunction}

outputs:
 output: ${invokeCommand.stdout}

What’s new

The 1.0 release of the Command provider marks a stable API for the 1.x series. Rather than simply putting a “v1” label on the existing provider, we also used this opportunity to enhance it with some previously requested capabilities to provide a more complete set of capabilities.

The API documentation in the Pulumi registry has examples in all Pulumi languages and is expanded.
Capturing stdout and stderr of commands can now be switched off, which is useful when they might contain secrets or are very noisy.
Environment handling for remote commands has better error handling and is better documented.
The CopyFile resource is superseded by the new CopyToRemote resource. It can copy whole directories in addition to individual files. The source of the copy is now a Pulumi asset or archive which provides full interoperability with the Pulumi ecosystem. The use of assets and archives also makes Pulumi run copy operations only if the source has changed. For an easy transition, the previous CopyFile resource will remain available with a deprecation notice until the next major version.

Here’s an example of copying a directory to a remote host. For brevity, the remote server is assumed to exist, but it could also be provisioned in the same Pulumi program.

import * as pulumi from "@pulumi/pulumi";
import { remote, types } from "@pulumi/command";
import * as fs from "fs";
import * as os from "os";
import * as path from "path";

export = async () => {
 const config = new pulumi.Config();

 // Get the private key to connect to the server. If a key is
 // provided, use it, otherwise default to the standard id_rsa SSH key.
 const privateKeyBase64 = config.get("privateKeyBase64");
 const privateKey = privateKeyBase64 ?
 Buffer.from(privateKeyBase64, 'base64').toString('ascii') :
 fs.readFileSync(path.join(os.homedir(), ".ssh", "id_rsa")).toString("utf8");

 const serverPublicIp = config.require("serverPublicIp");
 const userName = config.require("userName");

 // The configuration of our SSH connection to the instance.
 const connection: types.input.remote.ConnectionArgs = {
 host: serverPublicIp,
 user: userName,
 privateKey: privateKey,
 };

 // Set up source and target of the remote copy.
 const from = config.require("payload")!;
 const archive = new pulumi.asset.FileArchive(from);
 const to = config.require("destDir")!;

 // Copy the files to the remote.
 const copy = new remote.CopyToRemote("copy", {
 connection,
 source: archive,
 remotePath: to,
 });

 // Verify that the expected files were copied to the remote.
 // We want to run this after each copy, i.e., when something changed,
 // so we use the asset to be copied as a trigger.
 const find = new remote.Command("ls", {
 connection,
 create: `find ${to}/${from} | sort`,
 triggers: [archive],
 }, { dependsOn: copy });

 return {
 remoteContents: find.stdout
 }
}

import pulumi
import pulumi_command as command

config = pulumi.Config()

server_public_ip = config.require("serverPublicIp")
user_name = config.require("userName")
private_key = config.require("privateKey")
payload = config.require("payload")
dest_dir = config.require("destDir")

archive = pulumi.FileArchive(payload)

# The configuration of our SSH connection to the instance.
conn = command.remote.ConnectionArgs(
 host = server_public_ip,
 user = user_name,
 privateKey = private_key,
)

# Copy the files to the remote.
copy = command.remote.CopyToRemote("copy",
 connection=conn,
 source=archive,
 destination=dest_dir)

# Verify that the expected files were copied to the remote.
# We want to run this after each copy, i.e., when something changed,
# so we use the asset to be copied as a trigger.
find = command.remote.Command("find",
 connection=conn,
 create=f"find {dest_dir}/{payload} | sort",
 triggers=[archive],
 opts = pulumi.ResourceOptions(depends_on=[copy]))

pulumi.export("remoteContents", find.stdout)

package main

import (
 "fmt"

 "github.com/pulumi/pulumi-command/sdk/go/command/remote"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi/config"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 cfg := config.New(ctx, "")
 serverPublicIp := cfg.Require("serverPublicIp")
 userName := cfg.Require("userName")
 privateKey := cfg.Require("privateKey")
 payload := cfg.Require("payload")
 destDir := cfg.Require("destDir")

 archive := pulumi.NewFileArchive(payload)

 conn := remote.ConnectionArgs{
 Host: pulumi.String(serverPublicIp),
 User: pulumi.String(userName),
 PrivateKey: pulumi.String(privateKey),
 }

 copy, err := remote.NewCopyToRemote(ctx, "copy", &remote.CopyToRemoteArgs{
 Connection: conn,
 Source: archive,
 })
 if err != nil {
 return err
 }

 find, err := remote.NewCommand(ctx, "find", &remote.CommandArgs{
 Connection: conn,
 Create: pulumi.String(fmt.Sprintf("find %v/%v | sort", destDir, payload)),
 Triggers: pulumi.Array{
 archive,
 },
 }, pulumi.DependsOn([]pulumi.Resource{
 copy,
 }))
 if err != nil {
 return err
 }

 ctx.Export("remoteContents", find.Stdout)
 return nil
 })
}

using System.Collections.Generic;
using Pulumi;
using Command = Pulumi.Command;

return await Deployment.RunAsync(() =>
{
 var config = new Config();
 var serverPublicIp = config.Require("serverPublicIp");
 var userName = config.Require("userName");
 var privateKey = config.Require("privateKey");
 var payload = config.Require("payload");
 var destDir = config.Require("destDir");

 var archive = new FileArchive(payload);

 // The configuration of our SSH connection to the instance.
 var conn = new Command.Remote.Inputs.ConnectionArgs
 {
 Host = serverPublicIp,
 User = userName,
 PrivateKey = privateKey,
 };

 // Copy the files to the remote.
 var copy = new Command.Remote.CopyToRemote("copy", new()
 {
 Connection = conn,
 Source = archive,
 });

 // Verify that the expected files were copied to the remote.
 // We want to run this after each copy, i.e., when something changed,
 // so we use the asset to be copied as a trigger.
 var find = new Command.Remote.Command("find", new()
 {
 Connection = conn,
 Create = $"find {destDir}/{payload} | sort",
 Triggers = new[]
 {
 archive,
 },
 }, new CustomResourceOptions
 {
 DependsOn =
 {
 copy,
 },
 });

 return new Dictionary<string, object?>
 {
 ["remoteContents"] = find.Stdout,
 };
});

package generated_program;

import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.command.remote.Command;
import com.pulumi.command.remote.CommandArgs;
import com.pulumi.command.remote.CopyToRemote;
import com.pulumi.command.remote.inputs.*;
import com.pulumi.resources.CustomResourceOptions;
import com.pulumi.asset.FileArchive;
import java.util.List;
import java.util.ArrayList;
import java.util.Map;
import java.io.File;
import java.nio.file.Files;
import java.nio.file.Paths;

public class App {
 public static void main(String[] args) {
 Pulumi.run(App::stack);
 }

 public static void stack(Context ctx) {
 final var config = ctx.config();
 final var serverPublicIp = config.require("serverPublicIp");
 final var userName = config.require("userName");
 final var privateKey = config.require("privateKey");
 final var payload = config.require("payload");
 final var destDir = config.require("destDir");

 final var archive = new FileArchive(payload);

 // The configuration of our SSH connection to the instance.
 final var conn = ConnectionArgs.builder()
 .host(serverPublicIp)
 .user(userName)
 .privateKey(privateKey)
 .build();

 // Copy the files to the remote.
 var copy = new CopyToRemote("copy", CopyToRemoteArgs.builder()
 .connection(conn)
 .source(archive)
 .destination(destDir)
 .build());

 // Verify that the expected files were copied to the remote.
 // We want to run this after each copy, i.e., when something changed,
 // so we use the asset to be copied as a trigger.
 var find = new Command("find", CommandArgs.builder()
 .connection(conn)
 .create(String.format("find %s/%s | sort", destDir,payload))
 .triggers(archive)
 .build(), CustomResourceOptions.builder()
 .dependsOn(copy)
 .build());

 ctx.export("remoteContents", find.stdout());
 }
}


resources:
 # Copy the files to the remote.
 copy:
 type: command:remote:CopyToRemote
 properties:
 connection: ${conn}
 source: ${archive}
 remotePath: ${destDir}

 # Verify that the expected files were copied to the remote.
 # We want to run this after each copy, i.e., when something changed,
 # so we use the asset to be copied as a trigger.
 find:
 type: command:remote:Command
 properties:
 connection: ${conn}
 create: find ${destDir}/${payload} | sort
 triggers:
 - ${archive}
 options:
 dependsOn:
 - ${copy}

config:
 serverPublicIp:
 type: string
 userName:
 type: string
 privateKey:
 type: string
 payload:
 type: string
 destDir:
 type: string

variables:
 # The source directory or archive to copy.
 archive:
 fn::fileArchive: ${payload}
 # The configuration of our SSH connection to the instance.
 conn:
 host: ${serverPublicIp}
 user: ${userName}
 privateKey: ${privateKey}

outputs:
 remoteContents: ${find.stdout}

Getting started

If you’re currently using one of the v0 pre-release versions of the Command provider, you don’t need to make any changes to your existing code. However, the enhanced functionality of copying assets and archives is only available in the new CopyToRemote resource. If you’re using CopyFile, you should consider migrating to the new resource after the upgrade to 1.0.

Enjoy the new features, and we’re looking forward to seeing what you build with the Command provider!

Announcing OIDC Support for Pulumi Azure Providers

Thomas Kappler — Thu, 01 Jun 2023 00:00:00 +0000

We are happy to announce the delivery of Azure OIDC authentication, one of the most requested features for the Pulumi Azure Native Provider. With the v1.100.0 release, OpenID Connect (OIDC) authentication is now fully supported in both the Azure Native and Azure Classic providers. Let’s dig in to learn what it is, how it works, and why it’s useful.

Pulumi announced OIDC support for Deployments in January. This article is about the same OIDC protocol, used to authenticate your Pulumi program with Azure instead of Pulumi Deployments.

The What and Why of OIDC

OIDC stands for OpenID Connect, a standardized protocol for federated identity. This means that if you maintain an identity with two service providers, you can tell one of them to trust the other. Then, you can use credentials for the trusted service to get credentials for the other.

As an example, we’ll use a Pulumi program running in CI on GitHub Workflows, creating Azure resources. Without OIDC, this program would need a secret to authenticate with Azure - a client secret or a certificate. With OIDC, your program doesn’t need a secret. It can send its GitHub token to Azure instead. Based on the trust relationship between the two, Azure will exchange the GitHub token for an Azure token.

Secrets need to be safeguarded, rotated, and possibly revoked. If you’ve ever managed a system, whether production or CI/CD, the possibility of managing fewer secrets will be enticing. So how do we go about implementing this?

Enabling OIDC

There are two parts to enabling your Pulumi program to authenticate via OIDC. One is establishing the trust relationship between Azure and the other service that’s involved, like GitHub. This is a one-time operation. The second one is providing your program with the necessary configuration to perform the OIDC token exchange at runtime.

Fortunately, Microsoft has the first step covered with their establishing the trust relationship guide. The idea is that an Active Directory app registration holds the necessary federated credentials. Your program will then use this app’s tenant id to request the token exchange. The setup depends on whether you run on GitHub Actions, Kubernetes, or other providers.

On to the second step, configuring your Pulumi program. First, tell the Pulumi provider to use OIDC by setting the environment variable ARM_USE_OIDC or the Pulumi configuration azure-native:useOidc to true. Next, you’re in luck if your program runs on GitHub Actions: you’re done. GitHub exports the necessary values in form of variables that Pulumi understands.

For GitHub, pay close attention to the “entity type” of your Azure Active Directory credential as documented in the Azure guide. It can be a Git branch or tag, a pull request, or an environment as specified in the GitHub workflow. When your Action requests the OIDC token exchange, one of these needs to match exactly. This means that the only way to configure OIDC authentication for all branches or tags is to use environments in your workflows.

For other providers, you need to provide your Pulumi program with two more settings. Environment variable ARM_OIDC_REQUEST_TOKEN or configuration azure-native:oidcRequestToken is your provider’s token to exchange for an Azure token. ARM_OIDC_REQUEST_URL or azure-native:oidcRequestUrl is the URL to contact to initiate the token exchange.

A complete example

Let’s take a look at how the CI end-to-end tests of the Pulumi Azure Native provider are set up.

Configuration

First, we have an Active Directory App creatively named oidc-test. We simply copy its client and tenant ids and set them in the CI workflow, along with our subscription:

env:
ARM_CLIENT_ID: 11223344-****-****-****-************ [redacted]
ARM_SUBSCRIPTION_ID: 55667788-****-****-****-************ [redacted]
ARM_TENANT_ID: 99aabbcc-****-****-****-************ [redacted]
ARM_USE_OIDC=true

Note how there is no secret or certificate present.

We also give the required permissions to the workflow step where the authentication happens:

permissions:
id-token: write # required for OIDC auth

In Azure, we added the required credentials to the Active Directory App. You’ll find this under “Certificates & secrets” -> “Federated credentials” on the app’s page. Let’s look at two of the credentials. Both enable OIDC for all pull requests on the repository, one for each Azure provider.

End to End Flow

Now, when our GitHub workflow for CI tests runs, the Azure provider knows to authenticate via OIDC because ARM_USE_OIDC is set. It performs the following steps:

Call GitHub’s OIDC provider, found at the URL that GitHub automatically exports as ACTIONS_ID_TOKEN_REQUEST_URL. For authentication, it includes a token GitHub generates for each workflow run and exports as ACTIONS_ID_TOKEN_REQUEST_TOKEN. The response is an OIDC token in form of a JSON Web Token (JWT).
Send the OIDC token to Azure’s Active Directory endpoint. Include the client and tenant ids of our Active Directory App that we configured via ARM_CLIENT_ID and ARM_TENANT_ID. Active Directory looks up the trust relationship we configured and validates the token, and responds with a valid access token for Azure.
The Pulumi provider can now access Azure resources.

That’s it! Your CI job now runs without maintaining an Azure secret in GitHub, or wherever your Pulumi program runs.

Wrapping up

This article only demonstrated the basics. You can also customize your OIDC token’s claims to control access to Azure in a more fine-grained manner.

I hope this helps you simplify your Azure authentication. Please try it out and let us know if you hit any issues in the Pulumi Community Slack!