Pulumi Blog: Migration

Neo: Zero-downtime migration from CDK, Terraform & Azure ARM

Pulumi Neo Team — Wed, 21 Jan 2026 00:00:00 +0000

The barrier to migrating to Pulumi has always been the infrastructure you already have. Your existing resources can’t be disrupted, and manually importing them into a new tool is risky and time-consuming. Today, we’re excited to share how Neo removes this barrier entirely with automated, zero-downtime migration to Pulumi from AWS CDK, AWS CloudFormation, Terraform, CDKTF, and Azure ARM templates.

Why IaC migrations are hard

The promise of Infrastructure as Code is that your code perfectly describes your running infrastructure. But switching IaC tools breaks this promise in dangerous ways.

When you rewrite your infrastructure code in a new tool, you have two choices, both problematic. You can destroy and recreate all your resources to match the new code, accepting downtime and risk. Or you can try to import existing resources, which requires perfect knowledge of how every resource maps between the old and new systems. Many teams get stuck here, wanting Pulumi’s modern platform but unable to safely make the switch.

Neo’s insight: State knowledge enables perfect migrations

The key to safe migration isn’t just converting code - it’s understanding the complete relationship between your existing IaC tool’s state and your actual cloud resources. Each IaC tool maintains this relationship differently: CDK through CloudFormation stacks, Terraform/CDKTF through state files, and ARM through Azure deployments. But they all have complete knowledge of what they manage.

Neo leverages this existing state knowledge to orchestrate perfect resource transitions. Instead of asking you to manually find resource IDs and construct migration commands, Neo reads your current tool’s state, discovers every resource’s physical identity, and brings them under Pulumi management. This isn’t just automation - it’s using the source tool’s own knowledge against the migration problem.

In practice, this means Neo can bridge the gap between how tools name resources and where they actually live. A Lambda function that CDK knows as OrderHandler9I0J1K2L actually exists in AWS as my-app-OrderHandler-9I0J1K2L, while a Terraform resource at address aws_instance.web[2] maps to EC2 instance i-0abc123def456. Neo understands these mappings and handles the complex cases like composite IDs (FunctionName|StatementId), resource references, and dependency chains that must be migrated in order.

Because Neo uses the source tool’s own state knowledge, your infrastructure doesn’t change at all during migration. Not a single resource is modified, recreated, or even touched. We’re simply transferring ownership from one IaC tool to another.

This approach delivers three critical guarantees:

Zero downtime: Resources are never deleted or recreated
Zero risk: Since nothing changes, you can abandon the migration at any point without consequence
Zero surprises: Preview confirms no infrastructure changes before you commit

Migration in action: Four tools, one approach

Neo adapts its migration strategy to each tool’s unique characteristics while maintaining the same zero-downtime guarantee. Let’s explore how Neo handles migrations from each major IaC tool.

AWS CDK to Pulumi

For teams using AWS CDK, Neo leverages the CloudFormation layer that underpins CDK deployments. CDK’s architecture actually makes migration straightforward: since CDK synthesizes to CloudFormation templates, Neo can read the deployed stacks directly to understand every resource and its configuration. For detailed migration steps, see our CDK migration guide.

The challenge with CDK migrations isn’t the CloudFormation layer - it’s the cryptic resource naming. CDK generates logical IDs like OrdersTableA7B2C3D4 that map to physical resources with completely different names. These mappings are buried in CloudFormation metadata, and getting them wrong means either orphaning resources or accidentally creating duplicates. Neo navigates this complexity by reading CloudFormation’s own stack outputs and resource metadata, discovering the exact physical ID for every logical resource.

CDK also introduces complexity through its construct hierarchy. A single high-level construct might expand into dozens of CloudFormation resources, each with dependencies and references to others. Neo preserves these relationships during migration, ensuring that IAM roles still reference the right Lambda functions, API Gateway deployments still point to the correct stages, and security groups maintain their exact rules. The migration completes with your infrastructure unchanged and Pulumi’s preview confirming zero modifications.

Start a Neo task Migrate your CDK application

AWS CloudFormation to Pulumi

For teams using CloudFormation directly (rather than through CDK), Neo provides a streamlined migration path. CloudFormation stacks contain complete resource metadata - every resource’s logical ID maps to a physical resource in AWS, and CloudFormation tracks these relationships in its stack state. Neo reads this state directly to build a complete picture of your infrastructure. For detailed migration steps, see our CloudFormation migration guide.

The main challenge with CloudFormation migrations is the template language itself. CloudFormation templates use intrinsic functions like !Ref, !GetAtt, and !Sub that create implicit dependencies between resources. A security group might reference a VPC using !Ref MyVpc, while a Lambda function’s role uses !GetAtt LambdaRole.Arn. Neo evaluates these expressions against the actual deployed stack to resolve every reference to its concrete value.

CloudFormation also supports features like conditionals, mappings, and nested stacks that add layers of indirection. Neo handles these by examining what actually got deployed rather than trying to interpret every possible template path. The result is Pulumi code that manages your exact infrastructure configuration - not a theoretical interpretation of your template. The migration completes with pulumi preview confirming zero changes to your running resources.

Start a Neo task Migrate your CloudFormation stack

Terraform to Pulumi

Terraform and CDKTF migrations require two transformations: converting state to establish Pulumi’s connection to your resources, and transforming HCL configuration into Pulumi code. The state conversion is direct - Neo reads your Terraform state and converts it into Pulumi state, preserving the mappings between resource names and cloud IDs. This ensures Pulumi knows that aws_instance.web[2] corresponds to EC2 instance i-0abc123def456.

The code transformation analyzes your HCL to generate equivalent Pulumi code. Neo handles Terraform patterns like count and for_each loops, module structures, and resource dependencies, recreating them idiomatically in Pulumi. The generated code manages your existing infrastructure without modifications - pulumi preview confirms zero changes. For complete migration instructions, see our Terraform migration guide.

Start a Neo task Migrate your Terraform application

Azure ARM to Pulumi

ARM templates present unique migration challenges. Unlike CDK and Terraform, which maintain clear separation between code and state, ARM templates blur this line. The template is both the definition and, through deployment history, part of the state tracking. ARM’s template expression language, with its concat functions and resource ID constructors, makes it difficult to determine what resources actually exist until deployment time. For step-by-step migration guidance, see our ARM migration guide.

Neo orchestrates ARM migrations through intelligent AI-driven conversion. When an ARM template uses functions like concat(parameters('appName'), '-plan'), the conversion process evaluates these expressions using the actual parameter values to generate the correct resource names. Azure resource IDs follow predictable patterns - subscription IDs, resource groups, providers, and resource names - and Neo ensures these are correctly brought under Pulumi management using inline resource specifications directly in the generated code.

The biggest challenge with ARM migrations is handling the implicit dependencies and resource provider quirks. An App Service might implicitly create a service plan, a SQL database requires a server that might be defined in a linked template, and child resources like application settings need separate migration steps. Neo understands these Azure-specific patterns and generates the appropriate Pulumi code to manage every resource. The migration completes with a zero-diff preview, confirming your exact Azure configuration is preserved while giving you a more maintainable, type-safe way to manage it going forward.

Start a Neo task Migrate your ARM template

The architecture powering it all

While each tool requires specific handling, Neo’s core architecture remains consistent:

Universal orchestration layer

Neo acts as the intelligent migration coordinator, regardless of source tool:

Credential verification: Ensures proper cloud credentials are configured in Pulumi ESC
Resource inventory: Builds a complete catalog of existing resources
Conversion orchestration: Manages the code transformation
State migration: Brings existing resources under Pulumi management
Audit trail generation: Creates comprehensive migration reports

Unified state management engine

The state management engine works consistently across all tools:

Maps source resource IDs to Pulumi’s state management
Handles complex and composite resource identifiers
Provides fallback strategies for edge cases
Ensures idempotent operations

Human oversight

While Neo automates the heavy lifting, we maintain human checkpoints:

Review generated code before migrating resources
Verify preview shows zero changes
Approve the resulting pull request

What this means for your team

Migration friction no longer locks you into your current IaC tool. If you want Pulumi’s programming model, policy engine, and multi-cloud support, Neo gets you there without disrupting your infrastructure.

Ready to migrate? Check out our migration guides for CDK, CloudFormation, Terraform, or Azure ARM. Join us in the Pulumi Community Slack or reach out to your account team for a guided migration session.

CDKTF is deprecated: What's next for your team?

Adam Gordon Bell — Thu, 18 Dec 2025 10:00:00 -0800

In July, 2020, CDK for Terraform (CDKTF) was introduced, and last week, on December 10, it was officially deprecated. Support for CDKTF has stopped, the organization and repository have been archived, and HashiCorp/IBM will no longer be updating or maintaining it, leaving a lot of teams out there without a clear path forward.

For most teams, that means it’s time to start looking for a replacement.

It’s an unfortunate situation to suddenly find yourself in as a user of CDKTF, but you do have options, and Pulumi is one of them. In this post, we’ll help you understand what those options are, how Pulumi fits into them, and what it’d look like to migrate your CDKTF projects to Pulumi.

What are the alternatives to CDKTF?

Teams migrating away from CDKTF generally have three options:

Option 1: Fall back to HCL

HashiCorp’s official recommendation is to export your projects to HashiCorp Configuration Language (HCL) and manage them with Terraform. CDKTF even has a command that makes this fairly simple:

cdktf synth --hcl

Of course, if you’re using CDKTF, you probably chose it specifically to avoid HCL. So while possible, this probably isn’t the choice most teams would make unless they had to.

Option 2: Migrate to AWS CDK

If your team is all-in on AWS, another option would be to migrate to AWS CDK. It’s widely used, officially supported, the programming model is similar to CDKTF’s, and both CDK and CDKTF transpile to an intermediate format (CloudFormation YAML and Terraform JSON, respectively) that gets passed on to their underlying tools for deployment.

But while their programming and deployment models are conceptually similar, their resource models and APIs are entirely different. Here’s the code for an S3 bucket written in AWS CDK, for example:

import * as s3 from 'aws-cdk-lib/aws-s3';

const bucket = new s3.Bucket(this, 'my-bucket', {
 bucketName: 'my-example-bucket',
 versioned: true,
 publicReadAccess: false,
});

And here’s the code for a similarly configured bucket in CDKTF:

import { S3Bucket } from '@cdktf/provider-aws/lib/s3-bucket';

const bucket = new S3Bucket(this, 'my-bucket', {
 bucket: 'my-example-bucket',
 versioning: {
 enabled: true,
 },
 acl: 'private',
});

Notice how different these APIs are — and this is just one simple resource with only a few properties; imagine having to rewrite dozens or hundreds of them. Beyond that, there’s also the problem of state: How would you go about translating the contents of a Terraform state file containing hundreds of resources into the equivalent CloudFormation YAML or JSON?

Despite their surface similarities, CDKTF and AWS CDK have little in common. Migration would essentially mean a ground-up rewrite that’d also leave you without the multi-cloud support you already have with CDKTF. For most teams, that makes this option a practical non-starter.

Option 3: Migrate to Pulumi

This is where we should acknowledge our obvious bias — but we genuinely believe that for most users of CDKTF, Pulumi really is the simplest and most broadly compatible alternative.

Like CDKTF, Pulumi lets you build and manage your infrastructure with general-purpose languages like TypeScript, Python, Go, C#, and Java, and it supports organizing your code into higher-level abstractions called components, which you can think of like CDKTF constructs. Both organize cloud resources into stacks (think dev, prod), and both track deployment state similarly, with local, remote, and cloud-hosted options available.

Many of Pulumi’s most popular providers (e.g., the AWS provider) are also built from open-source Terraform schemas, which means their resource models will be nearly identical to what you’re used to with CDKTF. Here’s what an S3 bucket looks like in Pulumi, for example:

import * as aws from '@pulumi/aws';

const bucket = new aws.s3.Bucket('my-bucket', {
 bucket: 'my-example-bucket',
 versioning: {
 enabled: true,
 },
 acl: 'private',
});

You can also use any Terraform provider with Pulumi, and you can even reference Terraform modules directly from within your Pulumi code.

Pulumi is also different from CDKTF in several ways. One is that rather than transpile your source code to a format like JSON as CDKTF does (and then deploying it separately later), Pulumi uses its own declarative deployment engine that resolves the resource graph at runtime and provisions cloud resources directly, which is much faster and more flexible. You can learn more about the deployment model in How Pulumi Works.

Given the API similarities, the support for all Terraform providers and modules, the ability to coexist alongside Terraform-managed projects, and the built-in support for conversion (which we’ll cover next), we think Pulumi is the best alternative for most teams looking to migrate.

Migrate from CDKTF to Pulumi

Keep writing infrastructure in TypeScript, Python, Go, C#, or Java, and use Pulumi’s built-in convert and import commands to bring your CDKTF projects and state across.

Get started

What migrating to Pulumi looks like

Migrating a CDKTF project to Pulumi generally happens in three steps:

Conversion, which translates your CDKTF code into a new Pulumi program
Import, which reads the contents of your CDKTF state into a new Pulumi stack
Refactoring, which brings the code in the new program into alignment with the stack’s currently deployed resources

Conversion and import

Migration starts with exporting your CDKTF project to HCL with cdktf synth. From there, Pulumi’s built-in convert and import commands handle creating the new program and importing your state:

# Export your project to HCL.
cdktf synth --hcl

# Convert the HCL into a new Pulumi project.
pulumi convert --from terraform --language typescript

# Create a new Pulumi stack.
pulumi stack init dev

# Import your CDKTF stack's resources into your new Pulumi stack.
pulumi import --from terraform ./terraform.dev.tfstate

The converter automatically translates Terraform input variables, data sources, resources, and outputs into their Pulumi equivalents. You can read more about how this works in Converting Terraform HCL to Pulumi.

Refactoring

Once you’ve imported your state, you’ll often have to make some adjustments to the code to bring it in line with the new Pulumi stack. For instance, pulumi import marks new resources protected by default, to prevent them from being accidentally deleted — but since the code produced by pulumi convert doesn’t include the protect resource option, you’ll need to add it yourself. Fortunately the import step also emits code that you can copy into your program to make this process a little easier.

Refactoring can get a bit more complicated when custom logic and higher-level abstractions are involved, as fidelity to the original CDKTF code is often lost in the translation to HCL. In these situations, having the help of an LLM to recapture that original logic or translate your CDKTF constructs into Pulumi components can be a big time-saver.

An end-to-end example

The best way to get a feel for how this works, though, is to try it yourself.

The pulumi/cdktf-to-pulumi-example repository on GitHub contains a CDKTF project with multiple stacks written in TypeScript, along with a guide that walks you through the process of migrating that project to Pulumi. The guide covers everything we’ve discussed here so far, including:

Converting the CDKTF project into a new Pulumi project
Importing its actively running resources into Pulumi stacks
Modifying the generated code to align with imported state
Performing an initial deployment with Pulumi to complete the migration process

The walkthrough takes only a few minutes to complete, and it’s a great way to stand up an example of your own to get more familiar with Pulumi.

github.com/pulumi/cdktf-to-pulumi-example

What’s next?

If you’re moving on from CDKTF and looking for an alternative, there are a few possible paths forward. For teams that want to keep using real languages and avoid a ground-up rewrite, Pulumi offers the clearest way forward.

To learn more about how Pulumi works, how it differs from CDKTF and from Terraform, how to handle additional conversion scenarios, and more, we recommend:

Diving into the Pulumi docs to get familiar with core concepts and features of the platform
Reading Migrating from Terraform or CDKTF to Pulumi for more detailed, Terraform-specific migration guidance
Joining us in the Pulumi Community Slack to ask questions and learn from others who’ve successfully made the leap from Terraform and CDKTF to Pulumi
Checking out Pulumi for All Your IaC — Including Terraform and HCL to learn more about Pulumi’s native support for Terraform and HCL

And of course, feel free to reach out! We’d love to help in any way we can.

New: Use Terraform Modules in Pulumi Without Conversion

Anton Tayanovskyy — Mon, 23 Jun 2025 00:00:00 +0000

Today, we’re excited to announce a major advancement in Pulumi’s mission to make modern infrastructure as code accessible to every developer: direct support for executing Terraform modules. This new capability addresses one of the most significant challenges our users face when migrating from Terraform to Pulumi—complex projects with extensive module dependencies.

The Path to Modern Infrastructure as Code

At Pulumi, we believe the ideal infrastructure as code experience leverages the full power of modern programming languages, AI-assisted development, and cloud-native tooling. Our vision is that every team should be able to write infrastructure code in their preferred language—TypeScript, Python, Go, C#, or Java—with full IntelliSense, testing capabilities, and AI-powered assistance.

However, we also understand the reality that many organizations have invested years building extensive Terraform module estates. These modules often contain critical business logic, compliance configurations, and battle-tested patterns that teams can’t always rewrite overnight.

This new capability bridges that gap, giving you the best of both worlds: the ability to start new projects in Pulumi immediately while preserving your existing Terraform modules until you’re ready to migrate them.

Strategic Migration Approach

Here’s how we recommend teams approach this transition:

Phase 1: New Projects in Pulumi Start all new infrastructure projects in Pulumi to immediately gain the benefits of modern IaC—better developer experience, AI assistance, and cloud-native tooling.

Phase 2: Incremental Module Migration Use the new Terraform module support to bring existing modules into Pulumi projects without rewriting them. This gives you immediate access to Pulumi Cloud’s state management, deployment workflows, and team collaboration features across all your IaC.

Phase 3: Full Migration (When Ready) Gradually migrate your Terraform modules to native Pulumi components as time and resources permit. With AI-assisted development tools making this easier than ever, you can modernize at your own pace.

This approach eliminates the “big bang” migration risk while ensuring you’re not stuck with legacy tooling for new projects.

What This Means for Your Team

This isn’t just a technical feature—it’s a strategic enabler for teams looking to modernize their infrastructure practices. Here’s what this announcement means:

For Terraform Users: You can now migrate to Pulumi incrementally, keeping your existing modules intact while gaining access to Pulumi’s powerful programming language features, better state management, and enhanced developer experience.

For Platform Teams: You can standardize on Pulumi for new infrastructure while seamlessly incorporating existing Terraform modules, reducing migration risk and accelerating adoption.

For DevOps Engineers: You get the best of both worlds—the reliability of proven Terraform modules with Pulumi’s superior tooling, testing capabilities, and deployment workflows.

How It Works

The new feature consists of three key components:

1. Direct Module Execution

The pulumi package add terraform-module command now supports modules from Terraform and OpenTofu registries, as well as locally managed modules. This is powered by our new terraform-module provider, which orchestrates Terraform execution while maintaining the full Pulumi lifecycle.

2. Enhanced Conversion Support

Our pulumi convert --from terraform tool now supports a special annotation // @pulumi-terraform-module <pulumi-package-name> that tells the converter to execute modules directly rather than attempting to translate them. This preserves your existing module logic while enabling Pulumi’s benefits.

3. Configuration Synchronization

Pulumi providers now expose helper methods like awsProvider.terraformConfig() to keep configuration consistent between Pulumi and Terraform providers, eliminating the need to maintain duplicate configuration.

Real-World Impact

Let me walk you through a practical example that demonstrates the power of this integration.

First, install the latest version of the Pulumi CLI (v3.178.0 or later is required) and create a new Pulumi project using pulumi new $language such as pulumi new typescript.

Setting Up a VPC Module

Next, add a Terraform module to your Pulumi project:

$ pulumi package add terraform-module terraform-aws-modules/vpc/aws 6.0.0 vpcmod
Successfully generated a Nodejs SDK for the vpcmod package at /Users/anton/tmp/2025-06-23/blog/sdks/vpcmod

$ pulumi package add terraform-module terraform-aws-modules/vpc/aws 6.0.0 vpcmod

Successfully generated a Python SDK for the vpcmod package at /workdir/vpcmod

Resolved 13 packages in 111ms
 Built pulumi-vpcmod @ file:///workdir/sdks/vpcmod
Prepared 4 packages in 297ms
Installed 4 packages in 1ms
 + arpeggio==2.0.2
 + attrs==25.3.0
 + parver==0.5
 + pulumi-vpcmod==6.0.0 (from file:///workdir/sdks/vpcmod)

You can then import the SDK in your Python code with:

 import pulumi_vpcmod as vpcmod

$ pulumi package add terraform-module terraform-aws-modules/vpc/aws 6.0.0 vpcmod

Using Terraform CLI for schema inference
Successfully generated a Go SDK for the vpcmod package at /workdir/sdks/vpcmod
Go mod file updated to use local sdk for vpcmod
To use this package, import github.com/pulumi/pulumi-terraform-module/sdks/go/vpcmod/v6/vpcmod
Added package "vpcmod" to Pulumi.yaml

$ pulumi package add terraform-module terraform-aws-modules/vpc/aws 6.0.0 vpcmod

Using Terraform CLI for schema inference
Successfully generated a .NET SDK for the vpcmod package at /workdir/sdks/vpcmod

Reference `sdks\vpcmod\Pulumi.Vpcmod.csproj` added to the project.
You also need to add the following to your .csproj file of the program:

 <DefaultItemExcludes>$(DefaultItemExcludes);sdks/**/*.cs</DefaultItemExcludes>

You can then use the SDK in your .NET code with:

 using Pulumi.Vpcmod;

Added package "vpcmod" to Pulumi.yaml

$ pulumi package add terraform-module terraform-aws-modules/vpc/aws 6.0.0 vpcmod

Using Terraform CLI for schema inference
Successfully generated a Java SDK for the vpcmod package at /workdir/sdks/vpcmod

To use this SDK in your Java project, complete the following steps:

1. Copy the contents of the generated SDK to your Java project:
 cp -r /workdir/sdks/vpcmod/src/* /workdir/src

2. Add the SDK's dependencies to your Java project's build configuration.
 If you are using Maven, add the following dependencies to your pom.xml:

 <dependencies>
 <dependency>
 <groupId>com.google.code.findbugs</groupId>
 <artifactId>jsr305</artifactId>
 <version>3.0.2</version>
 </dependency>
 <dependency>
 <groupId>com.google.code.gson</groupId>
 <artifactId>gson</artifactId>
 <version>2.8.9</version>
 </dependency>
 </dependencies>

Added package "vpcmod" to Pulumi.yaml

$ pulumi package add terraform-module terraform-aws-modules/vpc/aws 6.0.0 vpcmod

Added package "vpcmod" to Pulumi.yaml

Pulumi automatically generates a local SDK with full support for your language:

$ ls sdks/vpcmod
README.md index.ts node_modules provider.ts tsconfig.json utilities.ts
bin module.ts package.json scripts types

$ ls sdks/vpcmod
build pulumi_vpcmod pulumi_vpcmod.egg-info setup.py

$ ls sdks/vpcmod
go.mod vpcmod

$ ls sdks/vpcmod
Inputs Provider.cs README.md
logo.png pulumi-plugin.json Utilities.cs
Module.cs Pulumi.Vpcmod.csproj version.txt

$ ls sdks/vpcmod
README.md src

$ ls sdks/vpcmod
vpcmod-6.0.0.yaml

Pulumi automatically links the generated SDK into your project.

Using the Module in Your Code

Now you can use the module with full IntelliSense support:

import * as pulumi from "@pulumi/pulumi";
import * as vpcmod from '@pulumi/vpcmod';

const vpc = new vpcmod.Module("test-vpc", {
 azs: ["us-west-2a", "us-west-2b"],
 name: `test-vpc-${pulumi.getStack()}`,
 cidr: "10.0.0.0/16",
 public_subnets: [
 "10.0.1.0/24",
 "10.0.2.0/24",
 ],
 private_subnets: [
 "10.0.3.0/24",
 "10.0.4.0/24",
 ],
 enable_nat_gateway: true,
 single_nat_gateway: true,
});

export const publicSubnets = vpc.public_subnets;
export const privateSubnets = vpc.private_subnets;

import pulumi
import pulumi_aws as aws
import pulumi_vpcmod as vpcmod

aws_provider = aws.Provider("awsprovider", region="us-east-1")

vpcmod_provider = vpcmod.Provider("vpcprovider", aws=aws_provider.terraform_config().result)

vpc = vpcmod.Module(
 "test-vpc",
 azs=["us-west-2a", "us-west-2b"],
 name=f"test-vpc{pulumi.get_stack()}",
 cidr="10.0.0.0/16",
 public_subnets=[
 "10.0.1.0/24",
 "10.0.2.0/24",
 ],
 private_subnets=[
 "10.0.3.0/24",
 "10.0.4.0/24",
 ],
 enable_nat_gateway=True,
 single_nat_gateway=True,
)

pulumi.export("publicSubnets", vpc.public_subnets)
pulumi.export("privateSubnets", vpc.private_subnets)

package main

import (
 "fmt"

 "github.com/pulumi/pulumi-terraform-module/sdks/go/vpcmod/v6/vpcmod"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func run(ctx *pulumi.Context) error {
 stack := ctx.Stack()

 vpc, err := vpcmod.NewModule(ctx, "test-vpc", &vpcmod.ModuleArgs{
 Azs: pulumi.StringArray{
 pulumi.String("us-west-2a"),
 pulumi.String("us-west-2b"),
 },
 Name: pulumi.String(fmt.Sprintf("test-vpc-%s", stack)),
 Cidr: pulumi.String("10.0.0.0/16"),
 Public_subnets: pulumi.StringArray{
 pulumi.String("10.0.1.0/24"),
 pulumi.String("10.0.2.0/24"),
 },
 Private_subnets: pulumi.StringArray{
 pulumi.String("10.0.3.0/24"),
 pulumi.String("10.0.4.0/24"),
 },
 Enable_nat_gateway: pulumi.Bool(true),
 Single_nat_gateway: pulumi.Bool(true),
 })
 if err != nil {
 return err
 }

 ctx.Export("publicSubnets", vpc.Public_subnets)
 ctx.Export("privateSubnets", vpc.Private_subnets)

 return nil
}

func main() {
 pulumi.Run(run)
}

using System.Collections.Generic;
using Pulumi;
using Vpc = Pulumi.Vpcmod;

return await Deployment.RunAsync(() =>
{
 var vpc = new Vpc.Module("test-vpc", new Vpc.ModuleArgs
 {
 Azs = new string [] {
 "us-west-2a",
 "us-west-2b",
 },
 Cidr = "10.0.0.0/16",
 Public_subnets = new string[] {
 "10.0.1.0/24",
 "10.0.2.0/24"
 },
 Private_subnets = new string[] {
 "10.0.1.0/24",
 "10.0.2.0/24"
 },
 Enable_nat_gateway=true,
 Single_nat_gateway=true
 });

 return new Dictionary<string, object?>
 {
 ["publicSubnets"] = vpc.Public_subnets,
 ["privateSubnets"] = vpc.Private_subnets
 };
});

package myproject;

import com.pulumi.Context;
import com.pulumi.Pulumi;
import com.pulumi.core.Output;
import com.pulumi.deployment.Deployment;
import com.pulumi.vpcmod.Module;
import com.pulumi.vpcmod.ModuleArgs;

public class App {

 public static void main(String[] args) {
 Pulumi.run(ctx -> {
 final var stackName = Deployment.getInstance().getStackName();

 final var vpc = new Module("test-vpc", ModuleArgs.builder()
 .name("test-vpc-"+stackName)
 .azs("us-west-2a", "us-west-2b")
 .cidr("10.0.0.0/16")
 .public_subnets("10.0.1.0/24", "10.0.2.0/24")
 .private_subnets("10.0.3.0/24", "10.0.4.0/24")
 .enable_nat_gateway(true)
 .single_nat_gateway(true)
 .build());

 ctx.export("publicSubnets", vpc.public_subnets());
 ctx.export("privateSubnets", vpc.private_subnets());
 });
 }
}

name: my-program

runtime: yaml

resources:
 vpc:
 type: vpcmod:index:Module
 properties:
 azs:
 - us-west-2a
 - us-west-2b
 name: test-vpc-${pulumi.stack}
 cidr: 10.0.0.0/16
 public_subnets:
 - 10.0.1.0/24
 - 10.0.2.0/24
 private_subnets:
 - 10.0.3.0/24
 - 10.0.4.0/24
 enable_nat_gateway: true
 single_nat_gateway: true

outputs:
 publicSubnets: ${vpc.public_subnets}
 privateSubnets: ${vpc.private_subnets}

packages:
 vpcmod:
 source: terraform-module
 version: 0.1.7
 parameters:
 - terraform-aws-modules/vpc/aws
 - 6.0.0
 - vpcmod

Seamless Deployment

When you run pulumi up, Pulumi orchestrates the Terraform module execution while providing its signature preview and deployment experience:

Previewing update (dev)

View in Browser (Ctrl+O): https://app.pulumi.com/anton-pulumi-corp/anton-blog/dev/previews/5851682a-a3e3-4f47-a50e-3d7b4f4e6c34

 Type Name Plan
 + pulumi:pulumi:Stack anton-blog-dev create
 + └─ vpcmod:index:Module test-vpc create
 + ├─ vpcmod:tf:aws_route_table_association module.test-vpc.aws_route_table_association.private[0] create
 + ├─ vpcmod:tf:aws_subnet module.test-vpc.aws_subnet.private[1] create
 + ├─ vpcmod:tf:aws_route_table_association module.test-vpc.aws_route_table_association.public[1] create
 + ├─ vpcmod:tf:aws_subnet module.test-vpc.aws_subnet.private[0] create
 + ├─ vpcmod:tf:aws_route_table module.test-vpc.aws_route_table.private[0] create
 + ├─ vpcmod:tf:aws_subnet module.test-vpc.aws_subnet.public[1] create
 + ├─ vpcmod:tf:aws_nat_gateway module.test-vpc.aws_nat_gateway.this[0] create
 + ├─ vpcmod:tf:aws_route module.test-vpc.aws_route.public_internet_gateway[0] create
 + ├─ vpcmod:tf:aws_route module.test-vpc.aws_route.private_nat_gateway[0] create
 + ├─ vpcmod:tf:aws_default_network_acl module.test-vpc.aws_default_network_acl.this[0] create
 + ├─ vpcmod:tf:aws_default_route_table module.test-vpc.aws_default_route_table.default[0] create
 + ├─ vpcmod:tf:aws_subnet module.test-vpc.aws_subnet.public[0] create
 + ├─ vpcmod:tf:aws_internet_gateway module.test-vpc.aws_internet_gateway.this[0] create
 + ├─ vpcmod:tf:aws_vpc module.test-vpc.aws_vpc.this[0] create
 + ├─ vpcmod:tf:aws_default_security_group module.test-vpc.aws_default_security_group.this[0] create
 + ├─ vpcmod:tf:aws_route_table module.test-vpc.aws_route_table.public[0] create
 + ├─ vpcmod:tf:aws_eip module.test-vpc.aws_eip.nat[0] create
 + ├─ vpcmod:tf:aws_route_table_association module.test-vpc.aws_route_table_association.public[0] create
 + └─ vpcmod:tf:aws_route_table_association module.test-vpc.aws_route_table_association.private[1] create

Resources:
 + 21 to create

The infrastructure deploys successfully, and the Terraform state is automatically stored securely in Pulumi Cloud with full secret encryption support.

Migration Made Simple

For teams starting with existing Terraform code, the migration process is straightforward. Simply annotate your modules with a special comment:

// @pulumi-terraform-module vpcmod
module "my-vpc" {
 source = "terraform-aws-modules/vpc/aws"
 version = "6.0.0"
 azs = ["us-west-2a", "us-west-2b"]
 name = "test-vpc-123"
 public_subnets = ["10.0.1.0/24", "10.0.2.0/24"]
 private_subnets = ["10.0.3.0/24", "10.0.4.0/24"]
 enable_nat_gateway = true
 single_nat_gateway = true
}

Then run the conversion (replace typescript with your language of choice):

pulumi convert --from terraform --language typescript --out my-pulumi-project

The resulting project is ready to deploy with full Pulumi functionality.

Configuration Management

One of the key benefits is seamless configuration management. Instead of maintaining separate configurations for Pulumi and Terraform providers, you can reuse your existing Pulumi provider configuration:

const awsProvider = new aws.Provider("awsprovider", {
 region: "us-west-2",
 // more configuration
});

// Pass the AWS configuration to your VPC module provider
const vpcmodProvider = new vpcmod.Provider("vpcprovider", {
 "aws": awsProvider.terraformConfig().result
});

// Use the VPC module provider in your Module
const vpc = new vpcmod.Module("test-vpc", {...},
 {provider: vpcmodProvider}
);

import pulumi
import pulumi_aws as aws
import pulumi_vpcmod as vpcmod

aws_provider = aws.Provider("awsprovider", region="us-west-2")

# Pass the AWS configuration to your VPC module provider
vpcmod_provider = vpcmod.Provider("vpcprovider", aws=aws_provider.terraform_config().result)

# Use the VPC module provider in your Module
vpc = vpcmod.Module("test-vpc", ..., opts=pulumi.ResourceOptions(provider=vpcmod_provider))

func run(ctx *pulumi.Context) error {
 stack := ctx.Stack()

 awsProvider, err := aws.NewProvider(ctx, "awsprovider", &aws.ProviderArgs{
 Region: pulumi.String("us-west-2"),
 })
 if err != nil {
 return err
 }

 awsProviderTfConfig, err := awsProvider.TerraformConfig(ctx)
 if err != nil {
 return err
 }

 // Pass the AWS configuration to your VPC module provider
 vpcProvider, err := vpcmod.NewProvider(ctx, "vpcprovider", &vpcmod.ProviderArgs{
 Aws: awsProviderTfConfig.Result(),
 })
 if err != nil {
 return err
 }

 // Use the VPC module provider in your Module
 vpc, err := vpcmod.NewModule(ctx, "test-vpc", &vpcmod.ModuleArgs{...}, pulumi.Provider(vpcProvider))
 if err != nil {
 return err
 }

 ctx.Export("publicSubnets", vpc.Public_subnets)
 ctx.Export("privateSubnets", vpc.Private_subnets)

 return nil
}

This demonstrates how Terraform modules integrate seamlessly with existing Pulumi programs, allowing you to compose infrastructure components naturally.

What’s Supported

The integration provides comprehensive support for Pulumi’s core features:

Full Lifecycle Management: Preview, update, refresh, and destroy operations work exactly as expected
Resource Visibility: The entire resource tree is visible in Pulumi’s UI and CLI
Type Safety: Generated TypeScript interfaces provide IntelliSense and compile-time checking
Secret Management: First-class support for Pulumi’s secret encryption
Provider Configuration: Full control over Terraform provider settings

Current Limitations

As with any new technology integration, there are some current limitations to be aware of:

Transforms resource options are not supported
Targeted operations like pulumi up --target are not available
The protect resource option is not supported
Dependent resources may have limited support in some scenarios

Looking Ahead

This release represents a significant milestone in our journey to make infrastructure as code accessible to every developer. We’re already working on the next phase, which includes:

Enhanced state import capabilities for seamless Terraform-to-Pulumi migrations
Improved type inference for complex module schemas
Expanded support for additional Terraform features

The Path Forward

While this capability enables you to preserve your existing Terraform modules, we encourage teams to view this as a stepping stone rather than a final destination. The full benefits of modern infrastructure as code—including AI-assisted development, advanced testing capabilities, and seamless integration with modern development workflows—are best realized with native Pulumi components.

With AI developer tools making code conversion easier than ever, we’re committed to helping teams migrate their Terraform modules to native Pulumi when they’re ready. This capability ensures you can start your modernization journey today without being blocked by existing technical debt.

Get Started Today

The Terraform module support is available immediately. Install the latest Pulumi CLI and try it out with your existing Terraform modules. We’ve prepared comprehensive examples showcasing real-world scenarios, from simple VPC deployments to complex multi-module architectures.

For detailed documentation on using Terraform modules in Pulumi, including advanced configuration options, troubleshooting guides, and best practices, see our comprehensive guide.

If you encounter any issues or have suggestions for improvement, please open an issue or reach out in our Community Slack. Your feedback is invaluable as we continue to evolve this capability.

This release embodies our commitment to meeting developers where they are while providing a clear path forward to modern infrastructure practices. We’re excited to see how teams use this capability to accelerate their infrastructure modernization journey.

Converting Full Terraform States to Pulumi

Fraser Waters — Wed, 25 Oct 2023 00:00:00 +0000

Building on our work of converting Terraform projects, we now have support for adopting resources from Terraform state.

In the prior blog post about converting from Terraform we called out that while it was great that the converter could migrate a Terraform project, but what if you had existing infrastructure resources.

Historically we’ve shared some AWS specific code for TypeScript and Go to import from .tfstate files (previously available in the tf2pulumi repository, now replaced by the Terraform converter plugin). These have been useful, but they’re limited to the AWS provider and either Go or TypeScript programs.

From v3.87.0 we have extended our converter system to also include state in addition to code. So now it’s possible to import resources from other state systems, the first of which is Terraform.

Converting a Real World Workspace

Terraform stores its state in workspaces. In the most simple case this is just a terraform.tfstate file in the directory with the code. For remote state it can be fetched with terraform state pull.

Given a terraform.tfstate file we can set up a Pulumi stack and tell it to import the resource from Terraform.

$ pulumi stack init
$ pulumi import --from terraform ./terraform.tfstate

Pulumi will try to convert the resources in the Terraform state into their matching Pulumi types and import strings (normally just the resource ID).

Some resources don’t use their ID as the import string. For some well known AWS resources the converter will handle this; for others, they might still cause an error at import time. However, if the import does fail (for any reason) the import file listing all the resources will be written out to the current directory, allowing you the opportunity to manually fix up issues with type tokens or import strings, and then try again using pulumi import --file.

We’re working on ensuring more types correctly translate, so feel free to open issues with any import errors you get while using this new converter to help us improve.

Get Started

Support for the new pulumi import --from terraform command is now available in v3.87.0 of the Pulumi CLI. Download the latest Pulumi CLI and give the new converter a try today. If you run into any issues, please let us know or reach out in the Pulumi Community Slack with any questions!

Pulumi adoption made easy with the new Migration Hub

Joe Duffy — Tue, 22 Aug 2023 00:00:00 +0000

Today we are launching Pulumi’s new Migration Hub, a comprehensive guide to help you seamlessly adopt Pulumi no matter where you are coming from, whether that’s Terraform, CloudFormation, … or even manually provisioned resources not yet governed by an infrastructure as code solution. Our new Expert Services group is ready to roll up their sleeves to help you adopt Pulumi faster. The Migration Hub also features many commercial offers for open source foundations, startups, and complementary migration, to minimize switching costs and risks. It’s never been easier to adopt Pulumi.

Why Pulumi?

Nearly 2,000 customers and over 130,000 practitioners have chosen Pulumi for their infrastructure as code needs. About half the time, these end users have an existing infrastructure as code solution that has stopped meeting their needs. The other half of the time, infrastructure may have been created in the cloud console (“click-ops”), using a cloud CLI, or some other manual provisioning tool, but now needs to become repeatable and scalable using infrastructure as code.

The most common reasons teams prefer Pulumi include:

Best-in-class productivity
Accessible to the entire organization, thanks to industry standard languages
Ability to deal with complexity at scale, thanks to stacks, config, sharing and reuse, and more
Perfect confidence thanks to policies and testing
Secure by-default, with built-in support for secrets
Highly programmable engine can be embedded anywhere, to build custom platforms and tools
Instant collaborative bridge between developers and infrastructure experts

Thanks to Pulumi, teams are able to do more with less, ship faster with confidence, tackle very challenging cloud infrastructure projects and scale with them, empower more of the team to be self-serve and collaborate more closely, and ultimately turn the cloud into a competitive advantage.

Pulumi is also true Apache 2.0 open source and we welcome anyone who wishes to build atop it.

Migration

Of course, it’s one thing to choose Pulumi, but another thing entirely to adopt it and get all of your infrastructure under its management, especially if you have existing infrastructure. That might include converting existing code, or even just importing existing manually provisioned cloud resources.

There are two major approaches to this challenge.

Self-Serve Migration

Pulumi offers many tools and techniques should you want to perform a migration on your own.

In the most general form, Pulumi provides an import command which can import any existing infrastructure no matter how it was originally created, click-ops included.

Pulumi also offers conversion tools for other popular infrastructure as code solutions, including HashiCorp Terraform, AWS CloudFormation, Azure Resource Manager (ARM), or Kubernetes YAML.

Although the time to migrate existing infrastructure varies wildly based on the size and complexity of your scenario, migrating a single cloud environment can often take as little as a few hours. And thanks to Pulumi being multi-language at its core, these tools will support you the same no matter which language you choose, JavaScript, TypeScript, Python, Go, C#, Java, and YAML alike.

To learn more about performing your own migration, please read this user guide.

Let Us Help You Out with Expert Services!

Although self-serve is an option, there can often still be uncertainty around this approach.

It may not be clear how much time a full migration will take, leaving the possibility that you’ll be left with multiple infrastructure as code solutions – either temporarily or even permanently. Although the tools are a huge help, they don’t always get you 100% of the way there. We also often hear that skilling yourself up on a one-time migration activity takes away from critical time spent on business initiatives.

All of this adds friction, risk, and uncertainty to the Pulumi adoption decision. The last thing we want is for the one-time migration to prevent you from adopting Pulumi. Thankfully, we are here to help!

Today we are announcing the formation of Pulumi’s Expert Services group. This team of industry and infrastructure as code experts is on standby and ready to roll up their sleeves to help you with migration, training, and other services efforts. No project is too small: we are currently working with customers with 100KLOCs of legacy infrastructure as code that needs to be migrated, for example.

Today we are also announcing an extension to our Terraform Migration Offer, in which we will bundle migration of your Terraform projects with your Pulumi Enterprise or Business Critical subscription. Today we are excited to announce that this same package is now available for all migration efforts, including AWS CloudFormation, Azure Resource Manager (ARM), …, or even infrastructure that was manually provisioned. We are also prepared to scale up from there depending on your needs.

Success Stories

Over time, Pulumi is becoming a more comprehensive platform, with new services like Pulumi Deployments for multi-cloud deployment workflows and Pulumi Insights for search, analytics, and AI for your cloud infrastructure. Pulumi also offers policies and secrets as core features of the platform. Enterprises are increasingly betting on Pulumi for their entire cloud management and platform needs.

By moving to Pulumi’s platform, many customers have seen considerable benefits:

Atlassian significantly increased developer velocity, increasing productivity by more than 2x and substantially decreasing the amount of time spent on maintenance tasks.

Mercedes-Benz got the whole team shipping faster and with more confidence by building a platform that helped developers be self-serve with infrastructure while scaling up to 100s of Kubernetes clusters worldwide.

Snowflake rapidly improved time to market by betting on Pulumi, empowering their engineers, and delivering the Data Cloud. Snowflake ultimately beat their IPO deadlines and knew legacy infrastructure as code wouldn’t do it.

Washington Trust Bank modernized their cloud infrastructure and practices, speeding up internal delivery, while also increasing their confidence thanks to policy as code guardrails.

Fauna adopted a born in the cloud mindset and was able to build a multi-cloud, highly scalable, modern SaaS application, actually increasing reliability even in the face of growing complexity.

In each of these cases, the customer evaluated Pulumi against alternative infrastructure as code solutions, but ultimately chose Pulumi due to the aforementioned benefits.

Special Offers

In addition to migration services, Pulumi offers two programs to help get up and running quickly:

Open Source: Any open source foundation or project can use Pulumi Team Edition free of charge.
Startup Discount: Any early stage startup gets $10,000 free credits and access to Pulumi Enterprise.

Calling All Partners!

As Pulumi’s market adoption has skyrocketed, we’ve begun to build out our official partner program. Although we don’t have anything major to announce (yet), we’d love to talk to you if infrastructure as code is an important part of your own customers’ needs. That includes ISV product integrations, global and regional systems integrators and consultants, and more. Note that Pulumi is already available in the AWS and Azure marketplaces. If you are interested in discussing a partnership, please contact us.

Get Started Today

Migrating from an existing solution can seem like a daunting task, but with Pulumi’s new Migration Hub and Expert Services, it has never been easier to adopt Pulumi than it is today. To learn more, visit the Migration Hub or simply contact us – we would love to hear from you.

Converting Full Terraform Programs to Pulumi

Justin Van Patten — Mon, 12 Jun 2023 00:00:00 +0000

Over the last 2 years, we’ve seen an increasing trend of cloud development teams migrating to Pulumi from Terraform. These teams often have experience with and meaningful investment in Terraform, but have also typically run into limits of expressivity, productivity, scalability, or reliability with their existing tools. One of the first questions we hear when they decide to move to Pulumi is “how will I migrate my existing Terraform projects over?”.

Today, we’re excited to announce new support for converting whole Terraform projects to Pulumi via the pulumi convert command in the Pulumi CLI. The new Terraform converter includes support for Terraform modules, core features of Terraform 1.4, and the majority of Terraform built-in functions, converting to Pulumi TypeScript, Python, Go, or C#. The new converter can significantly reduce the amount of time it takes to migrate Terraform to Pulumi. Let’s dig in to learn more about the new converter and how to use it.

Historically, we have offered a separate tf2pulumi tool to convert small snippets of Terraform to Pulumi (now replaced by the Terraform converter plugin). The new converter is no longer a separate tool. As of v3.71.0, you can run the new converter directly from the Pulumi CLI with the pulumi convert --from terraform command. And you can convert more than small snippets – the new converter supports converting full Terraform programs.

The new support in pulumi convert builds upon Pulumi’s CrossCode foundations for providing universal infrastructure as code support across a wide variety of programming languages and conversion tooling between them. It also introduces a new concept of converter plugin in the Pulumi engine, which allows conversion tools from other Infrastructure as Code platforms to be integrated into the same pulumi convert experience in the future, both as part of the core project, as well as by other ecosystem partners and contributors.

Several common use cases are supported via the new pulumi convert --from terraform support in the Pulumi CLI:

Converting your organization’s existing Terraform projects to Pulumi
Converting your organization’s existing Terraform modules to Pulumi, to be consumed as part of existing Pulumi projects
Converting 3rd party open source Terraform modules or projects which address a use case you want to incorporate into your existing Pulumi projects

Supported Terraform Features

The following major features are supported:

Variables, outputs, resources, and data sources
Terraform modules are converted to Pulumi components
Almost all HCL2 expression syntax

In cases where the converter does not yet support a feature, the pulumi convert command succeeds but generates a TODO in the form of a call to a notImplemented not_implemented notImplemented NotImplemented function that will need to be filled in manually. For most projects, the converter should be able to convert 90-95% of the code without any TODOs, with only a small percentage of items to address manually, significantly reducing migration time compared to doing an entire migration by hand. We are actively improving the converter by adding support for missing features and improving the overall quality of the converted code to reduce the amount of manual fix-ups required.

Converting a Real World Program

Let’s walk through converting a Terraform codebase to Pulumi. Avant Terraform Vault Setup is an open source project that provides a high-availability installation of Vault using a variety of managed AWS services. It defines a fairly complex installation with dozens of AWS resources in over 1,000 lines of Terraform HCL, including the main program and a Vault module. Let’s convert it to Pulumi.

First, clone the repo and cd into the directory containing the Terraform project:

$ git clone https://github.com/avantoss/vault-infra.git
$ cd vault-infra/terraform/main

Next, run the converter:

$ pulumi convert --from terraform --language typescript --out pulumi

$ pulumi convert --from terraform --language python --out pulumi

$ pulumi convert --from terraform --language go --out pulumi

$ pulumi convert --from terraform --language csharp --out pulumi

The converted code is generated in the specified pulumi output directory. A complete Pulumi project is generated, including two primary code files specific to this this program’s conversion:

index.js index.ts __main__.py main.go Program.cs Program.fs Program.vb App.java Pulumi.yaml main.tf contains the converted code for the main program
vault.ts vault.py vault.go vault.cs contains the Vault Pulumi component, converted from the Terraform module

Addressing TODOs

The file for the Vault component makes up the bulk of the implementation and contains some TODOs emitted by the converter that must be addressed manually.

For example, the converter doesn’t yet support the replace, element, and split built-in functions.

const plainDomain = notImplemented("replace(element(split(\":\",var.vault_dns_address),1),\"////\",\"\")");

plain_domain = not_implemented("replace(element(split(\":\",var.vault_dns_address),1),\"////\",\"\")")

plainDomain := notImplemented("replace(element(split(\":\",var.vault_dns_address),1),\"////\",\"\")");

var plainDomain = NotImplemented("replace(element(split(\":\",var.vault_dns_address),1),\"////\",\"\")");

We can fill in an implementation. Note that we get to use the full expressiveness of the native and familiar string manipulation libraries in our target programming language, instead of the relatively constrained options of the Terraform DSL.

const plainDomain = pulumi.output(args.vaultDnsAddress)
 .apply(a => a.split(":")[1].replace("//", ""));

plain_domain = pulumi.Output.from_input(args["vaultDnsAddress"]) \
 .apply(lambda a: a.split(":")[1].replace("//", ""))

plainDomain := args.VaultDnsAddress.ToStringOutput().ApplyT(func(a string) string {
 return strings.ReplaceAll(strings.Split(a, ":")[1], "//", "")
}).(pulumi.StringOutput)

var plainDomain = args.VaultDnsAddress
 .Apply(a => a.Split(':')[1].Replace("//", ""));

Wrapping Up

After addressing the remaining TODOs and some other tweaks so that the code compiles, we can now run the converted program with pulumi up to provision the Vault installation with Pulumi.

The converter has saved us a ton of time, converting over 1,000 lines of Terraform to a modern Pulumi language, with only a small number of manual fix-ups required. From here, we can leverage our IDE and compiler to further refactor and improve the code, one of the many benefits of Pulumi!

Convert your Terraform to Pulumi

Run pulumi convert to migrate your existing Terraform projects and modules to TypeScript, Python, Go, or C#, then refine the result with your language’s own tooling.

Learn more

Importing State

It’s great that the new converter can migrate Terraform projects to Pulumi for new deployments, but what if you want to import existing resource states from a .tfstate file to avoid unnecessarily recreating your infrastructure?

If you’re using TypeScript or Go, there is some additional code that can be added to your converted Pulumi program to import resource states from a .tfstate file. See the Importing Resources reference documentation for more details.

We’re working to make this even more seamless with built-in support for importing state from .tfstate files in a future update coming soon.

Get Started

Support for the new pulumi convert --from terraform command is now available in v3.71.0 of the Pulumi CLI. Download the latest Pulumi CLI and give the new converter a try today. If you run into any issues, please let us know or reach out in the Pulumi Community Slack with any questions!

How Skai Migrated to Amazon Keyspaces with Pulumi

Danny Zalkind — Tue, 16 May 2023 00:00:00 +0000

Danny Zalkind is the Senior Director of Infrastructure Engineering for Skai, an award-winning intelligent marketing platform. He brings his 15 years of experience of managing tech teams to his current role where he’s dedicated to allow Skai R&D to efficiently produce and serve software. You can find him on Linkedin.

As Skai continues its journey towards fully migrating to the cloud using Pulumi, we’ve taken another large bite out of the migration pie, moving our most critical data to AWS on top of Amazon Keyspaces, an Apache Cassandra–compatible database service.

This blog post will dive into how we used Pulumi and AWS Lambda to perform the migration.

Offloading a piece of the pie to AWS

First, some background on what we do. Skai helps marketers work smarter and faster with a unified platform to reach and convert shoppers. An important service we provide to our clients is optimization of their digital marketing campaign efficiency. We store billions of events per day to enable clients to continuously improve the value they get from their ad spend.

Our on-premises setup included a 100-node Apache Cassandra cluster to be able to withstand the scale of billions of events, some stored for prolonged periods of time. Maintaining and scaling this setup was a high effort from our Data Infra teams. It was clear to us it was time to offload that piece of the pie to our cloud provider.

Why Amazon Keyspaces

When looking for a solution, we were looking for a service that would mark the following checkboxes:

Fully serverless to allow us focus on our business instead of server maintenance.
Scalability should be automatic and easy to perform.
High availability and multi-region built-in.
Cost effective pay-as-you-go and commitment-based model.
Extensive support and rich documentation.
Fully automatable using Pulumi, our main Infrastructure as Code framework.

Migration method

Let’s take a look at how we set up the migration. We decided to migrate the data using a dual-write method, writing new data to both our on-prem Cassandra cluster and Keyspaces. Any historical data needed could be fetched from Snowflake, our data lake solution, using our Apache Spark pipeline.

Finally, to make sure that the data is valid and we have no data discrepancies on the Keyspaces side, we added a Lambda function to periodically sample the data and compare it to the data in our on-prem cluster.

We made a conscious decision to decouple the application code from the provisioning of the infrastructure, letting Pulumi handle all resource creation and configuration, while the application team used our standard CI/CD pipeline to deliver their code to the service owning the Keyspaces data.

Let’s start provisioning

Keyspaces requires creating a logical entity called a keyspace, which groups related tables relevant for one or more applications. After that, we iterate over our config and create each table with the relevant columns and the preferred provisioning mode — either “on-demand” or “provisioned,” depending on our workload pattern. A provisioned workload in pre-allocated and thus significantly (x7) lower in hourly price, suitable mainly for use cases where the workload is quite predictable with no spikes.

Finally we populate AWS tags in order to be able track costs and performance metrics.

The following Python code illustrates the process:

import pulumi
import pulumi_aws as aws
from pulumi_komponents.account_provider.account_provider import get_account_provider

# Define resource tagging
def __populate_tags(aws_config: pulumi.Config, component_type: str, tags: dict):
 return {
 "Project": pulumi.get_project(),
 "Environment": aws_config.require("account"),
 "Type": component_type,
 **tags,
 }

def pulumi_program():
 aws_config = pulumi.Config("aws")
 keyspaces_config = pulumi.Config("keyspaces")
 tags = keyspaces_config.require_object("tags")
 provider = get_account_provider(aws_config)

 # Define keyspace
 keyspace = aws.keyspaces.Keyspace(
 resource_name=f'keyspaces-{keyspaces_config.require("name")}',
 name=keyspaces_config.require("name"),
 tags=__populate_tags(aws_config, "keyspace", tags),
 opts=pulumi.ResourceOptions(provider=provider, protect=True),
 )

 # Define and iterate over keyspace tables
 tables = keyspaces_config.require_object("tables")

 for table_name, table_ddl in tables.items():
 columns = []
 partition_keys = []
 clustering_keys = []
 static_columns = []
 for col in table_ddl.get("schema").get("columns"):
 name, type = col.popitem()
 columns.append(
 aws.keyspaces.TableSchemaDefinitionColumnArgs(name=name, type=type)
 )
 for par in table_ddl.get("schema").get("partition_keys"):
 partition_keys.append(
 aws.keyspaces.TableSchemaDefinitionPartitionKeyArgs(name=par)
 )
 for cl in table_ddl.get("schema").get("clustering_keys") or []:
 name, order_by = cl.popitem()
 clustering_keys.append(
 aws.keyspaces.TableSchemaDefinitionClusteringKeyArgs(
 name=name, order_by=order_by
 )
 )
 for s_col in table_ddl.get("schema").get("static_columns") or []:
 static_columns.append(
 aws.keyspaces.TableSchemaDefinitionStaticColumnArgs(name=s_col)
 )

 mode = table_ddl.get("throughput").get("mode")

 aws.keyspaces.Table(
 resource_name=f"keyspaces-table-{table_name}",
 keyspace_name=keyspace.name,
 table_name=table_name,
 schema_definition=aws.keyspaces.TableSchemaDefinitionArgs(
 columns=columns,
 partition_keys=partition_keys,
 clustering_keys=clustering_keys,
 static_columns=static_columns,
 ),

 # Set capacity mode and units for read and write per Table
 capacity_specification=aws.keyspaces.TableCapacitySpecificationArgs(
 throughput_mode=mode,
 read_capacity_units=table_ddl.get("throughput").get("rcu")
 if mode == "PROVISIONED"
 else None,
 write_capacity_units=table_ddl.get("throughput").get("wcu")
 if mode == "PROVISIONED"
 else None,
 ),

 # Set general table properties such as TTL and recovery method
 default_time_to_live=None,
 ttl=aws.keyspaces.TableTtlArgs(status="ENABLED"),
 point_in_time_recovery=aws.keyspaces.TablePointInTimeRecoveryArgs(
 status="ENABLED"
 ),

 # Populate AWS resource tagging
 tags=__populate_tags(
 aws_config,
 "table",
 {
 "Table-type": "unarchived"
 if table_name.endswith("_unarchived")
 else "standard",
 **tags,
 },
 ),
 opts=pulumi.ResourceOptions(
 parent=keyspace, provider=provider, protect=True
 ),
 )

Comparison Lambda

Next we define and create a Lambda function that samples and compares data between the data newly ingested into Keyspaces, our on-premise Cassandra cluster, and our data lake Snowflake.

We created an abstract class implementation of AWS Lambda to standardize our various Lambda implementations and keep the code readable. This class returns all needed arguments, configuration, layers, etc. Here is some Python code that shows our abstraction, named AwsFunction:

class AwsFunction(AbstractAwsFunction):
 def __init__(self, args: AwsFunctionArgs):
 super().__init__(resource_type="AwsLambda:Function", args=args)

 # Set Lambda zip artificate path
 def _create_function(self) -> aws.lambda_.Function:
 code_archive_path = f"/tmp/{self.args.name}.zip"
 with zipfile.ZipFile(code_archive_path, "w") as zip_obj:
 zip_sub_folders(
 root_source=self.args.functions_absolute_path,
 destination=code_archive_path,
 selected_sub_folders=[ZIP_COMMON_DIR, self.args.code_directory_name],
 zip_obj=zip_obj,
 )

 # Get iam role, kms keys and layers
 self.function_role = self._get_function_role()
 self.kms_key = self._create_kms()
 layers_arn, layers = self.__build_layers()

 # Return function outputs
 return aws.lambda_.Function(
 resource_name=generate_resource_name(self.args.name, "function"),
 name=self.args.function_name,
 role=self.function_role.arn,
 package_type=self.args.package_type,
 architectures=[self.args.architecture.lambda_value],
 handler=self.args.handler,
 memory_size=self.args.memory_size,
 reserved_concurrent_executions=self.args.concurrent_executions,
 runtime=self.args.runtime,
 code=pulumi.FileArchive(code_archive_path),
 timeout=self.args.timeout,
 layers=layers_arn,
 tags=self._populate_tags(component="lambda", resource_type="function"),
 kms_key_arn=self.kms_key.arn,
 environment=aws.lambda_.FunctionEnvironmentArgs(
 variables=self.args.environment_variables.to_dict()
 ),
 vpc_config=self.args.vpc_config,
 opts=ResourceOptions(
 provider=self.args.provider, depends_on=[self.kms_key] + layers
 ),
 )

Now we instantiate the abstract class as part of our “building block” component class, create the function, triggers and invocation:

class AwsLambda(ComponentResource):
 def __init__(self, args: AwsLambdaArgs):
 super().__init__(f"AwsLambda", args.lambda_name)

 # Define the function nased on the abstract class
 aws_function = (
 AwsFunction(args.aws_function_args)
 if args.aws_function_args.package_type == "Zip"
 else AwsFunctionContainer(args.aws_function_args)
 )

 # Add triggers
 triggers = {}
 for trigger, config in args.aws_function_args.lambda_triggers.items():
 triggers[trigger] = LAMBDA_TYPES_TO_DEF[trigger](
 config, aws_function.function, args.provider, args.tags
 )

 # Add async invocation config
 FunctionEventInvokeConfig(
 generate_resource_name(args.lambda_name, "event-invoke-config"),
 function_name=aws_function.function.name,
 destination_config=FunctionEventInvokeConfigDestinationConfigArgs(
 on_failure=FunctionEventInvokeConfigDestinationConfigOnFailureArgs(
 destination=args.aws_function_args.on_failure_arn,
 )
 if args.aws_function_args.on_failure_arn
 else None,
 on_success=FunctionEventInvokeConfigDestinationConfigOnSuccessArgs(
 destination=args.aws_function_args.on_success_arn,
 )
 if args.aws_function_args.on_success_arn
 else None,
 ),
 qualifier=args.aws_function_args.qualifier,
 maximum_event_age_in_seconds=args.aws_function_args.maximum_event_age_in_seconds,
 maximum_retry_attempts=args.aws_function_args.maximum_retry_attempts,
 opts=ResourceOptions(provider=args.provider),
 )

Now we’re ready to call our class to create our comparison Lambda:

from pulumi_komponents.awslambda import AwsLambda, AwsLambdaArgs
from pulumi import Config

config = Config()

AwsLambda(AwsLambdaArgs(config))

Note: This only a partial code example, meant to demonstrate the way we use object-oriented paradigms to create reusable building-blocks of code to be consumed by various platform and R&D teams to provision AWS resources.

Putting it all together

Once we have our Keyspaces setup in place and the confidence that our dual-write process is writing the data correctly, we are now able to wait enough time to accumulate the data on the AWS side, using a feature toggle in our application to switch our backend systems to work with AWS Keyspaces.

Outcome

Following the successful migration, we have gained 50% reduction in backend process runtime, increased event processing throughput by 400% and freed up 30% of our team’s time to do other things, instead of maintaining cluster infrastructure.

I hope this post provides readers with useful techniques and ideas for migrating large-scale databases to the public cloud.

How we manage GitHub at Pulumi with Pulumi

Guinevere Saenger — Tue, 22 Mar 2022 11:29:33 -0700

We recently updated this article to reflect that parentTeamId of the Team resource accepts a string instead of an int since v5.9.1 of the Pulumi Github package.

Ah, GitHub. The home of all developers. The place where we share code. The world’s most awkward social media site. The secret LinkedIn for techies. The tool we use for company org structure, work planning, code ownership, and permissions…

Wait.

That’s quite a lot.

GitHub is good at many things, but a full-on organization management tool it is not.

Have you ever needed your manager to manually enable admin permission on a repo for you? Or have you needed to page the CEO to add you to a team, because your manager was out that day? Have you ever wondered who is on what team? Or which team owns a repo? What if you change teams, or a team changes names? A reorg happens, and the “platform-integrations” team is no more, but we still need to call it that on GitHub because it is the team with all the repository accesses?

When I joined Pulumi in 2021, all of the above happened to me within my first few weeks.

We at Pulumi wanted to reduce this kind of management friction, and we decided to solve it the Pulumi way: with declarative infrastructure using the Pulumi GitHub provider.

The Goal

First, we set a few ground rules. We decided that administrative privileges over Pulumi’s GitHub repositories should flow as follows:

An employee is part of one or more GitHub teams.
Teams have appropriately scoped permissions over each repository they own.
Any Pulumi employee should be able to request code repository access in an official way.

Additionally, any current team membership and repository access should not be disrupted during development of this tool.

Getting Started

Create a new Pulumi program:

$ pulumi new go
This command will walk you through creating a new Pulumi project.

Enter a value or leave blank to accept the (default), and press <ENTER>.
Press ^C at any time to quit.

project name: (team-management)
project description: (A minimal Go Pulumi program)
Created project 'team-management'

Please enter your desired stack name.
To create a stack in an organization, use the format <org-name>/<stack-name> (e.g. `acmecorp/dev`).
stack name: (dev) pulumi/prod
Created stack 'prod'

Installing dependencies...

go: downloading github.com/pulumi/pulumi/sdk/v3 v3.26.1
Finished installing dependencies

Your new project is ready to go! ✨

To perform an initial deployment, run 'pulumi up'

$ ls
Pulumi.yaml go.mod go.sum main.go

We now have a pulumi project YAML configuration file and the beginnings of a small Go program all set up.

According to the provider configuration documentation, we need to add a properly scoped token, as well as set “pulumi” as our GitHub organization owner.

$ export GITHUB_TOKEN=YYYYYYYYYYYYYY
$ export GITHUB_OWNER=pulumi

Capturing Current State of Resources

Now, Pulumi is great at creating new infrastructure from scratch via code. But this wasn’t a from-scratch situation. We had to migrate existing resources—GitHub teams—to Pulumi, without disrupting anyone’s access.

Enter Pulumi Import.

What Pulumi Import does, in a nutshell, is find existing infrastructure by unique ID (in the GitHub provider’s case, the team ID), and add them to a Pulumi Stack. You can find the specific import instructions on the registry documentation for each resource. In this case we want the GitHub import instructions.

$ pulumi import github:index/team:Team animals 1234567
Previewing import (pulumi/prod)

 Type Name Plan
 pulumi:pulumi:Stack team-mgmt-prod
 = └─ github:index:Team animals import

Resources:
 = 1 to import
 12 unchanged

Do you want to perform this import? yes
Importing (pulumi/prod)

 Type Name Status
 pulumi:pulumi:Stack team-mgmt-prod
 = └─ github:index:Team animals imported

Resources:
 = 1 imported
 1 unchanged

Now that we have imported the Team resource, it is part of our Pulumi Stack. But it is not part of our main.go program yet. Helpfully, Pulumi Import replies with sample code on how to add the imported resource to your Pulumi program, in the language you selected for your project:

Please copy the following code into your Pulumi application. Not doing so will cause Pulumi to report that an update will happen on the next update command.

Please note that the imported resources are marked as protected. To destroy them you will need to remove the `protect` option and run `pulumi update` *before* the destroy will take effect.


package main

import (
 "github.com/pulumi/pulumi-github/sdk/v4/go/github"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 _, err := github.NewTeam(ctx, "animals", &github.TeamArgs{
 CreateDefaultMaintainer: pulumi.Bool(false),
 Description: pulumi.String("should get imported"),
 Name: pulumi.String("animals"),
 Privacy: pulumi.String("closed"),
 }, pulumi.Protect(true))
 if err != nil {
 return err
 }
 return nil
 })
}

After adding the team to main.go as suggested, this is the output of pulumi preview:

Previewing update (pulumi/prod)

 Type Name Plan Info
 pulumi:pulumi:Stack team-mgmt-prod

Resources:
 2 unchanged

Since pulumi preview shows no changes, we now know that our code reflects the existing infrastructure. It’s a bit funny to think about your program working well when it does nothing, but this was a huge first step in preserving existing infrastructure and ensuring all of our coworkers could continue their daily work uninterrupted!

To finish up, we unprotect the resource:

$ pulumi state unprotect 'urn:pulumi:prod::team-mgmt::github:index/team:Team::animals'

Creating Configuration

Next, we import the remaining teams and refactor the code to handle multiple teams at once. This is also the time to write our organization structure into a YAML configuration file.

---
org: "pulumi-demo--org"
teams:
 - name: "animals"
 slug: "animals"
 description: "A team in charge of animals"
 - name: "newteam"
 slug: "newteam"
 description: "I am a new team"

We then read this information into a Go struct with the necessary properties to pass into the Pulumi provider.

type Team struct {
 Name string `yaml:"name"`
 Description string `yaml:"description"`
 Slug string `yaml:"slug"`
}
type Organization struct {
 Org string `yaml:"org"`
 Teams []Team `yaml:"teams"`
}

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {

//import accurate team info from yaml
orgFilePath, err := filepath.Abs("./org.yaml")
if err != nil {
 return err
}
yamlFile, err := ioutil.ReadFile(orgFilePath)
if err != nil {
 return err
}
var org Organization
err = yaml.Unmarshal(yamlFile, &org)
if err != nil {
 return err
}

for _, team := range org.Teams {
 ghTeam, err := github.NewTeam(ctx, team.Slug, &github.TeamArgs{
 Description: pulumi.String(team.Description),
 Name: pulumi.String(team.Name),
 Privacy: pulumi.String("closed"),
}, pulumi.Protect(false))
if err != nil {
fmt.Println("encountered error creating new Pulumi GitHub team: ", team.Name)
 return err
}
 return nil
 })
}

Notice how the GitHub provider allows us to use team names to create teams. In contrast, team IDs are only needed for importing to the Stack, not for maintaining state.

Relationships Are Hard

The next step involved a lot of thinking about team memberships and team nesting. GitHub allows teams to be nested. This groups teams by area of responsibility but also allows for certain properties, such as permissions, to be inherited by subteams.

Parent Team Parent
 |
 -------------------------
 | |
Child Teams Child1 Child2
(inherit all permissions)

To establish this relationship, we set a ParentTeamId on child teams. First off, we add a new field to our Teams struct:

type Team struct {
 Name string `yaml:"name"`
 Description string `yaml:"description"`
 Slug string `yaml:"slug"`
 ParentTeamId int
 Teams []Team `yaml:"teams"`
}

Here is where things get a little tricky. Any GitHub Team can have subteams, but not every Team has a parent Team. We also did not want to have to hardcode GitHub Team IDs into our configuration files. We needed to meet the following requirements:

Create a parent team, using a team name.
Obtain its team ID.
Create any child teams, also using a team name.
Write the parent team’s ID into the ParentTeamId field of each child team.
Do all of the above in a single pulumi up.

This is where maintaining infrastructure with Pulumi truly shines. In our code, we can use Pulumi Apply to hold on to the promise of a parent team ID, and pass this promise into the appropriate field:

func setupTeams(ctx *pulumi.Context, parentTeam *Team) error {

// set up parent team, i.e. engineering
 ghParentTeam, err := github.NewTeam(ctx, parentTeam.Slug, &github.TeamArgs{
 Description: pulumi.String(parentTeam.Description),
 Name: pulumi.String(parentTeam.Name),
 Privacy: pulumi.String("closed"),
 }, pulumi.Protect(false))
 if err != nil {
 fmt.Println("encountered error creating new Pulumi github team: ", parentTeam.Name)
 return err
 }

//set up nested teams
 for _, childTeam := range parentTeam.Teams {
 // set each child team's parent team ID to the current team ID
 ghChildTeam, err := github.NewTeam(ctx, childTeam.Slug, &github.TeamArgs{
 Description: pulumi.String(childTeam.Description),
 Name: pulumi.String(childTeam.Name),
 Privacy: pulumi.String("closed"),
 ParentTeamId: ghParentTeam.ID(),
 }, pulumi.Protect(false))
 if err != nil {
 fmt.Println("encountered error creating new Pulumi github team: ", childTeam.Name)
 return err
 }
 }
 return nil
}

Running this as part of main.go will result in beautifully nested teams on the GitHub UI. But with Pulumi, we can do even better. We can set pulumi.Parent() on the child teams:

for _, childTeam := range parentTeam.Teams {
 ghChildTeam, err := github.NewTeam(ctx, childTeam.Slug, &github.TeamArgs{#...},
 pulumi.Protect(false),
 pulumi.Parent(ghParentTeam))
 #...
}

With this, we can see the relationship between the resources on pulumi up:

Updating (prod)

 Type Name Status
 pulumi:pulumi:Stack team-mgmt-prod
 + └─ github:index:Team newteam created
 + └─ github:index:Team newchildteam created
Resources:
 + 2 created
2 unchanged

Which will be reflected on the Pulumi Stack’s Graph View in the Pulumi Service:

Adding this relationship to Pulumi is mostly a visual nicety in our case; however, certain Pulumi properties get inherited in a helpful way. Note that on GitHub, it is possible to create multiple layers of teams, whereas the example code in this blog post only shows a single layer of subteam structure. We are small enough at Pulumi to not need more, but you may want to expand on the solution if you do.

Adding People

Next, teams should have members! Let’s add them to the YAML config:

org: "pulumi-demo-org"
teams:
 - name: "animals"
 slug: "animals"
 description: "A team in charge of animals"
 members:
 - username: "platypus"
 - username: "owlcat"

A TeamMembership in GitHub is a cross reference between a Team and a User. Fortunately these do not need to be explicitly imported, as they are merely establishing relationships between GitHub Users and Teams. We can add TeamMemberships to the Stack with a Members struct, a new Members[] field on the Team struct, and an extra function. Again, Pulumi lets us use the promised output of the team ID to set the TeamId field in the TeamMembership:

type Team struct {
 Name string `yaml:"name"`
 Description string `yaml:"description"`
 ParentTeamId int
 Teams []Team `yaml:"teams"`
 Slug string `yaml:"slug"`
 Members []Member `yaml:"members"`
}

type Member struct {
 UserName string `yaml:"username"`
 Role string `yaml:"role"`
}

#...

func addMembers(ctx *pulumi.Context, members []Member, team *github.Team, teamName string) error {
 for _, member := range members {
 // unique name for TeamMembership
 utmName := teamName + "-" + member.UserName
 _, err := github.NewTeamMembership(ctx, utmName, &github.TeamMembershipArgs{
 TeamId: team.ID(),
 Username: pulumi.String(member.UserName),
 })
 if err != nil {
 return err
 }
 }
 return nil
}

Running pulumi up:

Updating (prod)

 Type Name Status
 pulumi:pulumi:Stack team-mgmt-prod
 + ├─ github:index:TeamMembership animals-owlcat created
 + └─ github:index:TeamMembership animals-platypus created

Resources:
 + 2 created
 4 unchanged

Alright! So now we have

Imported existing teams to a Pulumi Stack to track and organize all GitHub teams from a central repository, and
Added team members to teams.

That’s pretty great so far! While there are many org chart tools, what makes this one useful to us is that we can declare the desired org state in a config file, and then let Pulumi figure out what changes should be applied. Let’s automate that!

Add CI

In order for this tool to be used by everyone, we keep code and configuration in a GitHub repository. We can use Pulumi’s GitHub Action to run a pulumi preview on a pull request, and a pulumi up on merge to main.

Here’s what that looks like on the pull request:

name: Pull Request preview
on:
 pull_request:
 workflow_dispatch:

env:
 AWS_REGION: us-west-2
 GITHUB_TOKEN: ${{ secrets.PULUMI_GITHUB_TOKEN }}
 PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
 GITHUB_OWNER: pulumi

jobs:
 pull-request:
 runs-on: ubuntu-latest
 name: Preview
 steps:
 - uses: actions/checkout@v2
 - name: Install Go
 uses: actions/setup-go@v2
 with:
 go-version: 1.17
 - name: Build Binary
 run: |
 go mod tidy
 go build main.go
 - name: Pulumi Preview
 uses: pulumi/actions@v3
 with:
 command: preview
 refresh: true
 stack-name: pulumi/prod
 comment-on-pr: true

And the pulumi up on merge to main:

name: Update
on:
 push:
 branches:
 - "main"
 workflow_dispatch:

env:
 AWS_REGION: us-west-2
 GITHUB_TOKEN: ${{ secrets.PULUMI_GITHUB_TOKEN }}
 PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
 GITHUB_OWNER: pulumi

jobs:
 main:
 runs-on: ubuntu-latest
 name: Update
 steps:
 - uses: actions/checkout@v2
 - name: Install Go
 uses: actions/setup-go@v2
 with:
 go-version: 1.17
 - name: Build Binary
 run: |
 go mod tidy
 go build main.go
 - name: Pulumi update
 uses: pulumi/actions@v3
 with:
 command: up
 refresh: true
 stack-name: pulumi/prod

Note that we are calling refresh: true in both Workflows, which uses Pulumi Refresh to make sure that the existing GitHub resources are aligned with the resource state in our Stack.

Now, anyone with access to the GitHub management repo can:

Create, re-parent, delete, or rename teams and re-create any memberships via pull request to the config file.
Add and remove team members via pull request.
Audit and explicitly maintain org structure via git history and review processes.

But of course… there’s more!

Managing Permissions

The next step involved a lot of thinking about repository permissions and permission access.

Shortly after I joined Pulumi, my team was combined with another team. They brought along all of their repositories—none of which I had access to. Moreover, all of our team names were outdated. I could rename the teams using pulumi-github… but I could not transfer all of the repository access we all needed to our new team. Yet.

On GitHub, parent teams pass permissions down to child teams. But what if we wanted permissions to be more granular?

Here’s where TeamRepositories come in. A TeamRepository has three fields: a team ID, a repository name, and a permission. Just like TeamMemberships, this resource is a link between a Team and a Repository and does not need to be imported.

We gave each team their own config file with permission levels listing repository names:

team: "animals"
permissions:
 - role: "admin"
 repos:
 - "capybaras"
 - "owls"
 - role: "maintain"
 repos:
 - role: "push"
 repos:
 - role: "read"
 repos:

This can be read into our Pulumi program as follows:

// TeamPermissions describes a github team and the levels of permissions available (i.e. admin, maintain)
type TeamPermissions struct {
 TeamName string `yaml:"team-name"`
 Permissions []Permission `yaml:"permissions"`
}

// Permission is a single permission and a list of repos with that permission.
type Permission struct {
 Role string `yaml:"role"`
 Repos []string `yaml:"repos"`
}

#...

func setupTeamRepos(ctx *pulumi.Context, team *github.Team, teamName string) error {

 repoFilePath, err := filepath.Abs("./team-repos/" + teamName + ".yaml")
 if err != nil {
 fmt.Println(err.Error())
 return err
 }
 yamlFile, err := ioutil.ReadFile(repoFilePath)
 if err != nil {
 return err
 }

 var teamPerms TeamPermissions
 err = yaml.Unmarshal(yamlFile, &teamPerms)
 if err != nil {
 return err
 }
 for _, permission := range teamPerms.Permissions {
 for _, repo := range permission.Repos {
 _, err := github.NewTeamRepository(ctx, fmt.Sprintf("%s-%s-%s", repo, teamName, permission.Role), &github.TeamRepositoryArgs{
 Permission: pulumi.String(permission.Role),
 Repository: pulumi.String(repo),
 TeamId: team.ID(),
 })

 if err != nil {
 return err
 }
 }
 }
 return nil
}

And the output of running pulumi up:

Updating (prod)

 Type Name Status
 pulumi:pulumi:Stack team-mgmt-prod
 + ├─ github:index:TeamRepository owls-animals-admin created
 + ├─ github:index:TeamRepository capybaras-animals-admin created


Resources:
 + 2 created
 2 changes. 4 unchanged

With this addition, we can now combine teams with zero access disruptions as follows:

Create the new team in the org config.
Create a TeamRepository config for the new team with desired permission levels.
Open a pull request and let CI and Pulumi do the rest.

Now we can develop standards around repository permissions based on teams’ roles and areas of responsibility, and we have tooling in place that can maintain these standards for everyone in the org to see. In fact, we can standardize the meaning of “code ownership” via access levels in this way—one centralized management location rather than asking your grandboss to dig through the UI for you.

Future challenges

We want to automate some of these steps: adding new teammates to the org should also automatically add them to this repository so they can self-serve their onboarding and see Pulumi work in action!

To prevent state drift, we will run a regular reconciliation job in CI.

Of course, we will add other Pulumi providers on top of this org structure so we can sync our other tooling.

Easier IaC adoption with improved `pulumi import` experience

Fraser Waters — Tue, 15 Mar 2022 00:00:00 +0000

Last year, we introduced a new Pulumi feature that allows you to import existing infrastructure into your Pulumi program. Not only did it bring the resource into the Pulumi state file, but it could generate the source code for your Pulumi program too. Today, we’re excited to announce that we’ve listened to feedback and delivered a plethora of updates and fixes to streamline the import experience; to make it more useful, more convenient, and more powerful.

At Pulumi, we understand that many cloud engineers and platform teams around the world don’t have the luxury of greenfield projects, more often than not we’re stuck with the impossible task of “refactoring” or “migrating” existing projects to more modern stacks to help increase team productivity, velocity, and stability. These projects aren’t trivial and we want to make it easier for teams and organizations to bring their infrastructure into a cloud engineering world. Oh, and worry not you lucky greenfielders … even if you just wanna ClickOps your way through some resources and import them into your program; that’s gonna work just fine too; we won’t tell if you don’t 😉.

To help you understand the changes we’ve made to help you on your journey, let’s take a look at some side-by-side examples of how pulumi import used to work vs. what we’re releasing today in Pulumi 3.26.

Example: AWS S3 Bucket

Previously when importing an S3 bucket, we would set the default values for properties like acl and forceDestroy. The new behavior will not include properties with default values in the generated code, keeping your code inline with how you’d write it yourself. Artisanal codegen for the artisanal cloud engineer.

Code Comparison

This code would have been generated by the older implementation:

const my_bucket = new aws.s3.Bucket("my-bucket", {
 acl: "private",
 bucket: "my-bucket-3f85c54",
 forceDestroy: false,
}, {
 protect: true,
});

This code will be generated by the newer implementation:

const my_bucket = new aws.s3.Bucket("my-bucket", {
 arn: "arn:aws:s3:::my-bucket-3f85c54",
 bucket: "my-bucket-3f85c54",
 hostedZoneId: "Z3BJ6K6RIION7M",
 requestPayer: "BucketOwner",
}, {
 protect: true,
});

The above accurately reflects what the providers Read function has told us of the inputs set for this bucket, although due to the way the AWS provider works, we could elide all these properties. This is because the provider will use outputs saved in the state file and the new inputs to calculate the update diff. A property that exists in the old output and is missing from the new inputs is simply left at the old output value. As none of the properties for Bucket are marked required the following works as well and is equivalent:

const my_bucket = new aws.s3.Bucket("my-bucket", { protect: true });

Example: AWS EC2 Instances

Previously, when trying to import an EC2 instance with pulumi import, you’d be presented with a list of required properties that weren’t satisfied. Like above, the new behavior will read from the provider and inherit the missing properties, providing everything required to correctly import. This provides a cleaner and more intuitive experience for the user.

Old Behavior

$ pulumi import aws:ec2/instance:Instance test i-085d780737c600c7e
aws:ec2:Instance (test):
 error: aws:ec2/instance:Instance resource 'test' has a problem: Missing required argument: "instance_type": one of `instance_type,launch_template` must be specified. Examine values at 'Instance.InstanceType'.
 error: aws:ec2/instance:Instance resource 'test' has a problem: Missing required argument: "launch_template": one of `ami,instance_type,launch_template` must be specified. Examine values at 'Instance.LaunchTemplate'.
 error: aws:ec2/instance:Instance resource 'test' has a problem: Missing required argument: "ami": one of `ami,launch_template` must be specified. Examine values at 'Instance.Ami'.

New Behavior

The import will be successful and we’ll be presented with a rich resource in your language of choice.

const test = new aws.ec2.Instance("test", {
 ami: "ami-082b5a644766e0e6f",
 associatePublicIpAddress: true,
 availabilityZone: "us-west-2c",
 capacityReservationSpecification: {
 capacityReservationPreference: "open",
 },
 cpuCoreCount: 1,
 cpuThreadsPerCore: 1,
 creditSpecification: {
 cpuCredits: "standard",
 },
 iamInstanceProfile: "",
 instanceInitiatedShutdownBehavior: "stop",
 instanceType: "t2.micro",
 metadataOptions: {
 httpEndpoint: "enabled",
 httpPutResponseHopLimit: 1,
 httpTokens: "optional",
 },
 privateIp: "172.31.0.188",
 rootBlockDevice: {
 iops: 100,
 volumeSize: 8,
 volumeType: "gp2",
 },
 securityGroups: ["default"],
 subnetId: "subnet-43f43a1e",
 tenancy: "default",
 vpcSecurityGroupIds: ["sg-4d436f12"],
}, {
 protect: true,
});

test = aws.ec2.Instance("test",
 ami="ami-082b5a644766e0e6f",
 associate_public_ip_address=True,
 availability_zone="us-west-2c",
 capacity_reservation_specification=aws.ec2.InstanceCapacityReservationSpecificationArgs(
 capacity_reservation_preference="open",
 ),
 cpu_core_count=1,
 cpu_threads_per_core=1,
 credit_specification=aws.ec2.InstanceCreditSpecificationArgs(
 cpu_credits="standard",
 ),
 iam_instance_profile="",
 instance_initiated_shutdown_behavior="stop",
 instance_type="t2.micro",
 metadata_options=aws.ec2.InstanceMetadataOptionsArgs(
 http_endpoint="enabled",
 http_put_response_hop_limit=1,
 http_tokens="optional",
 ),
 private_ip="172.31.0.188",
 root_block_device=aws.ec2.InstanceRootBlockDeviceArgs(
 iops=100,
 volume_size=8,
 volume_type="gp2",
 ),
 security_groups=["default"],
 subnet_id="subnet-43f43a1e",
 tenancy="default",
 vpc_security_group_ids=["sg-4d436f12"],
 opts=pulumi.ResourceOptions(protect=True))

 _, err := ec2.NewInstance(ctx, "test", &ec2.InstanceArgs{
 Ami: pulumi.String("ami-082b5a644766e0e6f"),
 AssociatePublicIpAddress: pulumi.Bool(true),
 AvailabilityZone: pulumi.String("us-west-2c"),
 CapacityReservationSpecification: &ec2.InstanceCapacityReservationSpecificationArgs{
 CapacityReservationPreference: pulumi.String("open"),
 },
 CpuCoreCount: pulumi.Int(1),
 CpuThreadsPerCore: pulumi.Int(1),
 CreditSpecification: &ec2.InstanceCreditSpecificationArgs{
 CpuCredits: pulumi.String("standard"),
 },
 IamInstanceProfile: pulumi.Any(""),
 InstanceInitiatedShutdownBehavior: pulumi.String("stop"),
 InstanceType: pulumi.String("t2.micro"),
 MetadataOptions: &ec2.InstanceMetadataOptionsArgs{
 HttpEndpoint: pulumi.String("enabled"),
 HttpPutResponseHopLimit: pulumi.Int(1),
 HttpTokens: pulumi.String("optional"),
 },
 PrivateIp: pulumi.String("172.31.0.188"),
 RootBlockDevice: &ec2.InstanceRootBlockDeviceArgs{
 Iops: pulumi.Int(100),
 VolumeSize: pulumi.Int(8),
 VolumeType: pulumi.String("gp2"),
 },
 SecurityGroups: pulumi.StringArray{
 pulumi.String("default"),
 },
 SubnetId: pulumi.String("subnet-43f43a1e"),
 Tenancy: pulumi.String("default"),
 VpcSecurityGroupIds: pulumi.StringArray{
 pulumi.String("sg-4d436f12"),
 },
}, pulumi.Protect(true))

var test = new Aws.Ec2.Instance("test", new Aws.Ec2.InstanceArgs
 {
 Ami = "ami-082b5a644766e0e6f",
 AssociatePublicIpAddress = true,
 AvailabilityZone = "us-west-2c",
 CapacityReservationSpecification = new Aws.Ec2.Inputs.InstanceCapacityReservationSpecificationArgs
 {
 CapacityReservationPreference = "open",
 },
 CpuCoreCount = 1,
 CpuThreadsPerCore = 1,
 CreditSpecification = new Aws.Ec2.Inputs.InstanceCreditSpecificationArgs
 {
 CpuCredits = "standard",
 },
 IamInstanceProfile = "",
 InstanceInitiatedShutdownBehavior = "stop",
 InstanceType = "t2.micro",
 MetadataOptions = new Aws.Ec2.Inputs.InstanceMetadataOptionsArgs
 {
 HttpEndpoint = "enabled",
 HttpPutResponseHopLimit = 1,
 HttpTokens = "optional",
 },
 PrivateIp = "172.31.0.188",
 RootBlockDevice = new Aws.Ec2.Inputs.InstanceRootBlockDeviceArgs
 {
 Iops = 100,
 VolumeSize = 8,
 VolumeType = "gp2",
 },
 SecurityGroups =
 {
 "default",
 },
 SubnetId = "subnet-43f43a1e",
 Tenancy = "default",
 VpcSecurityGroupIds =
 {
 "sg-4d436f12",
 },
 }, new CustomResourceOptions
 {
 Protect = true,
 });
 ```

Again this reflects what Read has told us of the inputs set for this instance, and again due to the way the AWS provider works many of these inputs could be elided and the provider would pick up the values from the saved output set. The following is again equivalent to the above given the saved state:

const test = new aws.ec2.Instance("test", {
 ami: "ami-082b5a644766e0e6f",
 instanceType: "t2.micro"
}, {
 protect: true,
});

Everything is Better, Yes?

To quote our old friend, Harlan, “Everything is better, yes?”

While we’ve only shown you a couple of examples of this new behaviour, rest assured that the benefits of it stretch far and wide across the majority of Pulumi resources.

If you’ve ever had a problem importing a resource before, we encourage you to try it again and let us know if you run into any problems.

Curious how this all works? Let’s dive in.

Technical details

We’ve never really explained how the import system works before, which has lead to a lot of confusion from users when they encountered errors with it. Given the changes we’re making here we felt it a good time to give a more detailed write up of how the engine works to try and help users understand why their import results are the way they are.

This section requires some understanding of the Pulumi architecture, but that understanding is not necessary to be able to use pulumi import.

The importance of `Provider.Read`

Our import system depends on the ability for a resource provider to be able to read the existing state of a resource and report back to Pulumi the current value of it’s inputs and outputs.

This isn’t always possible to return accurately. For example if a resource’s inputs don’t match up 1-to-1 with it’s output state and the underlying provider can only read the current output state there’s no way for it to always construct the correct input state. There could also be bugs in the provider’s Read method that result in inaccurate reads. We’ve designed the import system to be tolerant to these cases, but it does mean we don’t expect perfect import results for every resource.

The current import system

To import resources with the currently the engine issues the following steps:

Use the providers Read function to get the inputs and outputs for the resource.
Remove all properties from the inputs except for those flagged as required.
Pass that reduced input property set to the providers Check method.
If there are any check failures return an error and fail the import.
Take the result of Check which will have filled in default properties and diff it with the original input set returned from Read.
If there are any diffs copy in the values from the original Read to the resulting input set.
Save the updated input set and the original output set into the stacks state.
Pass the resulting input set to the code generator to generate a new Resource call to match the imported state.

There are a number of problems with the above. The biggest is trying to strip the input set down to only properties that are marked as required. The required flag is a fairly blunt tool that can’t capture that some fields are only required sometimes based on the value of other fields. As such this stripping to just required properties causes many of the following calls to Check to raise check failures.

The next major issue is that we hard fail if check returns failures. There is always the risk of this happening due to limitations of Read, and it doesn’t help the user just throwing an error.

Finally in the cases where the above two issues weren’t hit this data flow had the odd result of setting optional fields to their default values. This isn’t needed and these fields would be better left as blank.

The new import system aims to improve the described issues.

The new import system

The engine will now issue the following steps to import resources:

Use the providers Read function to get the inputs and outputs for the resource.
Pass the input set to the providers Check method.
Warn if there are any check failures.
Save the input and output set from Read into the stacks state.
Pass the input set to the code generator to generate a new Resource call to match the imported state.

Note that we no longer end up with filled in default values in our final result, and we carry on in the face of check failures and so can still generate some code. If providers can give complete and accurate results from Read the above flow will result in correctly generated code. However it doesn’t result in quite the correct stack state, this is due to how the data flow of our import system currently works. Currently we generate code based on the value of inputs in the state file, however normally the value stored at inputs in the state file is the result from Check, not what is directly in the users program. As such when you run the first pulumi up after an import your state file will change slightly as the engine calls Check and saves the inputs with defaults filled in. This should normally be transparent to the user, but there may be some cases where providers think this is a trivial diff.

There are two complications with fixing the above. The first is that during import if Read doesn’t return valid results then Check can fail and so we won’t have a property set returned from it that we can save to the state file. This case will always have to fall back to just storing the properties returned by Read and triggering a check failure during up that will need to be manually resolved. The second complication is that the code generation for import works off the state file rather than a separate data flow specifically for imported resources. We plan on improving this but in the spirit of agility we felt that a slightly better import feature today would be more useful to our users than a perfect import feature later.

Terraform `Read`

Many of Pulumi’s resource providers make use of Terraform providers. We call these bridged providers and all of them use our “terraform-bridge” library to translate between the Terraform API and Pulumi API.

Terraform currently has a slightly odd behavior that missing values for some properties are zero initialized by its Refresh method (what we use to implement Pulumi’s Read method in the terraform bridge). This has the result that when Pulumi asks to read a given Terraform resource some top level properties (such as S3 buckets acl property) come back as missing, while other top level properties and all nested properties (such as S3 buckets versioning.mfaDelete) come back with some zero value.

For example reading an AWS S3 bucket object with the current Terraform bridge results in the following property set:

{
"__defaults": [],
"accelerationStatus": "",
"arn": "arn:aws:s3:::my-bucket-3f85c54",
"bucket": "my-bucket-3f85c54",
"corsRules": [],
"grants": [],
"hostedZoneId": "Z3BJ6K6RIION7M",
"lifecycleRules": [],
"loggings": [],
"objectLockConfiguration": null,
"replicationConfiguration": null,
"requestPayer": "BucketOwner",
"serverSideEncryptionConfiguration": null,
"tags": {
"__defaults": []
},
"versioning": {
"__defaults": [],
"enabled": false,
"mfaDelete": false
},
"website": null
}

We’ve made a change to the Terraform bridge to try and elide these zero properties that should just be missing. Reading the same S3 bucket with these changes gives the following much smaller property set:

{
"__defaults": [],
"arn": "arn:aws:s3:::my-bucket-3f85c54",
"bucket": "my-bucket-3f85c54",
"hostedZoneId": "Z3BJ6K6RIION7M",
"requestPayer": "BucketOwner",
"website": null
}

We’ll continue to work on the Terraform bridge to give better results here. For example the website property on the bucket should probably be missing rather than null.

Feedback

We’re aware this could have a large impact to some of our users so we wanted to make sure we explained these changes well. If you have questions or concerns about this change let us know in our community Slack or in GitHub Discussions.

Migrating My Infrastructure From Terraform to Pulumi

Erik Näslund — Tue, 22 Feb 2022 12:36:23 -0600

Pulumi community member Erik Näslund shares his thoughts on how to migrate from Terraform to Pulumi. Read on to learn all the details of his experience!

Since this post was published, Pulumi has added first-class support for HashiCorp Configuration Language (HCL). You can now write Pulumi programs in HCL directly, alongside general-purpose languages like TypeScript, Python, Go, and C#. To see how it works, see Pulumi HCL.

I’ve been using Terraform for a couple of years and overall I’ve been quite happy with it. However there’s a few things that started to bother me more and more recently.

Terraform uses a language called Hashicorp HCL to define the infrastructure. It’s a relatively simple declarative language, but it’s something I had to learn along the way. Just like any language it has it’s little quirks, and I often found myself spending more time than I wanted to figure out how to do certain things. As I’m doing all the infrastructure myself I really wanted to be able to use a language I’m familiar with, to make things simple.

I had written my own Terraform modules to organize my code. Modules are a great help when it comes to organizing your code in a logical way, and I strongly recommend it if you haven’t tried using them already. However, after using it for a while I realized that the Terraform type system left something to be desired. There are a few basic types available, but not much more than that. This caused bad “micro interactions” between me and Terraform. It was commonplace for me to submit an MR to my Gitlab CI where I passed in a security group name instead of an arn or id.

The third and perhaps most important point is that I think a custom language like HCL is the wrong way to go long term. Don’t get me wrong - I love the people at Hashicorp and I’m forever grateful for Terraform because it’s the first product that made me enjoy writing IaC. That being said, I think it’s a better choice to write libraries for existing languages, instead of trying to create a configuration language of your own. There’s a lot of constructs that I’m used to having in Python that I sorely miss having available in Terraform. In my opinion there’s no reason to develop your own configuration language since it’s a time consuming task, and I don’t find it better than a general purpose language anyways.

Why did I choose Pulumi?

I have this thing called “Fun time Friday” where I spend the first half day on Fridays exploring new things I’ve heard about. Most of the things I look at during that time end up being scrapped, and there’s zero pressure to push things into production.

However after exploring Pulumi I felt that it might be significantly better for me than Terraform. I decided to take a day to see if I could migrate my infrastructure.

Could I have used something else, like AWS CDK? I most probably could. The nice thing about running a company of one is that there’s no need for analysis paralysis! At this point I believed that Pulumi was a better choice for me than Terraform, so I went for it.

Personally I struggle a bit with CDK being mostly Amazon-centric. I sometimes use my IaC to configure things outside of AWS as well. The other issue for me is that CDK by default uses AWS CloudFormation to provision resources. I’ve been bitten a few times by CF. Some times it’s been because of buggy behavior, e.g. a stack that refused to terminate. Other times it’s been the fact that CF often lags behind in capabilities, so it’s common to have to wait a few months for a new feature to be usable through CF.

Sure, I could use CDK for Terraform to have CDK generate Terraform HCL for me. However, now we’re quickly getting away from the simple system I wanted in the first place.

Apologies to those of you who are screaming “FFS - CDK is the best thing ever!" at your monitors right now. You might be 100% right! From what I’ve seen CDK seems pretty awesome for the most part. I didn’t end up with Pulumi because CDK seemed bad. I ended up with Pulumi because it seemed like a slightly better fit for my needs :).

Pulumi in 3 minutes

You might already know a bit of Pulumi, but here’s a very quick overview plus some advice on where to go from here.

Pulumi allows you to write your IaC using TypeScript, Javascript, Python, Go or C#.
You define resources which have inputs and outputs. As an example, a resource could be an RDS database Instance. One of it’s inputs would be the database version to use. One of the outputs would be the hostname / address of the server.
The outputs are lazily evaluated, and their exact value is often not known until after you’ve executed your code. I.e. you wouldn’t have the db_instance.address until the instance had been created. Think of outputs as promises / futures.
You can use component resources to group your resources into logical groups - similar to how Terraform modules work. If you don’t do this there’s a high risk you’ll end up with a very messy setup further down the line.
Read Pulumi’s own overview and naming docs before attempting to use Pulumi yourself. I’d say that’s the most important bit to know about.
Pulumi are using the open core model where most features are free, but some extra goodies are on a paid tier. You can use the open source version and do state management yourself, but you miss out on certain features.

So what does Pulumi code look like? Below is a minimal extract of some of the Python code I’ve written (slightly altered).

# __main__.py
...
prod_redis_server = RedisServer(
 identifier='redis-prod',
 route53_zone=example_com_route53_zone,
 route53_name='redis-prod.example.com',
 security_groups=[my_ecs_service.task_security_group]
)
...

# components/redis_server.py
class RedisServer(ComponentResource):
 def __init__(
 self,
 identifier: str,
 route53_zone: aws.route53.Zone, # Route53 Zome where the DNS record will be created.
 route53_name: str, # Full DNS name of the CNAME record, e.g. 'redis.example.com'.
 security_groups: List[aws.ec2.SecurityGroup], # Security groups which should be granted access to the server.
 opts=None
 ):
 super().__init__('vt:RedisServer', identifier, None, opts)

 # Prefix with `self.` if you want to access the security group from outside this component.
 security_group = aws.ec2.SecurityGroup(
 identifier,
 ingress=[
 aws.ec2.SecurityGroupIngressArgs(
 from_port=6379,
 to_port=6379,
 protocol='tcp',
 security_groups=[sg.id for sg in security_groups],
 )
 ],
 opts=ResourceOptions(parent=self),
 )

 # Create all the other sub resources needed.
 ...

As you can see it’s really simple and clean for me to set up a Redis server now.

There’s also the fact that I can use the Pulumi Console (web interface) to check the state of my infrastructure really quickly. I found myself doing this instead of logging in to the AWS console or using the AWS cli.

How I went about the migration

After reading the migration guide I immediately tried out what seemed to be the obvious option - tf2pulumi. This is supposed to be able to convert Terraform code directly to Pulumi. In my case it simply errored out and was not able to convert my Terraform code. To be fair my TF code was using slightly more “advanced” features like custom written modules, so it wasn’t the easiest thing to handle for tf2pulumi. I recommend you try it out, because it sounds great in theory. However it wasn’t an option for me, so I had to go for a more manual approach. In retrospect I’m actually quite happy I did, because it gave me the opportunity to organise my code in a way that I preferred.

As of Pulumi CLI v3.71.0, tf2pulumi has been replaced with pulumi convert --from terraform. Converting Full Terraform Programs to Pulumi blog has more details.

Don’t worry though - “manual” imports using Pulumi is still rather easy to perform, albeit a bit time consuming. Here’s how I did it.

I started by testing out the AWS getting started guide to get a feel for how Pulumi worked. This was a great move and I recommend everyone else to do the same.

I was then ready to start importing my existing resources. The general workflow was to look at my existing Terraform code, rewrite it using Pulumi, and then import the existing resources. It was a bit time consuming in the beginning, but I got a lot faster at it after a few hours.

Since Pulumi starts out “empty” you can rewrite and import a piece at a time. I’d recommend doing this because you will most probably do some mistakes initially, and it’s good to get early feedback.

You can check the API reference for details on how to import each specific resource. The instructions on how to do it is usually at the bottom of the reference page for each resource type.

Once I had everything migrated to Pulumi I set up a Gitlab CI/CD flow which allowed me to preview and apply the infrastructure changes. Up until then I’d been doing it locally from my own computer. If possible, I recommend doing it that way. The extra time taken by involving a separate CI/CD pipeline would make your migration take a lot longer. It’s perfectly fine when everything is done and dusted, however when learning Pulumi you’ll often need to use the pulumi command to look at the resources.

Issues, workarounds and general tips

Use component resources to group resources

When your infrastructure starts growing it becomes hard to manage unless you group your resources. Pulumi’s solution to this are the so called component resources. They are “logical components” which you define yourself.

$ pulumi stack
...
─ vt:RedisServer vt-redis-prod
 │ ├─ aws:elasticache/parameterGroup:ParameterGroup vt-redis-prod
 │ ├─ aws:ec2/securityGroup:SecurityGroup vt-redis-prod
 │ └─ aws:elasticache/cluster:Cluster vt-redis-prod
...

As you can see I’ve defined a component called “vt:RedisServer” which I use to group all the resources associated with my Redis server. You’ll get a graph representation of your whole infrastructure in the Pulumi Web Console .

It might seem like a small thing, but it really helps to have a logical grouping of your resources as your infrastructure grows. It also help because you can suddenly decide which resources your component exposes and which ones are “internal” to the component. Check out the code sample from earlier in the article to see how to do this using self..

Importing a resource that has a parent was a bit confusing at first

Each resource in pulumi has a globally unique URN. You can show the URNs for the infrastructure you’ve codeified in pulumi by issuing pulumi stack --show-urns

$ pulumi stack --show-urns
...
├─ vt:RedisServer vt-redis-prod
 │ │ URN: urn:pulumi:prod::VT::vt:RedisServer::vt-redis-prod
 │ ├─ aws:elasticache/parameterGroup:ParameterGroup vt-redis-prod
 │ │ URN: urn:pulumi:prod::VT::vt:RedisServer$aws:elasticache/parameterGroup:ParameterGroup::vt-redis-prod
 │ ├─ aws:ec2/securityGroup:SecurityGroup vt-redis-prod
 │ │ URN: urn:pulumi:prod::VT::vt:RedisServer$aws:ec2/securityGroup:SecurityGroup::vt-redis-prod
 │ └─ aws:elasticache/cluster:Cluster vt-redis-prod
 │ URN: urn:pulumi:prod::VT::vt:RedisServer$aws:elasticache/cluster:Cluster::vt-redis-prod
...

Let’s have a look at one of the URNs in more detail, shall we?

urn:pulumi:prod::VT::vt:RedisServer$aws:elasticache/parameterGroup:ParameterGroup::vt-redis-prod

stack name: prod
project name: VT
parent resource type: vt:RedisServer
resource type: aws:elasticache/parameterGroup:ParameterGroup
resource name: vt-redis-prod

All URNs need to be unique, which means every resource of the same type, with the same parent (in the same stack and project) needs to have a unique resource name.

Normally I would have written and imported my parameter group definition something like this:

# components/redis_server.py
class RedisServer(ComponentResource):
 """
 Redis server running on ElastiCache (single node).
 """
 def __init__(self, identifier: str, opts=None):
 super().__init__('vt:RedisServer', identifier, None, opts)

 ...

 parameter_group = aws.elasticache.ParameterGroup(
 identifier,
 name=identifier,
 family='redis6.x',
 opts=ResourceOptions(parent=self),
 )

# __main__.py
vt_prod_redis_server = RedisServer(
 identifier='vt-redis-prod'
)

And I would then try to import it like this:

# Executing the import (THIS WON'T WORK, DON'T TRY IT)
pulumi import\
 aws:elasticache/parameterGroup:ParameterGroup\
 vt-redis-prod\
 vt-redis-prod

The first vt-redis-prod in the command above is the identifier used by the aws.elasticache.ParameterGroup. This is actually called “resource name” using Pulumi-speak. I prefer to call it identifier in my own code though, because “resource name” is too similar to the actual name it has to AWS for my liking.

The second vt-redis-prod is the actual name of my parameter group on AWS (remember we’re importing existing stuff).

However, the above won’t work for one simple reason - the URN won’t match because we haven’t told Pulumi that this parameter group resource we’re importing is a child of our RedisServer component. What’s the solution? Pretty simple, but somewhat obscure at the same time:

# Executing the import (the right way)
pulumi import\
 --parent vt_prod_redis_server=urn:pulumi:prod::VT::vt:RedisServer::vt-redis-prod\
 aws:elasticache/parameterGroup:ParameterGroup\
 vt-redis-prod\
 vt-redis-prod

You tell Pulumi about the parent by using the --parent argument (this is the simple part). The parent argument is a key=value pair, and the key is actually the python variable name you’ve used in your code. If you scroll back up you can see that I stored my RedisServer instance in a variable called vt_prod_redis_server. This part took longer than I’d like to admit for me to figure out.

The value is the URN of the parent component, which you can get with pulumi stack --show-urns.

You can’t import a child without having the parent in place first

That last one was a bit heavy, let’s take a breather and go for something a bit simpler!

When importing a child component you need to import the parent first. If you simply run pulumi up it will try to import all your resources all at once, and it won’t work because the parent haven’t been created yet.

Instead simply do this first to update just a single resource:

$ pulumi up --target urn:pulumi:prod::VT::vt:RedisServer::vt-redis-prod

This will cause my vt-redis-prod resource to be created in the Pulumi state. Nothing actually happens on AWS since it’s just a logical component. Once this is done you’ll be able to import all of it’s child resources as you saw above.

Writing IAM (JSON) policies with Pulumi outputs

Imagine having an IAM policy that referenced an ECR repository, just as an example. You might want the policy to grant access only to that particular ECR repository. You might be temped to try something like this:

aws.iam.Policy(
 ...
 policy=json.dumps({
 'Version': '2012-10-17',
 'Statement': [
 {
 'Sid': '1',
 'Effect': 'Allow',
 'Action': [
 'ecr:GetAuthorizationToken',
 ...
 ],
 'Resource': ecr_repository.arn
 }
 ]
 }))

While this might look sensible, it won’t quite work. Remember early on I sad that Pulumi outputs are futures, and not variables whose values you can access just like that. If you did like the example above your policy would just end up with something similar to 'Resource': <Pulumi.Output(...)> in it. Not quite what we wanted.

Instead we have to tell Pulumi to resolve these futures, so we get the actual value for them, like this:

aws.iam.Policy(
 ...
 policy=ecr_repository.arn.apply(lambda arn: json.dumps({
 'Version': '2012-10-17',
 'Statement': [
 {
 'Sid': '1',
 'Effect': 'Allow',
 'Action': [
 'ecr:GetAuthorizationToken',
 ...
 ],
 'Resource': arn
 }
 ]
 }))

If you have more than one Output you need to access you can use Output.all, like this:

policy=(
 Output.all(origin_access_identity.iam_arn, self.s3_bucket.arn)
 .apply(
 lambda args: json.dumps({
 'Version': '2008-10-17',
 'Id': 'PolicyForCloudFrontPrivateContent',
 'Statement': [
 {
 'Sid': '1',
 'Effect': 'Allow',
 'Principal': {
 'AWS': args[0]
 },
 'Action': 's3:GetObject',
 'Resource': f'{args[1]}/*'
 }
 ]
 })
 )
)

The .apply method can also be useful for troubleshooting when you need to access the particular value of an output.

This might seem tricky at first, but digest this for a bit, then have a look at the docs for Inputs and Outputs and I’m sure it’ll all make sense.

Don’t name resources unless you really have to

Pulumi is pretty smart when it comes to naming the actual resources on AWS. If you don’t specify a name when creating a resource Pulumi will use the identifier and append a hex character suffix .

This gives two great advantages. The first is that you can have multiple stacks next to each other, e.g. a “review” and a “production” environment. If you hard coded the names this wouldn’t be possible.

The second advantage is that it makes replacement of resources much easier. If you hard code the name you leave Pulumi no choice but to tear down the existing resource before creating the replacement one. This makes a big difference if you are to perform a version update of one of your dependencies. In case you allow Pulumi to handle the naming it can create the new resource before terminating the previous one. This coupled with a DNS entry update can make for updates with virtually zero downtime.

Since I imported existing resources I had to keep hard coded names in a few places. However I’ve taken the time to get rid of the hard coding as much as possible since then.

Integrating with CI/CD (Gitlab)

I use Gitlab for my Ci/CD, and I’m really happy with it. Let me show you what my very simple Pulumi pipeline looks like right now:

stages:
 - build

preview:
 stage: build
 image:
 name: pulumi/pulumi-python:3.2.1  # Choose the right image for your language
 entrypoint: [""] # This is important for things to work
 script:
 - pulumi stack select prod
 - pulumi preview --color always --diff  # I added '--diff' to always see more details
 only:
 - merge_requests  # Only run the preview on merge requests

update:
 stage: build
 image:
 name: pulumi/pulumi-python:3.2.1
 entrypoint: [""]
 script:
 - pulumi stack select prod
 - pulumi update --color always --diff --yes  # Answer "yes" when asking to apply the changes
 only:
 - master  # Only run the update on the master branch (post merge)

As you can see my pipeline is very simple for this little project which is still in an Alpha phase. There’s just the “production” environment, and changes are automatically applied once the MR is merged.

You’ll have to make use of your gitlab-fu (or other-ci-cd-fu) to adapt this to your own needs. Read the docs from Pulumi to get started. It’s usually very simple and straight forward!

I did things quite differently from what’s described in the Pulumi docs. I used the ready made image from Docker Hub, while they build their own. I did this to make the whole pipeline quicker. It seems to work really well so far, and I haven’t found any downsides to my approach (yet).

I couldn’t get the Gitlab integration webhooks to work

You are normally able to have Pulumi post a report of the planned changes attached to your merge request .

However I couldn’t make this work because of authentication issues. With the help of their support (which was excellent btw) we figured out that it was related to the fact that I was using Gitlab groups to organise my repositories.

As things stand right now you have to create a Pulumi Organization for each Gitlab Group. There’s a bit of an issue with Pulumi’s pricing right now, because the only way to be able to get more than 1 organisation is to go for their enterprise plan.

I think Pulumi will be fixing this in one way or another in the future. For now I’m still pretty happy just checking my changes from the Gitlab job output.

What’s the verdict?

So I’ve been using Pulumi for a few weeks now, and here are my impressions.

It’s really fast compared to Terraform. I didn’t even expect this one, but for some reason Pulumi manages to at least “feel” a lot faster. I haven’t done any measurements to verify this, but it definitely feels much faster.

Their documentation is really good, although it took me a bit to learn to navigate it.

Having type annotations is great! This makes it a lot more friendly to work with as a developer. No more passing around just plain strings in your code. However the type annotations doesn’t quite work in PyCharm because of how the Pulumi code is written. I consider this a PyCharm bug though, and I expect it to be fixed in the future. For now I can simply ctrl+b the class in question, and read the type annotations from the source code.

The Pulumi (Web) Console is a neat way to look at your infrastructure. I know there are similar solutions for Terraform, but I hadn’t used them personally.

The developer ergonomics are great. Using your language of choice helps a lot. When running a company of one, like myself, it’s really important to limit the number of things you need to learn as much as possible. While HCL is simple it’s yet another language to remember. If you’re working at a larger company it’s a great way to encourage developers to do more infrastructure work.

Pulumi is an open core type product, and you will have to pay to access the features on the higher tiers. Their current plans are very fairly priced. I myself is using the free community plan, which works great for me. There’s obviously the risk that they will remove the free plan in the future. However even if that happens I could either pay the fee, or transfer to state storage to AWS S3, and retain most of the product functionality. This feels like an acceptable level of risk for me.

My goal was to do the whole migration in one day. In reality it ended up being closer to two days. However, given that I learned Pulumi and migrated ~130 resources in that amount of time I’d say that Pulumi is quite easy to get started with.

All in all, I’m very happy with making the transition to Pulumi! I became way more productive managing my infrastructure. At the same time I ended up enjoying the work of doing so a lot more - which is really important too.

This article was originally posted at https://blog.ekik.org/my-experience-migrating-my-infrastructure-from-terraform-to-pulumi. Reposted here with permission.

Skai Migrates to AWS with Pulumi

Danny Zalkind — Tue, 04 May 2021 00:00:00 +0000

Danny Zalkind is the DevOps group manager for Skai, an award-winning intelligent marketing platform. He brings his 15 years of experience of managing tech teams to his current role where he’s dedicated to allow Skai R&D to efficiently produce and serve software. You can find him on Linkedin.

Skai is an independent, global marketing platform for strategy, measurement, and best-of-breed activation across all of the world’s most influential digital channels. Skai’s solution provides data-driven insights and optimization technology to help companies make informed decisions and scale performance across critical publishers.

Skai possesses a highly technical engineering organization with over 350 software engineers, data experts, and DevOps engineers.

Why Skai is moving to AWS

The public cloud is not entirely new to us. Skai’s microservices have been running on AWS for a few years now. Our core monolith service is hosted in a data center running our private cloud infrastructure with its hundreds of Terabytes of data.

As our business grew, we noticed that our private cloud infrastructure could not keep pace with our business needs. Adding new capacity to our private cloud could take months of purchasing hardware, preparation, shipping, and deployment.

We decided to adopt infrastructure elastic enough to support the rate of innovation and growth that the business demanded.

We sought out the best infrastructure provider to shift as many services as possible to the cloud. AWS is a clear choice because of its unmatched managed services: RDS for our MySQL databases, Lambda functions for our scheduled processes and various automation tasks, DMS for data migration from the datacenter, Keyspaces for our Cassandra cluster, and EKS for our container workloads.

How Pulumi helped with migration

Once the team selected a cloud platform, it was time to plan and execute the migration. Skai’s DevOps group was selected to lead the project.

Why Pulumi

Skai’s private cloud infrastructure was provisioned and deployed using a combination of Puppet and Fabric. The code was long, full of customizations, not readable, and challenging to troubleshoot.

We were looking for a new solution that would:

Support cloud-native and multi-cloud architectures
Allow adding complex logic to the deployment process
Support reusable code blocks for various use cases
Allow each team to author different parts of the infrastructure
Use a modern programming language
And not create bottlenecks in the release process between the teams

We immediately fell in love with Pulumi, with its programmatic design, extensive modules, and extensibility via dynamic providers.

Project and Stack design

When the Skai team decided to standardize on Pulumi, we designed our architecture to support our use case and development cycle.

We decided to create a project for each AWS organization OUs and a stack for each instance of our application service.

For example, a project would be “platform-production” and contain all stacks and resources of individual environments such as “app1234,” where the number is an environment identifier.

This provided the optimal mix of releasing new code versions per workload type and placing them in the development lifecycle while making granular configuration changes per each instance more manageable.

Programming language

Pulumi supports several programming languages. For us, Python was an obvious choice due to its highly extensible nature, simplicity, and the team’s considerable amount of prior experience with Python. Since most existing automations were written in Python, this made adopting Pulumi very easy for the team.

Multi-team collaboration

Our organization is structured in multiple teams, with each team responsible for different areas of our infrastructure:

The Core team in charge of shared infrastructure such as provisioning AWS accounts
Multiple application teams are responsible for provisioning specific applications, and
An architect who leads designs, builds proof of concepts, and writes code.

It was essential to build our IaC projects with minimal coupling between the teams while keeping each team aligned with the same overarching project goals.

To enable this “loosely-coupled, tightly-aligned” approach, we created component classes for different infrastructure elements, which we later referenced in different places in our end-to-end code.

import pulumi
from pulumi_aws import ec2
from pulumi_aws import ebs

class KSAppServer(pulumi.ComponentResource):

 def __init__(self,
 name: str,
 args: KSAppServerArgs,
 opts: pulumi.ResourceOptions = None):
 super().__init__('kenshoo:KS:KSAppServer', name, None, opts)
 self.name = name
 self.app_instance = ec2.Instance("ks-aws-ue-" + args.ks_id + "-app",
 instance_type=args.instance_size,
 vpc_security_group_ids=args.security_groups_id,
 subnet_id=args.subnet,
 ami=args.ami,
 user_data=args.user_data_content,
 key_name=args.key_pair,
 tags={
 'Name': "ks-aws-ue-" + args.ks_id + "-app"
 },
 opts=pulumi.ResourceOptions(parent=self, delete_before_replace=True))

This practice enabled the application team to work on classes without being entirely dependent on the progress of the Core infrastructure team, which worked on classes of more low-level infrastructure such as VPCs and networking objects.

What about those internal metadata systems?

One more challenge we faced is registering each new instance of our application with internal metadata and IT systems. In our data center setup, we relied on a set of Jenkins jobs, but now, we wanted to integrate these processes with our code and make sure that we always have a consistent state of registration. We decided to use Pulumi’s dynamic providers to achieve this:

from requests import post, get

URL = 'https://krem.ourdomain.com/api/'

class KremMetadataRegistration:
 def create(self, ks_id):
 post(url=f'{URL}/onboarding/init_new_ks',
 params={'ks': f'ks{ks_id}'})

 def delete(self, ks_id):
 response_id = get(url=f'{URL}/config', params={'ks': ks_id}).json()[
 'id']
 if response_id:
 post(url=f'{URL}/onboarding/offboarding',
 params={'id': response_id})

Next, we reference the dynamic provider in another class that we call from the end-to-end code of the whole provisioning flow.

from pulumi.dynamic import *
from .metadata.krem_metadata_registration import KremMetadataRegistration

class RegistrationSchemaInputs(object):
 ks_id: Input[str]
 registered_service: Input[bool]

 def __init__(self, ks, registered_service):
 self.ks = ks
 self.ks_id = ks.lstrip('ks')
 self.registered_service = registered_service

class SchemaProvider(ResourceProvider):
 def __init__(self):
 self.metadatas = {
 'krem': KremMetadataRegistration()
 }

We can streamline the registration and deregistration processes for multiple systems as part of our provisioning lifecycle, creating a more streamlined flow with this approach.

Summary

Moving a high-scale and mature application to the public cloud presented quite a few challenges. We have selected Pulumi as our IaC framework of choice due to its native programmatic nature, an endless number of modules, and the ability to customize the solution using our dynamic providers and custom logic.

This post gave you a glimpse into how we designed and implemented provisioning of our application on AWS. Join us at the second part of the blog that follows our journey of automating data migration.

Pulumi Import: Generate IaC for Existing Cloud Resources

Paul Stack — Tue, 24 Nov 2020 00:00:00 +0000

Most infrastructure projects require working with existing cloud resources, either by building on top of existing resources or adopting existing resources under management with a new and more robust infrastructure provisioning solution.

In June 2019, Pulumi introduced the ability to import existing infrastructure resources to be under Pulumi management no matter how you’ve provisioned these resources — manually in your cloud provider’s console or CLI, using an infrastructure as code tool like Terraform or AWS CloudFormation. Today, we are happy to announce a richer resource import experience.

As of v2.12.0, Pulumi has introduced a pulumi import command. This command will import the cloud resource into the Pulumi state and generate the code for the user’s Pulumi program in the appropriate language.

The Anatomy of the Import Command

No matter how resources have been provisioned in the cloud, Pulumi enables you to adopt and manage those resources as part of your infrastructure project. The pulumi import command is made of the following:

pulumi import [type] [name] [id]

When Pulumi performs an import, the resource is added to the Pulumi state, it is marked as a protected resource (by default), and it will emit the generated code that the user can add to their program before running a pulumi up. The resources are marked as protected to ensure that imported infrastructure is not accidentally deleted if the user forgets to include the code for the resource in their program before doing a deployment.

Importing an S3 Bucket

A resource needs to be imported into an existing Pulumi project. Let’s consider that a user wants to import an S3 bucket manually created in their AWS Account. To do this, we can execute the following command:

pulumi import aws:s3/bucket:Bucket infra-logs company-infra-logs

This command instructs Pulumi to import the S3 bucket called company-infra-logs with the Pulumi resource name infra-logs. The import command will output as follows:

The user can then see the generated code they need to use as part of their application. I imported this resource into a TypeScript application:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

const demo = new aws.s3.Bucket("infra-logs", {
 acl: "private",
 bucket: "company-infra-logs",
 forceDestroy: false,
}, {
 protect: true,
});

import pulumi
import pulumi_aws as aws

demo = aws.s3.Bucket("infra-logs",
 acl="private",
 bucket="company-infra-logs",
 force_destroy=False,
 opts=ResourceOptions(protect=True))

package main

import (
 "github.com/pulumi/pulumi-aws/sdk/v3/go/aws/s3"
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 _, err := s3.NewBucket(ctx, "infra-logs", &s3.BucketArgs{
 Acl: pulumi.String("private"),
 Bucket: pulumi.String("company-infra-logs"),
 ForceDestroy: pulumi.Bool(false),
 }, pulumi.Protect(true))
 if err != nil {
 return err
 }
 return nil
 })
}

using Pulumi;
using Aws = Pulumi.Aws;

class MyStack : Stack
{
 public MyStack()
 {
 var demo = new Aws.S3.Bucket("infra-logs", new Aws.S3.BucketArgs
 {
 Acl = "private",
 Bucket = "company-infra-logs",
 ForceDestroy = false,
 }, new CustomResourceOptions
 {
 Protect = true,
 });
 }

}

This is valid Pulumi application code and can be added to your Pulumi application. Notice that the Pulumi resource is marked as protect: true. This means Pulumi will not delete that resource unless this protection attribute is removed from the resource. If this code is not added to your Pulumi application, then a Pulumi update will return the following:

Diagnostics:
 pulumi:pulumi:Stack (import-example-dev):
 error: preview failed

 aws:s3:Bucket (demo):
 error: Preview failed: refusing to delete protected resource 'urn:pulumi:dev::import-example::aws:s3/bucket:Bucket::infra-logs'

Importing Multiple Resources

Should a user need to import multiple resources to be managed by Pulumi, the CLI import command can handle this scenario for us. This is helpful when using the pulumi import command as part of scripting larger bulk imports of cloud resource:

{
 "resources": [{
 "type": "aws:ec2/vpc:Vpc",
 "name": "application-vpc",
 "id": "vpc-0ad77710973388316"
 },
 {
 "type": "aws:ec2/subnet:Subnet",
 "name": "public-1",
 "id": "subnet-0fb5fdff92b9e5a3b"
 },
 {
 "type": "aws:ec2/subnet:Subnet",
 "name": "private-1",
 "id": "subnet-0a39d25dd9f7b7808"
 }
 ]
}

We can then run the command:

pulumi import -f resources.json

We can see the resultant code generation:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

const application_vpc = new aws.ec2.Vpc("application-vpc", {
 assignGeneratedIpv6CidrBlock: false,
 cidrBlock: "172.16.0.0/16",
 enableDnsSupport: true,
 instanceTenancy: "default",
 tags: {
 Name: "pulumi-vpc",
 Owner: "pulumi",
 Project: "pulumi-k8s-aws-cluster",
 },
}, {
 protect: true,
});
const public_1 = new aws.ec2.Subnet("public-1", {
 assignIpv6AddressOnCreation: false,
 cidrBlock: "172.16.32.0/19",
 mapPublicIpOnLaunch: true,
 tags: {
 Name: "pulumi-vpc-public-1",
 Owner: "pulumi",
 Project: "pulumi-k8s-aws-cluster",
 "kubernetes.io/role/elb": "1",
 type: "public",
 },
 vpcId: "vpc-0ad77710973388316",
}, {
 protect: true,
});
const private_1 = new aws.ec2.Subnet("private-1", {
 assignIpv6AddressOnCreation: false,
 cidrBlock: "172.16.160.0/19",
 mapPublicIpOnLaunch: false,
 tags: {
 Name: "pulumi-vpc-private-1",
 Owner: "pulumi",
 Project: "pulumi-k8s-aws-cluster",
 "kubernetes.io/role/internal-elb": "1",
 type: "private",
 },
 vpcId: "vpc-0ad77710973388316",
}, {
 protect: true,
});

import pulumi
import pulumi_aws as aws

application_vpc = aws.ec2.Vpc("application-vpc",
 assign_generated_ipv6_cidr_block=False,
 cidr_block="172.16.0.0/16",
 enable_dns_support=True,
 instance_tenancy="default",
 tags={
 "Name": "pulumi-vpc",
 "Owner": "pulumi",
 "Project": "pulumi-k8s-aws-cluster",
 },
 opts=ResourceOptions(protect=True))
public_1 = aws.ec2.Subnet("public-1",
 assign_ipv6_address_on_creation=False,
 cidr_block="172.16.32.0/19",
 map_public_ip_on_launch=True,
 tags={
 "Name": "pulumi-vpc-public-1",
 "Owner": "pulumi",
 "Project": "pulumi-k8s-aws-cluster",
 "kubernetes.io/role/elb": "1",
 "type": "public",
 },
 vpc_id="vpc-0ad77710973388316",
 opts=ResourceOptions(protect=True))
private_1 = aws.ec2.Subnet("private-1",
 assign_ipv6_address_on_creation=False,
 cidr_block="172.16.160.0/19",
 map_public_ip_on_launch=False,
 tags={
 "Name": "pulumi-vpc-private-1",
 "Owner": "pulumi",
 "Project": "pulumi-k8s-aws-cluster",
 "kubernetes.io/role/internal-elb": "1",
 "type": "private",
 },
 vpc_id="vpc-0ad77710973388316",
 opts=ResourceOptions(protect=True))

package main

import (
 "github.com/pulumi/pulumi-aws/sdk/v3/go/aws/ec2"
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 _, err := ec2.NewVpc(ctx, "application_vpc", &ec2.VpcArgs{
 AssignGeneratedIpv6CidrBlock: pulumi.Bool(false),
 CidrBlock: pulumi.String("172.16.0.0/16"),
 EnableDnsSupport: pulumi.Bool(true),
 InstanceTenancy: pulumi.String("default"),
 Tags: pulumi.StringMap{
 "Name": pulumi.String("pulumi-vpc"),
 "Owner": pulumi.String("pulumi"),
 "Project": pulumi.String("pulumi-k8s-aws-cluster"),
 },
 }, pulumi.Protect(true))
 if err != nil {
 return err
 }
 _, err = ec2.NewSubnet(ctx, "public_1", &ec2.SubnetArgs{
 AssignIpv6AddressOnCreation: pulumi.Bool(false),
 CidrBlock: pulumi.String("172.16.32.0/19"),
 MapPublicIpOnLaunch: pulumi.Bool(true),
 Tags: pulumi.StringMap{
 "Name": pulumi.String("pulumi-vpc-public-1"),
 "Owner": pulumi.String("pulumi"),
 "Project": pulumi.String("pulumi-k8s-aws-cluster"),
 "kubernetes.io/role/elb": pulumi.String("1"),
 "type": pulumi.String("public"),
 },
 VpcId: pulumi.String("vpc-0ad77710973388316"),
 }, pulumi.Protect(true))
 if err != nil {
 return err
 }
 _, err = ec2.NewSubnet(ctx, "private_1", &ec2.SubnetArgs{
 AssignIpv6AddressOnCreation: pulumi.Bool(false),
 CidrBlock: pulumi.String("172.16.160.0/19"),
 MapPublicIpOnLaunch: pulumi.Bool(false),
 Tags: pulumi.StringMap{
 "Name": pulumi.String("pulumi-vpc-private-1"),
 "Owner": pulumi.String("pulumi"),
 "Project": pulumi.String("pulumi-k8s-aws-cluster"),
 "kubernetes.io/role/internal-elb": pulumi.String("1"),
 "type": pulumi.String("private"),
 },
 VpcId: pulumi.String("vpc-0ad77710973388316"),
 }, pulumi.Protect(true))
 if err != nil {
 return err
 }
 return nil
 })
}

using Pulumi;
using Aws = Pulumi.Aws;

class MyStack : Stack
{
 public MyStack()
 {
 var application_vpc = new Aws.Ec2.Vpc("application-vpc", new Aws.Ec2.VpcArgs
 {
 AssignGeneratedIpv6CidrBlock = false,
 CidrBlock = "172.16.0.0/16",
 EnableDnsSupport = true,
 InstanceTenancy = "default",
 Tags =
 {
 { "Name", "pulumi-vpc" },
 { "Owner", "pulumi" },
 { "Project", "pulumi-k8s-aws-cluster" },
 },
 }, new CustomResourceOptions
 {
 Protect = true,
 });
 var public_1 = new Aws.Ec2.Subnet("public-1", new Aws.Ec2.SubnetArgs
 {
 AssignIpv6AddressOnCreation = false,
 CidrBlock = "172.16.32.0/19",
 MapPublicIpOnLaunch = true,
 Tags =
 {
 { "Name", "pulumi-vpc-public-1" },
 { "Owner", "pulumi" },
 { "Project", "pulumi-k8s-aws-cluster" },
 { "kubernetes.io/role/elb", "1" },
 { "type", "public" },
 },
 VpcId = "vpc-0ad77710973388316",
 }, new CustomResourceOptions
 {
 Protect = true,
 });
 var private_1 = new Aws.Ec2.Subnet("private-1", new Aws.Ec2.SubnetArgs
 {
 AssignIpv6AddressOnCreation = false,
 CidrBlock = "172.16.160.0/19",
 MapPublicIpOnLaunch = false,
 Tags =
 {
 { "Name", "pulumi-vpc-private-1" },
 { "Owner", "pulumi" },
 { "Project", "pulumi-k8s-aws-cluster" },
 { "kubernetes.io/role/internal-elb", "1" },
 { "type", "private" },
 },
 VpcId = "vpc-0ad77710973388316",
 }, new CustomResourceOptions
 {
 Protect = true,
 });
 }

}

Check out the video clip below for a demo.

Documentation

Full CLI Documentation is available, showing all of the different import command flags. The flags include writing directly to a file and the ability to ensure a resource is not protected on import. Examples of importing resources can be found on their specific resource documentation pages. Such an example for importing a VPC can be found in the VPC import documentation.

Adopting Existing Cloud Resources into Pulumi

Luke Hoban — Thu, 15 Aug 2019 00:00:00 +0000

See Migrating From Terraform to Pulumi for updated documentation on migrating from Terraform.

Most cloud infrastructure projects involve working with existing cloud resources — either building on top of existing resources or adopting existing resources under management with a new and more robust infrastructure provisioning solution.

Whether you are adopting resources that were deployed manually using your cloud provider’s console or CLI — or migrating existing infrastructure from tools like Terraform or CloudFormation — Pulumi makes it easy to adopt and manage your existing resources.

When working with existing resources, there are typically two scenarios:

You need to reference existing resources to use as inputs to new resources in Pulumi
You need to adopt existing resources under management so they can be updated/managed in Pulumi

We’ll review referencing existing resources, and then dive deeper into how you can adopt existing resources with Pulumi.

Referencing Existing Resources

For referencing existing resources, Pulumi offers several tools.

The .get methods available on every resource let you get all the details for a resource from the cloud provider based just on its id.
The StackReference resource lets you reference outputs of another stack for use as inputs to a stack, which is very useful for organizing projects and stacks.
terraform.state.RemoteStateReference(), aws.cloudformation.getStack() and azure.core.SubscriptionTemplateDeployment.get() let you reference outputs from existing Terraform, CloudFormation and ARM deployments respectively.

Together, these make it easy to reference existing infrastructure regardless of how it was provisioned.

Adopting Existing Resources

For adopting existing resources, Pulumi offers the import resource option to request that a resource defined in your Pulumi program adopt an existing resource in the cloud provider instead of creating a new one. In keeping with its focus on infrastructure as code, Pulumi lets you specify this import behavior inside the Pulumi code for your infrastructure deployment, instead of outside of it in some manual workflow. In its simplest form, it looks like this:

const myVpc = new aws.ec2.Vpc("my-vpc", {
 cidrBlock: "10.11.0.0/16",
}, { import: "vpc-02cd0cd2d807f340c"});

When Pulumi first sees a resource (in this case my-vpc), if it has an import option set, it will adopt the existing resource specified by that import ID from the cloud provider under management instead of creating a new resource.

We can see this when we deploy our Pulumi program with pulumi up:

$ pulumi up
Previewing update (dev):
Type Name Plan
pulumi:pulumi:Stack importtestaws-dev
= └─ aws:ec2:Vpc my-vpc import
Resources:
= 1 to import
1 unchanged

Because the import is provided in code, it can be configured in several different ways. You can look up import ids from Pulumi config instead of hard-coding them into programs, construct ids from predictable names based on program parameters, or even conditionally add the import: <id> property depending on whether one wants to deploy a stack in “adoption” or “creation” mode.

Import can be used for a wide variety of adoption scenarios, from importing a single resource to migrating an entire stack from an existing tool like Terraform. You can even automate an entire migration process across dozens of instances of infrastructure deployment. For small numbers of resources, you can just paste in individual resource IDs. For larger conversions, you can create a mapping of cloud resource ids (in JSON, CSV, or any other format) and load that into your Pulumi program with something like import: idMapping[name]. If you are importing or migrating dozens of stacks, you can even select between which of these mappings to use via a Pulumi config setting.

Walkthrough of Adopting Existing Infrastructure to Pulumi

To see one of these scenarios in action end-to-end, let’s walk through the process of deploying some “existing” infrastructure outside of Pulumi, and then adopting it under management of a Pulumi program. You can also check out the video below which walks through the same scenario.

We’ll use Terraform to provision our “existing” infrastructure. These resources could have been provisioned manually in the console, or using platform-native infrastructure-as-code solutions like CloudFormation, ARM or Kubernetes YAML.

We’ll also use TypeScript, though of course, the process is similar regardless of the language you choose.

1. Deploy “existing” infrastructure

For our “existing” infrastructure, we’ll use a small Azure deployment of a Virtual Machine and corresponding Resource Group and Networking resources. We’ll deploy this example using terraform. Save the code in that example to main.tf and run terraform init and then terraform apply to deploy it.

2. Create a Pulumi program

Next, we’ll create a Pulumi program to describe the same infrastructure that exists in Azure. We could write this by hand, but since we used Terraform to define the original infrastructure, we also have the option to use tf2pulumi to automatically convert our .tf files to Pulumi.

As of Pulumi CLI v3.71.0, tf2pulumi has been replaced with pulumi convert --from terraform. Converting Full Terraform Programs to Pulumi blog has more details.

$ pulumi new azure-typescript --force
$ tf2pulumi

We now have a Pulumi program in index.ts that describes the same infrastructure:

import * as pulumi from "@pulumi/pulumi";
import * as azure from "@pulumi/azure";

const config = new pulumi.Config();
const prefix = config.get("prefix") || "tfvmex";

const mainResourceGroup = new azure.core.ResourceGroup("main", {
 location: "West US 2",
 name: `${prefix}-resources`,
});
const mainVirtualNetwork = new azure.network.VirtualNetwork("main", {
 addressSpaces: ["10.0.0.0/16"],
 location: mainResourceGroup.location,
 name: `${prefix}-network`,
 resourceGroupName: mainResourceGroup.name,
});

...

If we try to run a pulumi up now to deploy this though, we see that as we expect, we will try to deploy a new copy of the infrastructure:

$ pulumi up
Previewing update (dev2):
Type Name Plan
+ pulumi:pulumi:Stack importazure-dev2 create
+ ├─ azure:core:ResourceGroup main create
+ ├─ azure:network:VirtualNetwork main create
+ ├─ azure:network:Subnet internal create
+ ├─ azure:network:NetworkInterface main create
+ └─ azure:compute:VirtualMachine main create
Resources:
+ 6 to create

Instead of creating new resources though, we want to adopt the existing ones.

3.Add `import` IDs

As we saw earlier, to import existing resources, we simply add import: <id> to the resource options of a resource defined in Pulumi. That resource will then be imported instead of created.

We can try this for one resource first. We can look in our terraform.tfstate file or in the Azure Portal and see that the id of the Resource Group is /subscriptions/0282681f-7a9e-424b-80b2-96babd57a8a1/resourceGroups/tfvmex-resources. So let’s use that:

const mainResourceGroup = new azure.core.ResourceGroup("main", {
 location: "West US 2",
 name: `${prefix}-resources`,
}, { import: "/subscriptions/0282681f-7a9e-424b-80b2-96babd57a8a1/resourceGroups/tfvmex-resources" });

4. Adopt resources

Now our preview shows that it will import the ResourceGroup instead:

$ pulumi up
Previewing update (dev2):
Type Name Plan
+ pulumi:pulumi:Stack importazure-dev2 create
= └─ azure:core:ResourceGroup main import
Resources:
+ 1 to create
= 1 to import
2 changes

We can complete this process, adding import: <id> to all resources, and then import them all to Pulumi.

Updating (dev2):
Type Name Status
+ pulumi:pulumi:Stack importazure-dev2 created
= ├─ azure:core:ResourceGroup main imported
= ├─ azure:network:VirtualNetwork main imported
= ├─ azure:network:Subnet internal imported
= ├─ azure:network:NetworkInterface main imported
= └─ azure:compute:VirtualMachine main imported
Resources:
+ 1 created
= 5 imported
6 changes
Duration: 8s

As noted earlier, if we were doing this at larger scale, instead of putting resource IDs inline in the code, we could load the IDs from a file or from Pulumi config. These IDs could be automatically extracted from our .tfstate file or from queries against our cloud provider.

5. Managing the resources with Pulumi

Once all of the resources are adopted into Pulumi, we can manage them by making incremental changes in Pulumi, and can discontinue using the existing .tf files to manage this set of infrastructure. For example, if we want to add a new tag to our Virtual Machine, we can use the code below. We can also now remove the import property if we want, or else leave it intact so long as the id of the resource remains the same. (If in the future we need to make a change that replaces one of these imported resources, Pulumi will require that the id property is removed, as it will no longer be correct.)

const mainVirtualMachine = new azure.compute.VirtualMachine("main", {
 location: mainResourceGroup.location,
 name: `${prefix}-vm`,
 networkInterfaceIds: [mainNetworkInterface.id],
 osProfile: {
 // Note: This comes directly from the original Terraform example, but
 // you would typically replace this with secret config in Pulumi, which
 // you can do after importing these resources into Pulumi.
 adminPassword: "Password1234!",
 adminUsername: "testadmin",
 computerName: "hostname",
 },
 osProfileLinuxConfig: {
 disablePasswordAuthentication: false,
 },
 resourceGroupName: mainResourceGroup.name,
 storageImageReference: {
 offer: "UbuntuServer",
 publisher: "Canonical",
 sku: "16.04-LTS",
 version: "latest",
 },
 storageOsDisk: {
 caching: "ReadWrite",
 createOption: "FromImage",
 managedDiskType: "Standard_LRS",
 name: "myosdisk1",
 },
 tags: {
 environment: "staging",
+ managedBy: "Pulumi",
 },
 vmSize: "Standard_DS1_v2",
});

Deploying this, Pulumi shows we will update the existing resource to add the new tag:

 ~ tags: {
+ managedBy: "Pulumi"
}

Since the resources are now being managed by Pulumi, we can use the Pulumi Service to see them:

And then navigate to the Virtual Machine in the Azure Portal and see the newly added managedBy: Pulumi tag.

The result is that we have migrated our existing infrastructure into Pulumi and can now use all the rich features of Pulumi to manage it going forward!

Conclusion

With Pulumi, you can easily work with existing cloud infrastructure, either referencing the existing infrastructure as inputs to new cloud infrastructure defined in Pulumi, or adopting existing infrastructure under management with Pulumi.

Get started with Pulumi today!