Pulumi Blog: Import

Migrate CloudFormation to Pulumi with Discovered Stacks

Alejandro Cotroneo — Thu, 30 Jul 2026 00:00:00 +0000

With Discovered Stacks, Pulumi Cloud does the bookkeeping for a CloudFormation migration: every resource in the stack gets an explicit migration status, and the migration is done when the code provably matches the cloud. In this tutorial, we take one real CloudFormation stack from discovered to migrated and managed by Pulumi IaC, end to end.

What we’re migrating

Our example is payments-api, a CloudFormation stack with 61 resources: a VPC, an Aurora ledger database behind an RDS Proxy, an assets S3 bucket, a DynamoDB ledger table, a Kinesis payment-events pipeline, and the IAM roles, KMS keys, and secrets that wire them together. The plan has five steps:

Find the stack in Pulumi Cloud.
Review the migration breakdown at a glance.
Start the migration.
Resolve the stragglers, so every resource is accounted for.
Confirm the quality gate: a zero-diff pulumi preview.

We’ll use Pulumi Neo to do the heavy lifting, but nothing here depends on it. The same flow works with your own coding agent or entirely by hand, because migration status is derived from actual stack state — however the work gets done, the console shows the same progress.

Step 1: Find your stack in Pulumi Cloud

Discovered Stacks builds on Discovery, so the only prerequisite is a scanned cloud account — the AWS account holding your CloudFormation stacks.

Once a scan has run, open the Stacks page and turn on Show Discovered Stacks. Your CloudFormation stacks appear alongside your Pulumi stacks. The project name comes from the CloudFormation stack (payments-api), and the stack name encodes the account and region it came from, so the same template deployed to two regions shows up as two distinct discovered stacks.

Step 2: Plan the migration at a glance

Open the discovered stack’s Migration tab. It lays out all 61 resources of payments-api by status, so you can visualize the migration before touching anything:

54 Ready: mapped to a Pulumi type and confirmed to exist — importable right now.
2 Not found: mapped, but Discovery couldn’t confirm their current state.
5 No exact match: no direct Pulumi type. We’ll come back to these in step 4.

Every row also pairs the origin type (AWS::S3::Bucket) with the Pulumi type it maps to (aws:s3/bucket:Bucket).

Step 3: Start the migration

Most of payments-api has a direct migration path — 54 of its 61 resources are Ready — so we’ll reach for the tab’s quick actions and let them do the bulk of the work.

From the discovered stack’s Actions menu, select Migrate with Neo. Neo asks where the code should live — a git repository, a target project and stack, a language — and then works through the migration:

Fetches the discovered resources and their statuses through the Discovered Stacks API.
Imports the Ready resources in batches with pulumi import, building up a Pulumi program as it goes.
Runs pulumi preview after each batch and reconciles the generated code against the real cloud state.
Opens a pull request with the program and a migration report.

Because pulumi import writes state as it runs, the console updates live: statuses flip from Ready to Migrated batch by batch, without anyone marking a checkbox.

Step 4: Resolve the stragglers

That leaves seven rows that need review. In this case, Neo handled them automatically, leaving an annotation on each that records the decision it made:

The 2 Not found rows are CloudWatch log groups. Discovery couldn’t confirm their state, but they’re live, so the import succeeded and the resources were resolved without further inspection.
Three No exact match rows are inline IAM policies, which our Pulumi AWS Provider imported as part of their parent role. The roles are already migrated, so each policy is marked resolved.
One is a Secrets Manager target attachment; Pulumi expresses that link through the database’s own configuration, so it’s marked resolved once the database is migrated.
The last is a CDK-generated custom resource that strips the rules from the VPC’s default security group. Pulumi models that directly as an aws.ec2.DefaultSecurityGroup, so there’s nothing to import — it’s marked resolved.

Step 5: The quality gate

With every resource imported or resolved, the migration earns its trust in one final check: a zero-diff pulumi preview. The generated program, run against the live cloud, proposes no changes.

Resources:
 56 unchanged

A clean preview means the code matches reality. If it shows a diff, the code gets fixed until it doesn’t; the cloud is never modified to make the code look right. And notably, there’s no pulumi up in this story: importing already synced the state to Pulumi Cloud, so the first up you run is for the first real change you make after the migration.

That zero-diff preview isn’t the finish line, it’s a checkpoint you can build on. Imported code is faithful but rarely the code you’d write by hand: generated names, repeated blocks, inline configuration. Now that it provably matches the cloud, refactor freely — pull settings into stack config, split the program into modules, collapse repetition into loops, group related resources into components. Rerun pulumi preview after each change: a clean zero-diff means you reshaped the code without touching the infrastructure. When a refactor shifts a resource’s identity (for example, moving it into a new component), an alias keeps it a no-op.

Where you end up

The payments-api program now lives in your repository, reviewed and merged like any other code. The discovered stack remains as the migration record until you decide to remove the CloudFormation stack. And because progress is derived from real state, if the CloudFormation stack changes later, those changes surface against your Pulumi stack. Migration doesn’t have to be all-or-nothing either: for a larger stack, migrate resources progressively and safely by never running pulumi up while CloudFormation is managing the resources. That’s the migration end to end: a CloudFormation stack turned into a Pulumi program you own and manage as code, every resource accounted for and provably in sync.

The same flow works for Azure Resource Manager deployments, which Discovery models as discovered stacks too. And for Pulumi-hosted Terraform stacks, a Migration tab offers the identical experience, with statuses derived from the Terraform state.

To go deeper:

Go from Cloud to Code in Minutes with Visual Import

Pulumi Insights Team — Thu, 08 May 2025 14:41:53 -0500

Note: This post discusses Pulumi Copilot, which Pulumi Neo has replaced. Learn about Neo →

Pulumi is excited to announce the release of Visual Import, a new feature that transforms the way platform teams onboard existing cloud infrastructure into Pulumi. With Visual Import, you can discover unmanaged cloud resources across your organization, organize them into logical units, and instantly generate high-quality Pulumi code—with AI enhancements built in.

Visual Import is now available for all Team, Enterprise and Business Critical customers with Pulumi Insights enabled.

Modern Infrastructure, Codified Faster

One of the biggest challenges platform teams face when adopting infrastructure as code is migrating existing cloud deployments. Legacy scripts, manual configurations, and scattered resource ownership all slow down progress. Visual Import solves this by making the journey from cloud to code seamless and intuitive.

Instead of manually digging through consoles and writing boilerplate code by hand, Visual Import enables teams to:

Discover all cloud resources—across any number of accounts—via Pulumi Insights.
Select and group resources into meaningful stacks, services, or projects.
Generate production-ready Pulumi code in your language of choice.
Enhance that code using Pulumi Copilot, which refines formatting, naming, references, and documentation.
Integrate with your development workflows by downloading, editing, and importing Pulumi code into your existing repositories.

Whether you’re modernizing infrastructure, unifying multi-account environments, or enabling developers to adopt infrastructure as code more easily, Visual Import gives you a fast, repeatable onramp to scalable, consistent cloud management.

How It Works

Visual Import is a guided three-step workflow within the Pulumi Cloud Console:

Select Resources: Use powerful filters and grouping tools to find the resources you want to codify. Whether you’re capturing a full service or just a few related components, Visual Import gives you visibility and precision.
Review References: Automatically discover and select dependent resources—ensuring your code captures the right context and connectivity between infrastructure elements.
Generate Code: Choose your preferred language and preview or download fully-formed Pulumi code. You can refine it manually or invoke Pulumi Copilot to improve naming, structure, and comments.

Throughout the workflow, you maintain full control. Select what you need, tweak it as required, and integrate on your terms. You can find full documentation and usage details in the Visual Import docs.

Key Benefits

Accelerated Migration

Visual Import dramatically shortens the time it takes to transition existing cloud infrastructure into a managed, version-controlled state. What previously required weeks of manual discovery, scripting, and trial-and-error can now be accomplished in a matter of hours.

Whether you’re consolidating legacy environments, auditing infrastructure before a cloud cost review, or preparing for a compliance milestone, Visual Import automates much of the busywork—so your team can focus on what matters.

Increased Developer Productivity

Manually transitioning existing infrastructure to code often pulls developers away from their primary focus—building and improving services. By automating code generation and applying intelligent AI enhancements, Visual Import lets teams spend less time writing boilerplate code and fighting resource drift, and more time shipping meaningful changes.

Pulumi Copilot adds a layer of polish that typically requires hands-on review and iteration, saving your team valuable time while helping new contributors write clear, consistent infrastructure code from the start.

Consistent, Reliable Infrastructure

Unifying your cloud environments under a single IaC framework brings consistency, accountability, and repeatability to operations. Pulumi’s typed IaC model makes configurations explicit and auditable, while policies and CI/CD integrations enforce best practices from the start.

By codifying your infrastructure, you reduce the risk of configuration drift, simplify onboarding, and ensure reproducibility across staging and production.

From Insights to IaC—Seamlessly

Visual Import builds on Pulumi Insights, providing a natural path from discovery to codification. If you’re already using Insights to monitor and visualize your infrastructure, you can now convert those resources into managed, versioned code in just a few clicks.

And with built-in support for dependency analysis and cross-account visibility, you can finally treat your entire infrastructure portfolio as a single, coherent system—even if it wasn’t built that way.

Leverage the Full Power of Pulumi Cloud

Once resources are in code, you can take full advantage of Pulumi Cloud’s full suite of features: define and enforce policies with CrossGuard, gain visibility into your entire infrastructure through Pulumi Insights, manage state with ESC’s secrets management, and automate workflows through Pulumi Deployments. Together, these features turn your infrastructure into a governed, observable, and continuously deployable system built for modern cloud operations.

Try It Today

Visual Import is now available in the Pulumi Cloud Console for Team, Enterprise and Business Critical users. If you’re ready to modernize your infrastructure and empower your teams with a fast path to Infrastructure as Code, there’s no better place to start.

Pulumi adoption made easy with the new Migration Hub

Joe Duffy — Tue, 22 Aug 2023 00:00:00 +0000

Today we are launching Pulumi’s new Migration Hub, a comprehensive guide to help you seamlessly adopt Pulumi no matter where you are coming from, whether that’s Terraform, CloudFormation, … or even manually provisioned resources not yet governed by an infrastructure as code solution. Our new Expert Services group is ready to roll up their sleeves to help you adopt Pulumi faster. The Migration Hub also features many commercial offers for open source foundations, startups, and complementary migration, to minimize switching costs and risks. It’s never been easier to adopt Pulumi.

Why Pulumi?

Nearly 2,000 customers and over 130,000 practitioners have chosen Pulumi for their infrastructure as code needs. About half the time, these end users have an existing infrastructure as code solution that has stopped meeting their needs. The other half of the time, infrastructure may have been created in the cloud console (“click-ops”), using a cloud CLI, or some other manual provisioning tool, but now needs to become repeatable and scalable using infrastructure as code.

The most common reasons teams prefer Pulumi include:

Best-in-class productivity
Accessible to the entire organization, thanks to industry standard languages
Ability to deal with complexity at scale, thanks to stacks, config, sharing and reuse, and more
Perfect confidence thanks to policies and testing
Secure by-default, with built-in support for secrets
Highly programmable engine can be embedded anywhere, to build custom platforms and tools
Instant collaborative bridge between developers and infrastructure experts

Thanks to Pulumi, teams are able to do more with less, ship faster with confidence, tackle very challenging cloud infrastructure projects and scale with them, empower more of the team to be self-serve and collaborate more closely, and ultimately turn the cloud into a competitive advantage.

Pulumi is also true Apache 2.0 open source and we welcome anyone who wishes to build atop it.

Migration

Of course, it’s one thing to choose Pulumi, but another thing entirely to adopt it and get all of your infrastructure under its management, especially if you have existing infrastructure. That might include converting existing code, or even just importing existing manually provisioned cloud resources.

There are two major approaches to this challenge.

Self-Serve Migration

Pulumi offers many tools and techniques should you want to perform a migration on your own.

In the most general form, Pulumi provides an import command which can import any existing infrastructure no matter how it was originally created, click-ops included.

Pulumi also offers conversion tools for other popular infrastructure as code solutions, including HashiCorp Terraform, AWS CloudFormation, Azure Resource Manager (ARM), or Kubernetes YAML.

Although the time to migrate existing infrastructure varies wildly based on the size and complexity of your scenario, migrating a single cloud environment can often take as little as a few hours. And thanks to Pulumi being multi-language at its core, these tools will support you the same no matter which language you choose, JavaScript, TypeScript, Python, Go, C#, Java, and YAML alike.

To learn more about performing your own migration, please read this user guide.

Let Us Help You Out with Expert Services!

Although self-serve is an option, there can often still be uncertainty around this approach.

It may not be clear how much time a full migration will take, leaving the possibility that you’ll be left with multiple infrastructure as code solutions – either temporarily or even permanently. Although the tools are a huge help, they don’t always get you 100% of the way there. We also often hear that skilling yourself up on a one-time migration activity takes away from critical time spent on business initiatives.

All of this adds friction, risk, and uncertainty to the Pulumi adoption decision. The last thing we want is for the one-time migration to prevent you from adopting Pulumi. Thankfully, we are here to help!

Today we are announcing the formation of Pulumi’s Expert Services group. This team of industry and infrastructure as code experts is on standby and ready to roll up their sleeves to help you with migration, training, and other services efforts. No project is too small: we are currently working with customers with 100KLOCs of legacy infrastructure as code that needs to be migrated, for example.

Today we are also announcing an extension to our Terraform Migration Offer, in which we will bundle migration of your Terraform projects with your Pulumi Enterprise or Business Critical subscription. Today we are excited to announce that this same package is now available for all migration efforts, including AWS CloudFormation, Azure Resource Manager (ARM), …, or even infrastructure that was manually provisioned. We are also prepared to scale up from there depending on your needs.

Success Stories

Over time, Pulumi is becoming a more comprehensive platform, with new services like Pulumi Deployments for multi-cloud deployment workflows and Pulumi Insights for search, analytics, and AI for your cloud infrastructure. Pulumi also offers policies and secrets as core features of the platform. Enterprises are increasingly betting on Pulumi for their entire cloud management and platform needs.

By moving to Pulumi’s platform, many customers have seen considerable benefits:

Atlassian significantly increased developer velocity, increasing productivity by more than 2x and substantially decreasing the amount of time spent on maintenance tasks.

Mercedes-Benz got the whole team shipping faster and with more confidence by building a platform that helped developers be self-serve with infrastructure while scaling up to 100s of Kubernetes clusters worldwide.

Snowflake rapidly improved time to market by betting on Pulumi, empowering their engineers, and delivering the Data Cloud. Snowflake ultimately beat their IPO deadlines and knew legacy infrastructure as code wouldn’t do it.

Washington Trust Bank modernized their cloud infrastructure and practices, speeding up internal delivery, while also increasing their confidence thanks to policy as code guardrails.

Fauna adopted a born in the cloud mindset and was able to build a multi-cloud, highly scalable, modern SaaS application, actually increasing reliability even in the face of growing complexity.

In each of these cases, the customer evaluated Pulumi against alternative infrastructure as code solutions, but ultimately chose Pulumi due to the aforementioned benefits.

Special Offers

In addition to migration services, Pulumi offers two programs to help get up and running quickly:

Open Source: Any open source foundation or project can use Pulumi Team Edition free of charge.
Startup Discount: Any early stage startup gets $10,000 free credits and access to Pulumi Enterprise.

Calling All Partners!

As Pulumi’s market adoption has skyrocketed, we’ve begun to build out our official partner program. Although we don’t have anything major to announce (yet), we’d love to talk to you if infrastructure as code is an important part of your own customers’ needs. That includes ISV product integrations, global and regional systems integrators and consultants, and more. Note that Pulumi is already available in the AWS and Azure marketplaces. If you are interested in discussing a partnership, please contact us.

Get Started Today

Migrating from an existing solution can seem like a daunting task, but with Pulumi’s new Migration Hub and Expert Services, it has never been easier to adopt Pulumi than it is today. To learn more, visit the Migration Hub or simply contact us – we would love to hear from you.

Google Cloud: Bulk Importing Resources into Pulumi

Josh Kodroff — Thu, 16 Mar 2023 00:00:00 +0000

Point and click in the console is great when you’re first starting out learning a new cloud or managed service, but it quickly becomes a hindrance when cloud infrastructure is widely adopted by an organization. The point at which the term “widely adopted” becomes applicable to your situation differs, but at some point in their careers, many infrastructure and platform engineers are faced with situations where a large number of critical infrastructure resources were created through “click ops” with no ability to track changes, reproduce environments consistently, and so on. When this happens (and it will probably happen to many of you), it’s time to import those resources into infrastructure as code.

Fortunately, Pulumi has one of the smoothest and most powerful import processes of any IaC tool. In this post, we’re going to show you how to automate the bulk importation of Google Cloud resources into Pulumi! This approach will also work on resources that were created by another IaC tool.

In a previous post titled Automating Pulumi Import with Manually Created Resources, we covered the details of how pulumi import works and demonstrated a workflow of bulk importing resources from AWS. We won’t repeat the details of how pulumi import works here, so please refer to “Automating Pulumi Import with Manually Created Resources” for those details.

Writing an account scraper

In our AWS account scraper, we took the approach of querying each resource type using boto3, the AWS Python SDK. In contrast, Google Cloud has a tool called Config Connector that allows us to query all resources in a Google Cloud project with a single command, and without needing to write the code to query each resource type via the Google Cloud SDK.

Our process for importing from Google Cloud has the following steps:

Query resources using Config Connector and output them to a YAML file.
In a Python script, take the Config Connector YAML output and transform it into JSON suitable for a bulk pulumi import operation.
Run the pulumi import command and place the generated code in our Pulumi program.

Config Connector

Config Connector is designed to manage Google Cloud resources represented as Kubernetes Custom Resource Definitions (CRDs). We will not be using this capability of Config Connector, but if you are interested in managing Pulumi stacks as CRDs in a GitOps workflow, check out the Pulumi Kubernetes Operator. Instead, we’ll use Config Connector’s bulk export capability to query our Google Cloud environment for resources and output them as Kubernetes manifests, then transform those Kubernetes manifests into JSON suitable for a bulk pulumi import operation via a script written in Python.

First, we need to install the config-connector CLI per the instructions in the Config Connector docs. Then, to generate our Kubernetes manifests, we run the following command:

config-connector bulk-export --project <YOUR PROJECT ID> --output bulk-export.yaml --iam-format policymember --on-error continue

This command will generate a YAML file with a series of Kubernetes manifests like the following:

---
apiVersion: artifactregistry.cnrm.cloud.google.com/v1beta1
kind: ArtifactRegistryRepository
metadata:
 annotations:
 cnrm.cloud.google.com/project-id: my-google-cloud-project
 labels:
 goog-managed-by: cloudfunctions
 name: gcf-artifacts
spec:
 description: This repository is created and used by Cloud Functions for storing
 function docker images.
 format: DOCKER
 location: europe-west1
 resourceID: gcf-artifacts
---
apiVersion: compute.cnrm.cloud.google.com/v1beta1
kind: ComputeDisk
metadata:
 labels:
 goog-gke-volume: ""
 name: gke-helloworld-2a71d4f-pvc-1e6aa931-d742-437e-a52b-413999ace2be
spec:
 description: '{"kubernetes.io/created-for/pv/name":"pvc-1e6aa931-d742-437e-a52b-413999ace2be","kubernetes.io/created-for/pvc/name":"wpdev-wordpress","kubernetes.io/created-for/pvc/namespace":"default"}'
 location: us-central1-a
 physicalBlockSizeBytes: 4096
 projectRef:
 external: my-google-cloud-project
 resourceID: gke-helloworld-2a71d4f-pvc-1e6aa931-d742-437e-a52b-413999ace2be
 size: 10
 type: pd-standard
---
# etc...

Now that we have a single file containing all of our Google Cloud resources, we can take this file as input for Python script.

Mapping from Kubernetes manifests to `pulumi import` JSON

We need to write some code in order to convert the Kubernetes manifests generated by Config Connector into a JSON format suitable for pulumi import. For detailed information on bulk import in Pulumi and all supported fields, see Bulk Import Operations in the Pulumi docs.

In order for a resource to be imported, three fields are required:

type, which indicates the Pulumi resource type, e.g. gcp:compute/network:Network.
name, which, when combined with type, uniquely identifies the resource within the Pulumi program. In the following example, the Pulumi name is my-network:
```
const network = new gcp.compute.Network("my-network");
```
id, which informs the pulumi import command how to query the Google Cloud API for the existing resource for its attributes and output the generated code with all attributes filled in. IDs for import vary by the type of resource, although in Google Cloud, they often follow a predictable pattern which we’ll discuss in further detail soon. The format of a resource’s ID can be found in the Pulumi Registry in the Import section of the resource’s API page (example).

In order to get the type, we need to translate the Kubernetes kind to a Pulumi type. We can accomplish this by keeping a straightforward mapping in a Python dict:

resource_type_mappings = {
 'ArtifactRegistryRepository': {
 'pulumi_type': 'gcp:artifactregistry/repository:Repository'
 },
 'ComputeDisk': {
 'pulumi_type': 'gcp:compute/disk:Disk',
 },
 'ComputeFirewall': {
 'pulumi_type': 'gcp:compute/firewall:Firewall'
 },
 # etc.

Obtaining the name is simple - virtually all of the Kubernetes manifests in our sample data contained a unique or near-unique identifier under k8s_resource['metadata']['name']. Occasionally, we saw naming collisions that cause errors, for example where a compute instance (VM) and its main disk resource shared the same name. At the time of writing, the pulumi import command requires that names be unique within the context of a bulk import. However, this is not a requirement when authoring Pulumi programs - the combination of (name, type) must be unique. In these cases, we simply ensured a unique name by appending -1 to the resource name. This isn’t the most elegant solution, but it works well enough when collisions are relatively infrequent:

def ensure_unique_name(pulumi_resource):
 """Ensures that each resource has a unique name. `pulumi import` will fail
 if multiple resources have the same name due to
 https://github.com/pulumi/pulumi/issues/6032"""

 for resource in resources:
 if 'name' in resource and 'name' in pulumi_resource and resource['name'] == pulumi_resource['name']:
 pulumi_resource['name'] += "-1"
 return

Finally, we need to map the id value. In many cases, the Google Cloud resource ID is in the form {{region}}/{{location}}/{{resource ID}}, and can be mapped from fields that are consistent in the Kubernetes manifest:

def get_default_id(k8s_resource):
 """Returns the most common form of an ID of a Google Cloud resource, adding
 region and zone if they can be determined"""
 id = ""

 if 'region' in k8s_resource['spec']:
 id += f"{k8s_resource['spec']['region']}/"

 if 'location' in k8s_resource['spec']:
 id += f"{k8s_resource['spec']['location']}/"

 id += k8s_resource['spec']['resourceID']

 return id

For any resource types that do not fit this pattern, we’ll need to do some additional work.

Mapping resources with custom ID requirements

If the Kubernetes manifest for our resource does not follow the standard pattern, we’ll need to write a custom function to get the ID. One example of resources that do not follow the standard pattern are compute instances and compute instance groups, which store the project ID in a different key:

apiVersion: compute.cnrm.cloud.google.com/v1beta1
kind: ComputeInstance
metadata:
 annotations:
 cnrm.cloud.google.com/project-id: my-project-id
# ...

Therefore, we need to write a custom function to handle grabbing the IDs from these types:

def get_compute_instance_id(k8s_resource):
 return f"{k8s_resource['metadata']['annotations']['cnrm.cloud.google.com/project-id']}/{k8s_resource['spec']['zone']}/{k8s_resource['metadata']['name']}"

And then map our custom function to the appropriate typed in our type mappings dict:

resource_type_mappings = {
 # ...
 'ComputeInstance': {
 'pulumi_type': 'gcp:compute/instance:Instance',
 'get_id': get_compute_instance_id,
 },
 'ComputeInstanceGroup': {
 'pulumi_type': 'gcp:compute/instanceGroup:InstanceGroup',
 'get_id': get_compute_instance_id,
 },
 # ...
}

The full code also contains some additional mapping and processing for Google Cloud IAM resources, but it follows the same basic patterns as the mapping for other types.

Importing to Pulumi

Now that we have our conversion script written, we can run it with the following command:

python3 main.py -i sample-output.yaml -o pulumi-import.json -p <your-project-id>

The command will generate output like the following:

{
 "resources": [
 {
 "type": "gcp:artifactregistry/repository:Repository",
 "name": "gcf-artifacts",
 "id": "europe-west1/gcf-artifacts"
 },
 {
 "type": "gcp:compute/disk:Disk",
 "name": "gke-helloworld-2a71d4f-pvc-1e6aa931-d742-437e-a52b-413999ace2be",
 "id": "us-central1-a/gke-helloworld-2a71d4f-pvc-1e6aa931-d742-437e-a52b-413999ace2be"
 },
// ...

Once we’re in our Pulumi program’s root, we can instruct pulumi import to do a bulk import and output the generated code to a file in the language of our program (this example includes TypeScript output, but pulumi import will automatically generate code in the correct language for your project).

pulumi import -f ../config-connector-transform/pulumi-import.json -y -s dev -o index.ts

And we can see that in our Pulumi program, all the attributes of our imported resources are present, which means no further actions are (typically - see note below for known exceptions) necessary! We are ready to manage our critical infrastructure as code from now on!

const gcf_artifacts = new gcp.artifactregistry.Repository("gcf-artifacts", {
 description: "This repository is created and used by Cloud Functions for storing function docker images.",
 format: "DOCKER",
 labels: {
 "goog-managed-by": "cloudfunctions",
 },
 location: "europe-west1",
 project: "your-project-id",
 repositoryId: "gcf-artifacts",
});

const gke_helloworld_2a71d4f_pvc_1e6aa931_d742_437e_a52b_413999ace2be = new gcp.compute.Disk("gke-helloworld-2a71d4f-pvc-1e6aa931-d742-437e-a52b-413999ace2be", {
 description: "{\"kubernetes.io/created-for/pv/name\":\"pvc-1e6aa931-d742-437e-a52b-413999ace2be\",\"kubernetes.io/created-for/pvc/name\":\"wpdev-wordpress\",\"kubernetes.io/created-for/pvc/namespace\":\"default\"}",
 labels: {
 "goog-gke-volume": "",
 },
 name: "gke-helloworld-2a71d4f-pvc-1e6aa931-d742-437e-a52b-413999ace2be",
 physicalBlockSizeBytes: 4096,
 project: "your-project-id",
 size: 10,
 zone: "us-central1-a",
});

You may need to massage some of the generated output for any *IAMPolicy resources because deleted principals cannot be contained in IAM policies due to a limitation of the Google Classic provider, but they are still contained in the generated code returned by pulumi import, which in turn comes from the Google API.

Next Steps

Now that we have our Google Cloud resources under Pulumi management, we might want to consider some additional steps to build upon our IaC solution:

Set up a CI/CD pipeline: By adding our Pulumi code to a delivery pipeline, we can help ensure that all infrastructure changes go through an automated process. Pulumi has helpful integrations and documentation for many popular tools. See Continuous Delivery in the Pulumi docs for more information.
Add policy as code: By adding policy as code to our IaC pipeline, we can ensure that our infrastructure is in compliance with any security or regulatory requirements before it’s ever provisioned in the cloud, as well as checks on any existing Pulumi resources. For more information on Pulumi’s policy as code capabilities, see CrossGuard (Policy as Code) in the Pulumi docs.

Conclusion

We’ve shown how the Google Cloud account scraper combined with pulumi import enables organizations to quickly get all of their Google Cloud resources under management with Pulumi. Managing cloud resources with Pulumi allows organizations to utilize modern, automated, code-centric approach, and pulumi import makes adopting Pulumi as painless as possible, even when critical resources have been created via point and click in the console!

If you use the Google Cloud account scraper and find something you need is not supported, please submit an issue, or better yet, submit a pull request!

Automating Pulumi Import with Manually Created Resources

Josh Kodroff — Mon, 30 Jan 2023 00:00:00 +0000

A few weeks ago, I was speaking with a consultant at one of the big firms who asked me how he could introduce Pulumi into a client’s organization when that client had created many infrastructure resources manually through the AWS console and was running production workloads on those resources.

Introducing modern cloud infrastructure tooling and automation is relatively simple (or at least more straightforward) when organizations decide to adopt IaC from the start of their cloud journey, but what about organizations who have gone far enough down the route of manually created cloud infrastructure to see the perils of that approach? Many teams come to this realization only when they’ve deployed too many production workloads to start over from scratch. If your organization is looking at Pulumi as an IaC solution, it’s worth bringing these resources under management because of the low effort and high value of having a single pane of glass to manage all of your resources.

Bringing resources under IaC (and Pulumi specifically) gives an organization an immediate return on investment. Note that while we list some benefits specific to the Pulumi Service, Pulumi supports a number of backends for both state management and secrets management and the Pulumi Service is not required in order to adopt Pulumi:

Version control/change history: With IaC placed under version control, teams can easily roll back to previous versions of infrastructure, see who changed what, and when, which provides critical visibility into how infrastructure has evolved over time. The Pulumi Service provides a great experience here, with a dashboard that collects version history and deployment logs along with git contextual data in a single pane of glass.
Automation and pipelines: Because IaC is code, it enjoys the same main benefits from automated pipelines as application code: a repeatable and reliable process. We can also enjoy significant security benefits: When IaC and automated pipelines become the standard way to deploy infrastructure, organizations can significantly reduce the number of users with direct access to critical infrastructure (i.e., via the cloud console or CLI) and the level of privileges needed (e.g., read-only rather than full admin). Once we get our pipeline established so that we have repeatable automated delivery of our infrastructure, we can add automated quality assurance to Pulumi programs via unit testing and policy as code.
Drift detection and correction: Most IaC tools will detect changes made out of band to resources, e.g., changes made manually in the console, and attempt to reconcile the resource back to its declared configuration. Thus, by bringing manually created resources under IaC, organizations can put a definitive end to the slow and error-prone practice of manually managing cloud resources. In Pulumi, the pulumi refresh command will refresh all resources in the state file. For more information on strategies to address configuration drift, check out Patterns of Drift Detection with Pulumi in the Pulumi blog.

About Pulumi Import

The pulumi import command is executed within the context of a Pulumi stack. The user supplies the type of resource to be imported, a name with which to identify the resource in the Pulumi program and state file, and a resource-dependent ID. For example, we might run the following command to import an AWS VPC (this command is sourced from the API docs for the AWS provider in the Pulumi Registry):

pulumi import aws:ec2/vpc:Vpc imported-vpc vpc-0b0a6ad0766eccf3b

When we run the command, Pulumi queries the AWS API for the VPC’s attributes. The resource’s attributes are added to the stack’s state file and the command prints out code to be added to our Pulumi program:

$ pulumi import aws:ec2/vpc:Vpc imported-vpc vpc-0b0a6ad0766eccf3b
Previewing import (dev)

View Live: https://app.pulumi.com/jkodrofftest/pulumi-import-blog-sample/dev/previews/2e462988-0771-4fd0-8e13-bf0b88eb9d4b

Type Name Plan
 + pulumi:pulumi:Stack pulumi-import-blog-sample-dev create
 = └─ aws:ec2:Vpc imported-vpc import

Resources:
 + 1 to create
 = 1 to import
 2 changes

Do you want to perform this import? yes
Importing (dev)

View Live: https://app.pulumi.com/jkodrofftest/pulumi-import-blog-sample/dev/updates/1

Type Name Status
 + pulumi:pulumi:Stack pulumi-import-blog-sample-dev created (4s)
 = └─ aws:ec2:Vpc imported-vpc imported (3s)

Resources:
 + 1 created
 = 1 imported
 2 changes

Duration: 5s

Please copy the following code into your Pulumi application. Not doing so
will cause Pulumi to report that an update will happen on the next update command.

Please note that the imported resources are marked as protected. To destroy them
you will need to remove the `protect` option and run `pulumi update` *before*
the destroy will take effect.

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

const imported_vpc = new aws.ec2.Vpc("imported-vpc", {
 cidrBlock: "10.0.0.0/16",
 tags: {
 Name: "vpc",
 },
}, {
 protect: true,
});

Note that in the preceding example we have TypeScript output, but pulumi import will automatically detect the language our Pulumi project uses and will output code in the correct language.

Our resource also has the protect: true option specified by default, which means that Pulumi will not delete (or recreate) the resource if the program changes. This is an overridable default setting chosen for safety: If a resource is being imported in the first place, it’s typically because there’s an important workload using it, and we should be very careful before deleting or re-creating it.

Now that we’ve run the pulumi import command and added the code that the command outputs, our resource can now be managed by Pulumi. If we run the pulumi preview command, we will see that our resource has been added to our stack and that no changes have been detected, meaning that Pulumi has correctly imported all of our VPC’s attributes:

$ pulumi preview
Previewing update (dev)

View Live: https://app.pulumi.com/jkodrofftest/pulumi-import-blog-sample/dev/previews/6ed858d9-c4f5-4770-9c22-c9aed39646bd

 Type Name Plan
 pulumi:pulumi:Stack pulumi-import-blog-sample-dev

Resources:
 2 unchanged

Automated Batch Import Capabilities

For projects with many resources, pulumi import also has an option that allows it to take a list of resources in a specially formatted JSON file for batch import. We’ll be using this batch import capability to add our manually-created resources into the Pulumi state. A basic sample JSON input follows:

{
 "resources": [
 {
 "type": "aws:ec2/vpc:Vpc",
 "name": "my-vpc",
 "id": "vpc-094958e4051c478c3"
 },
 {
 "type": "aws:ec2/vpc:Vpc",
 "name": "my-other-vpc",
 "id": "vpc-0f12a82357335a28f"
 }
 ]
}

Because pulumi import gives us the full code for an imported resource with all attributes (as demonstrated in the previous section), and because pulumi import has an option for batch import from an external file source (both key differentiators between Pulumi and other IaC tools) we can use pulumi import to import existing resources at scale in an automated fashion.

If you are looking to import resources that were created with a non-Pulumi IaC tool as opposed to manually in the console, the approach described in this article will still work, but you may also want to consider the pulumi convert --from terraform command and cf2pulumi tool for resources created with Terraform and CloudFormation, respectively. For a comprehensive guide on the various options for importing existing resources into Pulumi, see Migrating to Pulumi.

Designing an End-to-End Solution

Now that we’ve explained the benefits of bringing manually created resources under IaC, and the capabilities of the pulumi import command, we need to design a solution that will allow us to leverage pulumi import’s batch import capabilities so that we can bring all resources in our cloud environment under Pulumi management.

Our solution for an automated batch import of resources into Pulumi comprises the following steps:

Create an account scraper using our cloud provider’s SDK (or CLI) to query for the resources to be imported.
Take the output from the cloud provider’s SDK and transform it into a JSON file suitable as an import for a batch pulumi import operation.
Run pulumi import to bring the resources under Pulumi control.

Writing an Account Scraper

In our account scraper, we will use the AWS Python SDK, boto3, to query resources created in the AWS console, but a similar approach can be applied to the many cloud and SaaS providers supported by Pulumi. The full code for the sample solution can be found on GitHub at https://github.com/pulumi/pulumi-import-aws-account-scraper. If you find the tool useful, please submit issues and/or PRs to support additional resource types!

Because the ID attribute (or multiple attributes in the case of some resources) vary based on the resource type (in AWS at least, your cloud provider may vary), we need to write custom code for each resource type that we want to import. (Note that the account scraper code only queries a subset of AWS resource types, but the code can be easily modified to accommodate additional resource types.) For example, the following code shows a query for associations between route tables and VPC subnets:

def import_route_table_associations(ec2_client):
 pulumi_resources = []

 route_tables = ec2_client.describe_route_tables()['RouteTables']

 for route_table in route_tables:
 for association in route_table["Associations"]:
 if 'SubnetId' not in association:
 continue
 pulumi_resources.append({
 "type": "aws:ec2/routeTableAssociation:RouteTableAssociation",
 "name": f"import-{association['RouteTableAssociationId']}",
 "id": f"{association['SubnetId']}/{route_table['RouteTableId']}",
 })

 return pulumi_resources

We can run the entire account scraper program to output our resources to a single JSON file by running the following command. Detailed instructions are contained in the account scraper repository:

python account_scraper.py > /path/to/file.json

Batch Import

Now that we’ve generated our JSON file, we can use the pulumi import command to import all of our resources in a single command and output our code to its own file:

pulumi import -f /path/to/file.json -o imported-resources.ts -y

And with that, our resources are now managed by Pulumi!

Next Steps

Now that we have our resources under Pulumi management, we might want to consider some additional steps to build upon our IaC solution:

Set up a CI/CD pipeline: By adding our Pulumi code to a delivery pipeline, we can help ensure that all infrastructure changes go through an automated process. Pulumi has helpful integrations and documentation for many popular tools. See Continuous Delivery in the Pulumi docs for more information.
Add policy as code: By adding policy as code to our IaC pipeline, we can ensure that our infrastructure is in compliance with any security or regulatory requirements before it’s ever provisioned in the cloud, as well as checks on any existing Pulumi resources. For more information on Pulumi’s policy as code capabilities, see CrossGuard (Policy as Code) in the Pulumi docs.

Conclusion

We’ve shown how pulumi import enables organizations to quickly move from slow and error prone manual management of cloud resources to a modern, automated, code-centric approach, even when we need to preserve those manually created resources because they are running critical workloads. It’s never too late for an organization to transition to managing infrastructure resources as code, and Pulumi makes this transition as painless as possible!

Easier IaC adoption with improved `pulumi import` experience

Fraser Waters — Tue, 15 Mar 2022 00:00:00 +0000

Last year, we introduced a new Pulumi feature that allows you to import existing infrastructure into your Pulumi program. Not only did it bring the resource into the Pulumi state file, but it could generate the source code for your Pulumi program too. Today, we’re excited to announce that we’ve listened to feedback and delivered a plethora of updates and fixes to streamline the import experience; to make it more useful, more convenient, and more powerful.

At Pulumi, we understand that many cloud engineers and platform teams around the world don’t have the luxury of greenfield projects, more often than not we’re stuck with the impossible task of “refactoring” or “migrating” existing projects to more modern stacks to help increase team productivity, velocity, and stability. These projects aren’t trivial and we want to make it easier for teams and organizations to bring their infrastructure into a cloud engineering world. Oh, and worry not you lucky greenfielders … even if you just wanna ClickOps your way through some resources and import them into your program; that’s gonna work just fine too; we won’t tell if you don’t 😉.

To help you understand the changes we’ve made to help you on your journey, let’s take a look at some side-by-side examples of how pulumi import used to work vs. what we’re releasing today in Pulumi 3.26.

Example: AWS S3 Bucket

Previously when importing an S3 bucket, we would set the default values for properties like acl and forceDestroy. The new behavior will not include properties with default values in the generated code, keeping your code inline with how you’d write it yourself. Artisanal codegen for the artisanal cloud engineer.

Code Comparison

This code would have been generated by the older implementation:

const my_bucket = new aws.s3.Bucket("my-bucket", {
 acl: "private",
 bucket: "my-bucket-3f85c54",
 forceDestroy: false,
}, {
 protect: true,
});

This code will be generated by the newer implementation:

const my_bucket = new aws.s3.Bucket("my-bucket", {
 arn: "arn:aws:s3:::my-bucket-3f85c54",
 bucket: "my-bucket-3f85c54",
 hostedZoneId: "Z3BJ6K6RIION7M",
 requestPayer: "BucketOwner",
}, {
 protect: true,
});

The above accurately reflects what the providers Read function has told us of the inputs set for this bucket, although due to the way the AWS provider works, we could elide all these properties. This is because the provider will use outputs saved in the state file and the new inputs to calculate the update diff. A property that exists in the old output and is missing from the new inputs is simply left at the old output value. As none of the properties for Bucket are marked required the following works as well and is equivalent:

const my_bucket = new aws.s3.Bucket("my-bucket", { protect: true });

Example: AWS EC2 Instances

Previously, when trying to import an EC2 instance with pulumi import, you’d be presented with a list of required properties that weren’t satisfied. Like above, the new behavior will read from the provider and inherit the missing properties, providing everything required to correctly import. This provides a cleaner and more intuitive experience for the user.

Old Behavior

$ pulumi import aws:ec2/instance:Instance test i-085d780737c600c7e
aws:ec2:Instance (test):
 error: aws:ec2/instance:Instance resource 'test' has a problem: Missing required argument: "instance_type": one of `instance_type,launch_template` must be specified. Examine values at 'Instance.InstanceType'.
 error: aws:ec2/instance:Instance resource 'test' has a problem: Missing required argument: "launch_template": one of `ami,instance_type,launch_template` must be specified. Examine values at 'Instance.LaunchTemplate'.
 error: aws:ec2/instance:Instance resource 'test' has a problem: Missing required argument: "ami": one of `ami,launch_template` must be specified. Examine values at 'Instance.Ami'.

New Behavior

The import will be successful and we’ll be presented with a rich resource in your language of choice.

const test = new aws.ec2.Instance("test", {
 ami: "ami-082b5a644766e0e6f",
 associatePublicIpAddress: true,
 availabilityZone: "us-west-2c",
 capacityReservationSpecification: {
 capacityReservationPreference: "open",
 },
 cpuCoreCount: 1,
 cpuThreadsPerCore: 1,
 creditSpecification: {
 cpuCredits: "standard",
 },
 iamInstanceProfile: "",
 instanceInitiatedShutdownBehavior: "stop",
 instanceType: "t2.micro",
 metadataOptions: {
 httpEndpoint: "enabled",
 httpPutResponseHopLimit: 1,
 httpTokens: "optional",
 },
 privateIp: "172.31.0.188",
 rootBlockDevice: {
 iops: 100,
 volumeSize: 8,
 volumeType: "gp2",
 },
 securityGroups: ["default"],
 subnetId: "subnet-43f43a1e",
 tenancy: "default",
 vpcSecurityGroupIds: ["sg-4d436f12"],
}, {
 protect: true,
});

test = aws.ec2.Instance("test",
 ami="ami-082b5a644766e0e6f",
 associate_public_ip_address=True,
 availability_zone="us-west-2c",
 capacity_reservation_specification=aws.ec2.InstanceCapacityReservationSpecificationArgs(
 capacity_reservation_preference="open",
 ),
 cpu_core_count=1,
 cpu_threads_per_core=1,
 credit_specification=aws.ec2.InstanceCreditSpecificationArgs(
 cpu_credits="standard",
 ),
 iam_instance_profile="",
 instance_initiated_shutdown_behavior="stop",
 instance_type="t2.micro",
 metadata_options=aws.ec2.InstanceMetadataOptionsArgs(
 http_endpoint="enabled",
 http_put_response_hop_limit=1,
 http_tokens="optional",
 ),
 private_ip="172.31.0.188",
 root_block_device=aws.ec2.InstanceRootBlockDeviceArgs(
 iops=100,
 volume_size=8,
 volume_type="gp2",
 ),
 security_groups=["default"],
 subnet_id="subnet-43f43a1e",
 tenancy="default",
 vpc_security_group_ids=["sg-4d436f12"],
 opts=pulumi.ResourceOptions(protect=True))

 _, err := ec2.NewInstance(ctx, "test", &ec2.InstanceArgs{
 Ami: pulumi.String("ami-082b5a644766e0e6f"),
 AssociatePublicIpAddress: pulumi.Bool(true),
 AvailabilityZone: pulumi.String("us-west-2c"),
 CapacityReservationSpecification: &ec2.InstanceCapacityReservationSpecificationArgs{
 CapacityReservationPreference: pulumi.String("open"),
 },
 CpuCoreCount: pulumi.Int(1),
 CpuThreadsPerCore: pulumi.Int(1),
 CreditSpecification: &ec2.InstanceCreditSpecificationArgs{
 CpuCredits: pulumi.String("standard"),
 },
 IamInstanceProfile: pulumi.Any(""),
 InstanceInitiatedShutdownBehavior: pulumi.String("stop"),
 InstanceType: pulumi.String("t2.micro"),
 MetadataOptions: &ec2.InstanceMetadataOptionsArgs{
 HttpEndpoint: pulumi.String("enabled"),
 HttpPutResponseHopLimit: pulumi.Int(1),
 HttpTokens: pulumi.String("optional"),
 },
 PrivateIp: pulumi.String("172.31.0.188"),
 RootBlockDevice: &ec2.InstanceRootBlockDeviceArgs{
 Iops: pulumi.Int(100),
 VolumeSize: pulumi.Int(8),
 VolumeType: pulumi.String("gp2"),
 },
 SecurityGroups: pulumi.StringArray{
 pulumi.String("default"),
 },
 SubnetId: pulumi.String("subnet-43f43a1e"),
 Tenancy: pulumi.String("default"),
 VpcSecurityGroupIds: pulumi.StringArray{
 pulumi.String("sg-4d436f12"),
 },
}, pulumi.Protect(true))

var test = new Aws.Ec2.Instance("test", new Aws.Ec2.InstanceArgs
 {
 Ami = "ami-082b5a644766e0e6f",
 AssociatePublicIpAddress = true,
 AvailabilityZone = "us-west-2c",
 CapacityReservationSpecification = new Aws.Ec2.Inputs.InstanceCapacityReservationSpecificationArgs
 {
 CapacityReservationPreference = "open",
 },
 CpuCoreCount = 1,
 CpuThreadsPerCore = 1,
 CreditSpecification = new Aws.Ec2.Inputs.InstanceCreditSpecificationArgs
 {
 CpuCredits = "standard",
 },
 IamInstanceProfile = "",
 InstanceInitiatedShutdownBehavior = "stop",
 InstanceType = "t2.micro",
 MetadataOptions = new Aws.Ec2.Inputs.InstanceMetadataOptionsArgs
 {
 HttpEndpoint = "enabled",
 HttpPutResponseHopLimit = 1,
 HttpTokens = "optional",
 },
 PrivateIp = "172.31.0.188",
 RootBlockDevice = new Aws.Ec2.Inputs.InstanceRootBlockDeviceArgs
 {
 Iops = 100,
 VolumeSize = 8,
 VolumeType = "gp2",
 },
 SecurityGroups =
 {
 "default",
 },
 SubnetId = "subnet-43f43a1e",
 Tenancy = "default",
 VpcSecurityGroupIds =
 {
 "sg-4d436f12",
 },
 }, new CustomResourceOptions
 {
 Protect = true,
 });
 ```

Again this reflects what Read has told us of the inputs set for this instance, and again due to the way the AWS provider works many of these inputs could be elided and the provider would pick up the values from the saved output set. The following is again equivalent to the above given the saved state:

const test = new aws.ec2.Instance("test", {
 ami: "ami-082b5a644766e0e6f",
 instanceType: "t2.micro"
}, {
 protect: true,
});

Everything is Better, Yes?

To quote our old friend, Harlan, “Everything is better, yes?”

While we’ve only shown you a couple of examples of this new behaviour, rest assured that the benefits of it stretch far and wide across the majority of Pulumi resources.

If you’ve ever had a problem importing a resource before, we encourage you to try it again and let us know if you run into any problems.

Curious how this all works? Let’s dive in.

Technical details

We’ve never really explained how the import system works before, which has lead to a lot of confusion from users when they encountered errors with it. Given the changes we’re making here we felt it a good time to give a more detailed write up of how the engine works to try and help users understand why their import results are the way they are.

This section requires some understanding of the Pulumi architecture, but that understanding is not necessary to be able to use pulumi import.

The importance of `Provider.Read`

Our import system depends on the ability for a resource provider to be able to read the existing state of a resource and report back to Pulumi the current value of it’s inputs and outputs.

This isn’t always possible to return accurately. For example if a resource’s inputs don’t match up 1-to-1 with it’s output state and the underlying provider can only read the current output state there’s no way for it to always construct the correct input state. There could also be bugs in the provider’s Read method that result in inaccurate reads. We’ve designed the import system to be tolerant to these cases, but it does mean we don’t expect perfect import results for every resource.

The current import system

To import resources with the currently the engine issues the following steps:

Use the providers Read function to get the inputs and outputs for the resource.
Remove all properties from the inputs except for those flagged as required.
Pass that reduced input property set to the providers Check method.
If there are any check failures return an error and fail the import.
Take the result of Check which will have filled in default properties and diff it with the original input set returned from Read.
If there are any diffs copy in the values from the original Read to the resulting input set.
Save the updated input set and the original output set into the stacks state.
Pass the resulting input set to the code generator to generate a new Resource call to match the imported state.

There are a number of problems with the above. The biggest is trying to strip the input set down to only properties that are marked as required. The required flag is a fairly blunt tool that can’t capture that some fields are only required sometimes based on the value of other fields. As such this stripping to just required properties causes many of the following calls to Check to raise check failures.

The next major issue is that we hard fail if check returns failures. There is always the risk of this happening due to limitations of Read, and it doesn’t help the user just throwing an error.

Finally in the cases where the above two issues weren’t hit this data flow had the odd result of setting optional fields to their default values. This isn’t needed and these fields would be better left as blank.

The new import system aims to improve the described issues.

The new import system

The engine will now issue the following steps to import resources:

Use the providers Read function to get the inputs and outputs for the resource.
Pass the input set to the providers Check method.
Warn if there are any check failures.
Save the input and output set from Read into the stacks state.
Pass the input set to the code generator to generate a new Resource call to match the imported state.

Note that we no longer end up with filled in default values in our final result, and we carry on in the face of check failures and so can still generate some code. If providers can give complete and accurate results from Read the above flow will result in correctly generated code. However it doesn’t result in quite the correct stack state, this is due to how the data flow of our import system currently works. Currently we generate code based on the value of inputs in the state file, however normally the value stored at inputs in the state file is the result from Check, not what is directly in the users program. As such when you run the first pulumi up after an import your state file will change slightly as the engine calls Check and saves the inputs with defaults filled in. This should normally be transparent to the user, but there may be some cases where providers think this is a trivial diff.

There are two complications with fixing the above. The first is that during import if Read doesn’t return valid results then Check can fail and so we won’t have a property set returned from it that we can save to the state file. This case will always have to fall back to just storing the properties returned by Read and triggering a check failure during up that will need to be manually resolved. The second complication is that the code generation for import works off the state file rather than a separate data flow specifically for imported resources. We plan on improving this but in the spirit of agility we felt that a slightly better import feature today would be more useful to our users than a perfect import feature later.

Terraform `Read`

Many of Pulumi’s resource providers make use of Terraform providers. We call these bridged providers and all of them use our “terraform-bridge” library to translate between the Terraform API and Pulumi API.

Terraform currently has a slightly odd behavior that missing values for some properties are zero initialized by its Refresh method (what we use to implement Pulumi’s Read method in the terraform bridge). This has the result that when Pulumi asks to read a given Terraform resource some top level properties (such as S3 buckets acl property) come back as missing, while other top level properties and all nested properties (such as S3 buckets versioning.mfaDelete) come back with some zero value.

For example reading an AWS S3 bucket object with the current Terraform bridge results in the following property set:

{
"__defaults": [],
"accelerationStatus": "",
"arn": "arn:aws:s3:::my-bucket-3f85c54",
"bucket": "my-bucket-3f85c54",
"corsRules": [],
"grants": [],
"hostedZoneId": "Z3BJ6K6RIION7M",
"lifecycleRules": [],
"loggings": [],
"objectLockConfiguration": null,
"replicationConfiguration": null,
"requestPayer": "BucketOwner",
"serverSideEncryptionConfiguration": null,
"tags": {
"__defaults": []
},
"versioning": {
"__defaults": [],
"enabled": false,
"mfaDelete": false
},
"website": null
}

We’ve made a change to the Terraform bridge to try and elide these zero properties that should just be missing. Reading the same S3 bucket with these changes gives the following much smaller property set:

{
"__defaults": [],
"arn": "arn:aws:s3:::my-bucket-3f85c54",
"bucket": "my-bucket-3f85c54",
"hostedZoneId": "Z3BJ6K6RIION7M",
"requestPayer": "BucketOwner",
"website": null
}

We’ll continue to work on the Terraform bridge to give better results here. For example the website property on the bucket should probably be missing rather than null.

Feedback

We’re aware this could have a large impact to some of our users so we wanted to make sure we explained these changes well. If you have questions or concerns about this change let us know in our community Slack or in GitHub Discussions.

Pulumi Import: Generate IaC for Existing Cloud Resources

Paul Stack — Tue, 24 Nov 2020 00:00:00 +0000

Most infrastructure projects require working with existing cloud resources, either by building on top of existing resources or adopting existing resources under management with a new and more robust infrastructure provisioning solution.

In June 2019, Pulumi introduced the ability to import existing infrastructure resources to be under Pulumi management no matter how you’ve provisioned these resources — manually in your cloud provider’s console or CLI, using an infrastructure as code tool like Terraform or AWS CloudFormation. Today, we are happy to announce a richer resource import experience.

As of v2.12.0, Pulumi has introduced a pulumi import command. This command will import the cloud resource into the Pulumi state and generate the code for the user’s Pulumi program in the appropriate language.

The Anatomy of the Import Command

No matter how resources have been provisioned in the cloud, Pulumi enables you to adopt and manage those resources as part of your infrastructure project. The pulumi import command is made of the following:

pulumi import [type] [name] [id]

When Pulumi performs an import, the resource is added to the Pulumi state, it is marked as a protected resource (by default), and it will emit the generated code that the user can add to their program before running a pulumi up. The resources are marked as protected to ensure that imported infrastructure is not accidentally deleted if the user forgets to include the code for the resource in their program before doing a deployment.

Importing an S3 Bucket

A resource needs to be imported into an existing Pulumi project. Let’s consider that a user wants to import an S3 bucket manually created in their AWS Account. To do this, we can execute the following command:

pulumi import aws:s3/bucket:Bucket infra-logs company-infra-logs

This command instructs Pulumi to import the S3 bucket called company-infra-logs with the Pulumi resource name infra-logs. The import command will output as follows:

The user can then see the generated code they need to use as part of their application. I imported this resource into a TypeScript application:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

const demo = new aws.s3.Bucket("infra-logs", {
 acl: "private",
 bucket: "company-infra-logs",
 forceDestroy: false,
}, {
 protect: true,
});

import pulumi
import pulumi_aws as aws

demo = aws.s3.Bucket("infra-logs",
 acl="private",
 bucket="company-infra-logs",
 force_destroy=False,
 opts=ResourceOptions(protect=True))

package main

import (
 "github.com/pulumi/pulumi-aws/sdk/v3/go/aws/s3"
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 _, err := s3.NewBucket(ctx, "infra-logs", &s3.BucketArgs{
 Acl: pulumi.String("private"),
 Bucket: pulumi.String("company-infra-logs"),
 ForceDestroy: pulumi.Bool(false),
 }, pulumi.Protect(true))
 if err != nil {
 return err
 }
 return nil
 })
}

using Pulumi;
using Aws = Pulumi.Aws;

class MyStack : Stack
{
 public MyStack()
 {
 var demo = new Aws.S3.Bucket("infra-logs", new Aws.S3.BucketArgs
 {
 Acl = "private",
 Bucket = "company-infra-logs",
 ForceDestroy = false,
 }, new CustomResourceOptions
 {
 Protect = true,
 });
 }

}

This is valid Pulumi application code and can be added to your Pulumi application. Notice that the Pulumi resource is marked as protect: true. This means Pulumi will not delete that resource unless this protection attribute is removed from the resource. If this code is not added to your Pulumi application, then a Pulumi update will return the following:

Diagnostics:
 pulumi:pulumi:Stack (import-example-dev):
 error: preview failed

 aws:s3:Bucket (demo):
 error: Preview failed: refusing to delete protected resource 'urn:pulumi:dev::import-example::aws:s3/bucket:Bucket::infra-logs'

Importing Multiple Resources

Should a user need to import multiple resources to be managed by Pulumi, the CLI import command can handle this scenario for us. This is helpful when using the pulumi import command as part of scripting larger bulk imports of cloud resource:

{
 "resources": [{
 "type": "aws:ec2/vpc:Vpc",
 "name": "application-vpc",
 "id": "vpc-0ad77710973388316"
 },
 {
 "type": "aws:ec2/subnet:Subnet",
 "name": "public-1",
 "id": "subnet-0fb5fdff92b9e5a3b"
 },
 {
 "type": "aws:ec2/subnet:Subnet",
 "name": "private-1",
 "id": "subnet-0a39d25dd9f7b7808"
 }
 ]
}

We can then run the command:

pulumi import -f resources.json

We can see the resultant code generation:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

const application_vpc = new aws.ec2.Vpc("application-vpc", {
 assignGeneratedIpv6CidrBlock: false,
 cidrBlock: "172.16.0.0/16",
 enableDnsSupport: true,
 instanceTenancy: "default",
 tags: {
 Name: "pulumi-vpc",
 Owner: "pulumi",
 Project: "pulumi-k8s-aws-cluster",
 },
}, {
 protect: true,
});
const public_1 = new aws.ec2.Subnet("public-1", {
 assignIpv6AddressOnCreation: false,
 cidrBlock: "172.16.32.0/19",
 mapPublicIpOnLaunch: true,
 tags: {
 Name: "pulumi-vpc-public-1",
 Owner: "pulumi",
 Project: "pulumi-k8s-aws-cluster",
 "kubernetes.io/role/elb": "1",
 type: "public",
 },
 vpcId: "vpc-0ad77710973388316",
}, {
 protect: true,
});
const private_1 = new aws.ec2.Subnet("private-1", {
 assignIpv6AddressOnCreation: false,
 cidrBlock: "172.16.160.0/19",
 mapPublicIpOnLaunch: false,
 tags: {
 Name: "pulumi-vpc-private-1",
 Owner: "pulumi",
 Project: "pulumi-k8s-aws-cluster",
 "kubernetes.io/role/internal-elb": "1",
 type: "private",
 },
 vpcId: "vpc-0ad77710973388316",
}, {
 protect: true,
});

import pulumi
import pulumi_aws as aws

application_vpc = aws.ec2.Vpc("application-vpc",
 assign_generated_ipv6_cidr_block=False,
 cidr_block="172.16.0.0/16",
 enable_dns_support=True,
 instance_tenancy="default",
 tags={
 "Name": "pulumi-vpc",
 "Owner": "pulumi",
 "Project": "pulumi-k8s-aws-cluster",
 },
 opts=ResourceOptions(protect=True))
public_1 = aws.ec2.Subnet("public-1",
 assign_ipv6_address_on_creation=False,
 cidr_block="172.16.32.0/19",
 map_public_ip_on_launch=True,
 tags={
 "Name": "pulumi-vpc-public-1",
 "Owner": "pulumi",
 "Project": "pulumi-k8s-aws-cluster",
 "kubernetes.io/role/elb": "1",
 "type": "public",
 },
 vpc_id="vpc-0ad77710973388316",
 opts=ResourceOptions(protect=True))
private_1 = aws.ec2.Subnet("private-1",
 assign_ipv6_address_on_creation=False,
 cidr_block="172.16.160.0/19",
 map_public_ip_on_launch=False,
 tags={
 "Name": "pulumi-vpc-private-1",
 "Owner": "pulumi",
 "Project": "pulumi-k8s-aws-cluster",
 "kubernetes.io/role/internal-elb": "1",
 "type": "private",
 },
 vpc_id="vpc-0ad77710973388316",
 opts=ResourceOptions(protect=True))

package main

import (
 "github.com/pulumi/pulumi-aws/sdk/v3/go/aws/ec2"
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 _, err := ec2.NewVpc(ctx, "application_vpc", &ec2.VpcArgs{
 AssignGeneratedIpv6CidrBlock: pulumi.Bool(false),
 CidrBlock: pulumi.String("172.16.0.0/16"),
 EnableDnsSupport: pulumi.Bool(true),
 InstanceTenancy: pulumi.String("default"),
 Tags: pulumi.StringMap{
 "Name": pulumi.String("pulumi-vpc"),
 "Owner": pulumi.String("pulumi"),
 "Project": pulumi.String("pulumi-k8s-aws-cluster"),
 },
 }, pulumi.Protect(true))
 if err != nil {
 return err
 }
 _, err = ec2.NewSubnet(ctx, "public_1", &ec2.SubnetArgs{
 AssignIpv6AddressOnCreation: pulumi.Bool(false),
 CidrBlock: pulumi.String("172.16.32.0/19"),
 MapPublicIpOnLaunch: pulumi.Bool(true),
 Tags: pulumi.StringMap{
 "Name": pulumi.String("pulumi-vpc-public-1"),
 "Owner": pulumi.String("pulumi"),
 "Project": pulumi.String("pulumi-k8s-aws-cluster"),
 "kubernetes.io/role/elb": pulumi.String("1"),
 "type": pulumi.String("public"),
 },
 VpcId: pulumi.String("vpc-0ad77710973388316"),
 }, pulumi.Protect(true))
 if err != nil {
 return err
 }
 _, err = ec2.NewSubnet(ctx, "private_1", &ec2.SubnetArgs{
 AssignIpv6AddressOnCreation: pulumi.Bool(false),
 CidrBlock: pulumi.String("172.16.160.0/19"),
 MapPublicIpOnLaunch: pulumi.Bool(false),
 Tags: pulumi.StringMap{
 "Name": pulumi.String("pulumi-vpc-private-1"),
 "Owner": pulumi.String("pulumi"),
 "Project": pulumi.String("pulumi-k8s-aws-cluster"),
 "kubernetes.io/role/internal-elb": pulumi.String("1"),
 "type": pulumi.String("private"),
 },
 VpcId: pulumi.String("vpc-0ad77710973388316"),
 }, pulumi.Protect(true))
 if err != nil {
 return err
 }
 return nil
 })
}

using Pulumi;
using Aws = Pulumi.Aws;

class MyStack : Stack
{
 public MyStack()
 {
 var application_vpc = new Aws.Ec2.Vpc("application-vpc", new Aws.Ec2.VpcArgs
 {
 AssignGeneratedIpv6CidrBlock = false,
 CidrBlock = "172.16.0.0/16",
 EnableDnsSupport = true,
 InstanceTenancy = "default",
 Tags =
 {
 { "Name", "pulumi-vpc" },
 { "Owner", "pulumi" },
 { "Project", "pulumi-k8s-aws-cluster" },
 },
 }, new CustomResourceOptions
 {
 Protect = true,
 });
 var public_1 = new Aws.Ec2.Subnet("public-1", new Aws.Ec2.SubnetArgs
 {
 AssignIpv6AddressOnCreation = false,
 CidrBlock = "172.16.32.0/19",
 MapPublicIpOnLaunch = true,
 Tags =
 {
 { "Name", "pulumi-vpc-public-1" },
 { "Owner", "pulumi" },
 { "Project", "pulumi-k8s-aws-cluster" },
 { "kubernetes.io/role/elb", "1" },
 { "type", "public" },
 },
 VpcId = "vpc-0ad77710973388316",
 }, new CustomResourceOptions
 {
 Protect = true,
 });
 var private_1 = new Aws.Ec2.Subnet("private-1", new Aws.Ec2.SubnetArgs
 {
 AssignIpv6AddressOnCreation = false,
 CidrBlock = "172.16.160.0/19",
 MapPublicIpOnLaunch = false,
 Tags =
 {
 { "Name", "pulumi-vpc-private-1" },
 { "Owner", "pulumi" },
 { "Project", "pulumi-k8s-aws-cluster" },
 { "kubernetes.io/role/internal-elb", "1" },
 { "type", "private" },
 },
 VpcId = "vpc-0ad77710973388316",
 }, new CustomResourceOptions
 {
 Protect = true,
 });
 }

}

Check out the video clip below for a demo.

Documentation

Full CLI Documentation is available, showing all of the different import command flags. The flags include writing directly to a file and the ability to ensure a resource is not protected on import. Examples of importing resources can be found on their specific resource documentation pages. Such an example for importing a VPC can be found in the VPC import documentation.

Pulumi Blog: Import

Migrate CloudFormation to Pulumi with Discovered Stacks

What we’re migrating

Step 1: Find your stack in Pulumi Cloud

Step 2: Plan the migration at a glance

Step 3: Start the migration

Step 4: Resolve the stragglers

Step 5: The quality gate

Where you end up

Go from Cloud to Code in Minutes with Visual Import

Modern Infrastructure, Codified Faster

How It Works

Key Benefits

Accelerated Migration

Increased Developer Productivity

Consistent, Reliable Infrastructure

From Insights to IaC—Seamlessly

Leverage the Full Power of Pulumi Cloud

Try It Today

Pulumi adoption made easy with the new Migration Hub

Why Pulumi?

Migration

Self-Serve Migration

Let Us Help You Out with Expert Services!

Success Stories

Special Offers

Calling All Partners!

Get Started Today

Google Cloud: Bulk Importing Resources into Pulumi

Writing an account scraper

Config Connector

Mapping from Kubernetes manifests to pulumi import JSON

Mapping resources with custom ID requirements

Importing to Pulumi

Next Steps

Conclusion

Automating Pulumi Import with Manually Created Resources

About Pulumi Import

Automated Batch Import Capabilities

Designing an End-to-End Solution

Writing an Account Scraper

Batch Import

Next Steps

Conclusion

Easier IaC adoption with improved `pulumi import` experience

Example: AWS S3 Bucket

Code Comparison

Example: AWS EC2 Instances

Old Behavior

New Behavior

Everything is Better, Yes?

Technical details

The importance of Provider.Read

The current import system

The new import system

Terraform Read

Feedback

Pulumi Import: Generate IaC for Existing Cloud Resources

The Anatomy of the Import Command

Importing an S3 Bucket

Importing Multiple Resources

Documentation

Mapping from Kubernetes manifests to `pulumi import` JSON

The importance of `Provider.Read`

Terraform `Read`