Pulumi Blog: Development environment

Using Pulumi Inside Node.js Monorepos

Julien Poissonnier — Tue, 26 Mar 2024 17:19:00 +0100

One of Pulumi’s core goals is to provide cloud engineers with access to the very best software engineering tooling available. Using traditional programming languages like Node.js, Python, Go, .NET and Java means the latest and greatest software engineering tools from each of these ecosystems is available to bring to bear on managing cloud infrastructure, natively integrated with your existing development environments.

In the Node.js ecosystem, we’ve seen an explosion of great tooling over the last couple of years around support for monorepos - larger repositories built out of many smaller projects, and sharing code and dependencies smartly across all the various projects. We’ve seen many of our Pulumi Node.js users adopting these tools and repo structures, including tools like Yarn Workspaces, pnpm, Turborepo, and especially Nx.

While it has always been possible to apply these tools to Pulumi Node.js projects in TypeScript or JavaScript just like any other Node.js project, we’ve recently made a number of enhancements and fixes to make sure that Pulumi works truly seamlessly with these tools.

In this post, we’ll show how you can build a seamless development workflow by integrating Pulumi code level abstractions, such as Component Resources, with a monorepo-based build system like Nx.

Component resources let us create reusable components that manage logical groupings of resources. With modern monorepo tooling, and our recent improvements to make Pulumi aware of monorepo setups, we can colocate these reusable components, Pulumi infrastructure programs, and application code all in one monorepo, and have Nx manage the build and deploy-time dependencies for us.

We will walk through an example project that deploys a website built with Astro to AWS S3. The complete code can be found at pulumi/examples/nx-monorepo.

Our example monorepo has the following structure:

website: A website built with Astro.
components/s3folder: A component resource that manages a S3 bucket and its access policies.
components/website-deploy: A component resource resource that manages files in a S3 bucket
infra: A Pulumi program that uses the s3folder and website-deploy component resources to deploy the generated website.

By using npm workspaces we can have multiple npm packages managed from a singular top-level package. Npm will take care of installing the dependencies for all our packages and enables packages within the monorepo to reference each other. In the package.json at the root of the monorepo we set up the workspaces matching our folder structure. Yarn workspaces work similarly and are fully supported by Pulumi.

// package.json
{
 ...
 "workspaces": [
 "components/*",
 "infra",
 "website"
 ]
}

TypeScript

Pulumi has builtin TypeScript support and compiles your code on the fly without manual build-step, however this is currently limited to TypeScript 3.8. We are working on providing more choice here, but in the mean time Nx makes it easy to add a build-step to compile code using any version of TypeScript. For this example we are using the latest and greatest, TypeScript 5.4.

As of v3.113.0 of the @pulumi/pulumi NPM package any version of TypeScript is supported natively. To select the version to use, add it as a dependency to your package.json.

Declaring Dependencies

If we were to manually build and deploy the code in our monorepo, we would have to run the following steps:

Run astro build to generate the HTML output that we want to deploy.
Run npm run build for s3folder and website-deploy.
Run npm run build for infra afterwards, since this imports s3folder and website-deploy.
Run pulumi up to deploy our website.

Instead of running these steps manually, we will leverage Nx’s task running mechanism, which can be made aware of the dependencies between the packages in the monorepo. This ensures we don’t accidentally forget a step. On top of that, Nx provides caching, so we also avoid re-running unnecessary tasks on subsequent runs.

Because we are using npm workspaces, Nx can understand the dependencies declared in the package.json for each of the packages in our monorepo. For example in infra/package.json we declare that we depend on s3folder, website and website-deploy. We also let Nx know that the deploy target in the infra package depends on the build step, and the HTML generation step of the website package:

// infra/package.json
{
 "name": "infra",
 "dependencies": {
 "@pulumi/pulumi": "latest",
 "s3folder": "*",
 "website-deploy": "*",
 "website": "*"
 },
 "scripts": {
 "build": "tsc",
 "deploy": "pulumi up --stack dev"
 },
 "nx": {
 "targets": {
 "deploy": {
 "cache": true,
 "dependsOn": [
 "build",
 "website:generate"
 ]
 }
 }
 }
}

In nx.json we let Nx know that each package’s build-step depends on its dependencies’ build-steps. With that Nx now knows that in order build infra, it first needs to build its dependencies.

// nx.json
{
 "extends": "nx/presets/npm.json",
 "$schema": "./node_modules/nx/schemas/nx-schema.json",
 "targetDefaults": {
 "build": {
 "cache": true,
 "dependsOn": [
 "^build"
 ]
 }
 }
}

The full dependency graph, as understood by Nx, can be visualized by running npx nx deploy infra --graph.

Walkthrough

Now that we have taught Nx all about the dependencies in our monorepo, we can run npx nx deploy infra and Nx will run all the required steps in the correct order, using parallelism where possible.

 ✔ 4/4 dependent project tasks succeeded [0 read from cache]
Hint: you can run the command with --verbose to see the full dependent project outputs
————————————————————————————————————————————————————————————————————————————————
> nx run infra:build
> infra@1.0.0 build
> tsc
> nx run infra:deploy
> infra@1.0.0 deploy
> pulumi up --stack dev
The stack 'dev' does not exist.
If you would like to create this stack now, please press <ENTER>, otherwise press ^C:
Created stack 'dev'
Previewing update (dev)
View in Browser (Ctrl+O): https://app.pulumi.com/julienp/nx-monorepo/dev/previews/fc7630fd-7dc4-4c7e-baa0-3d6e014fc90a
Type Name Plan
+ pulumi:pulumi:Stack nx-monorepo-dev create
+ ├─ pulumi:examples:WebsiteDeploy my-website create
+ │ └─ aws:s3:BucketObject index.html create
+ └─ pulumi:examples:S3Folder my-folder create
+ ├─ aws:s3:Bucket my-folder create
+ ├─ aws:s3:BucketPublicAccessBlock public-access-block create
+ └─ aws:s3:BucketPolicy bucketPolicy create
Outputs:
websiteUrl: output<string>
Resources:
+ 7 to create
Do you want to perform this update? yes
Updating (dev)
...
NX Successfully ran target deploy for project infra and 4 tasks it depends on (27s)

The output tells us that Nx found the 4 dependent tasks for the deploy task of the infra package and ran them successfully.

If we now run npx nx deploy infra again, we can see Nx’s caching mechanism in action. Nx notices that there are no code changes to the infra package, and that none of its dependencies have changed either, so it just replays the output from the previous run.

 ✔ 3/3 dependent project tasks succeeded [3 read from cache]
Hint: you can run the command with --verbose to see the full dependent project outputs
————————————————————————————————————————————————————————————————————————————————
>nx run infra:build [existing outputs match the cache, left as is]
> infra@1.0.0 build
> tsc
> nx run infra:deploy [existing outputs match the cache, left as is]
> infra@1.0.0 deploy
> pulumi up --stack dev
Previewing update (dev)
...
NX Successfully ran target deploy for project infra and 4 tasks it depends on (29ms)
Nx read the output from the cache instead of running the command for 5 out of 5 tasks.

Now let’s create a new page in our website by adding a markdown file in the src/pages folder.

echo "Hello, World!" > website/src/pages/hello.md

When we run npx nx deploy infra again, Nx notices that there is a change to the website and reruns the HTML generation step. Since one of the dependencies of infra has now changed, the stack is re-deployed.

 ✔ 2/2 dependent project tasks succeeded [2 read from cache]
Hint: you can run the command with --verbose to see the full dependent project outputs
————————————————————————————————————————————————————————————————————————————————
> nx run infra:build
> infra@1.0.0 build
> tsc
> nx run infra:deploy
> infra@1.0.0 deploy
> pulumi up --stack dev
Previewing update (dev)
View in Browser (Ctrl+O): https://app.pulumi.com/v-julien-pulumi-corp/nx-monorepo/dev/previews/8d9db825-77e1-4d1a-8f9b-8279481e7e40
Type Name Plan
pulumi:pulumi:Stack nx-monorepo-dev
└─ pulumi:examples:WebsiteDeploy my-website
+ └─ aws:s3:BucketObject hello/index.html create
Resources:
+ 1 to create
8 unchanged
Do you want to perform this update? yes
Updating (dev)
...
NX Successfully ran target deploy for project infra and 4 tasks it depends on (19s)
Nx read the output from the cache instead of running the command for 2 out of 5 tasks.

We can see that Nx used the cached results for 2 of our tasks, s3folder and website-deploy didn’t change, so Nx skips rebuilding them. Nx also ran the other 3 tasks to generate the website’s HTML, building the infra package and finally deploying the infra package.

Ship infrastructure from your monorepo

Colocate reusable Pulumi component resources, infrastructure programs, and application code in one monorepo, then build and deploy them together with Nx. Manage every stack in Pulumi Cloud.

Get started

Conclusion

In this post we’ve shown how we can combine Pulumi’s code level abstractions with a build system to create a seamless developer workflow. By using a monorepo we can colocate our reusable components with the Pulumi programs that use the components, as well as our application code. Tools like Nx allow us to intelligently manage the dependencies between the packages in the monorepo, ensuring we are always deploying the correct version of the code.

Pulumi provides you with access to the latest and greatest software engineering tools (like Nx), and allows you to bring them to bear on managing cloud infrastructure, natively integrated with your existing development environments.

Organizational Patterns: A Developer Portal

Matty Stratton — Thu, 30 Dec 2021 11:00:39 -0600

Using Pulumi is more than just writing code and components. In addition to common software development practices, there are also a number of success patterns related to how your company or team builds and deploys Pulumi programs to successfully build, deploy, and manage your infrastructure and applications. In this continuation of a series, I will explore one of these patterns - using the Pulumi Automation API to create a developer portal.

It is becoming more and more common to consider building out centralized web-based “portals” for developers (and other team members) to provision the infrastructure that their services and applications require. In many ways, the pattern of a developer portal takes portions of both the Single Infra Repo and Automation Team patterns covered in previous posts. The difference is that the developer portal is a single, centralized web application that can be used by multiple teams and organizations to provision infrastructure and applications, rather than writing the Pulumi programs themselves.

Depending upon the needs of the various teams, the developer portal could include options to deploy virtual machines into the cloud of their choice (or the organization’s standard cloud; perhaps the choice isn’t…a choice!), to provision cloud resources such as virtual networks, security groups, database instances, or even deployments to a kubernetes cluster.

Generally speaking, the developer portal is created and maintained by a central platform team. This is the team who knows the success patterns/standards that are employed by the organization, and can implement these patterns in the developer portal. The platform team is also responsible for maintaining the infrastructure that the developer portal requires.

One of the great advantages of providing a single portal for provisioning these resources is that all the “other stuff” that is often required (besides the resources from the hyperscaler) can be handled by the developer portal. For example, if the portal is used to provision a virtual machine, the code behind it can register the virtual machine with the company’s infrastructure management system.

A great example of what is possible in a portal can be seen with Komal Ali’s example, the Self Service Infrastructure Platyform. This is example code only, but it shows some of the capabilities of a developer portal.

In the example, the web portal is created using a combination of Python, Flask and Jinja templates (along with the Pulumi Automation API) to provide push-button access to deploy cloud resources according to the standards and patterns of the organization. The code snippets below illustrate some of the concepts (this code is not meant to be run, but rather to illustrate the concepts - the full example code can be found at github.com/komalali/self-service-platyform).

from flask import (Blueprint, current_app, request, flash,
 redirect, url_for, render_template)

import pulumi
import pulumi_aws as aws
import pulumi.automation as auto
import os
from pathlib import Path

bp = Blueprint("virtual_machines", __name__, url_prefix="/vms")
instance_types = ['c5.xlarge', 'p2.xlarge', 'p3.2xlarge']

In the initial snippet above, we are setting up what we need for the Flask application, but the key items to make note of are including the pulumi_aws and pulumi.automation packages, which will be used for the cloud specific resources and automation.


def create_pulumi_program(keydata: str, instance_type=str):
 ami = aws.ec2.get_ami(most_recent=True,
 owners=["amazon"],
 filters=[aws.GetAmiFilterArgs(name="name", values=["*amzn2-ami-minimal-hvm*"])])

 group = aws.ec2.SecurityGroup('web-secgrp',
 description='Enable SSH access',
 ingress=[aws.ec2.SecurityGroupIngressArgs(
 protocol='tcp',
 from_port=22,
 to_port=22,
 cidr_blocks=['0.0.0.0/0'],
 )])

Continuing on, we define the Pulumi program, and set up some of the standard settings that we consider our defaults and that are not selectable by the user.


 server = aws.ec2.Instance('dlami-server',
 instance_type=instance_type,
 vpc_security_group_ids=[group.id],
 key_name=keypair.id,
 ami=ami.id)

 pulumi.export('public_dns', server.public_dns)

This final snippet is what creates the actual instance, and it exports the public DNS of the instance so that it can be accessed by the user and presented in the portal.

Conclusion

This pattern has been implemented and deployed at many larger enterprises, where the platform team has the resources to dedicate to managing the portal as an actual product. This is a key part to remember - the developer portal is a product whose consumers are the developers and other teams in the organization, and it needs to be managed as such!

Don’t forget to check out the previous two posts in this series, Organizational Patterns - A Single Infra Repo and Organizational Patterns - An Automation Team".

Organizational Patterns: An Automation Team

Matty Stratton — Wed, 22 Dec 2021 11:49:08 -0600

I want to approach this pattern with a bit of caution - there are patterns and anti-patterns related to having a single central team focused on automation.

The anti-pattern is to have an “automation team” within your organization who is tasked with creating all infrastructure as code for the entire organization. This is a common pattern, but it is generally rife with issues.

Understanding how to build and deploy a service or application requires a certain amount of domain knowledge of that service. If you have a single set of people across your entire company who need to write Pulumi programs for each and every application or service, you are now expecting that small group to not only be experts in Pulumi, but also every bit of infrastructure and software applications (both third party and created internally) across your organization.

A more successful pattern is to have the automation team act as enablers and providers for the other groups within the company. This is a team of subject matter experts in using Pulumi, and not only do they have the knowledge to build and deploy infrastructure, but they are familiar with various patterns and good practices for how to build and deploy infrastructure.

There are different mechanisms for how this automation team can enable others. One way is to embed them temporarily within a product team to work alongside them to build their automation. Then, when the product team is done, the automation team member can go and help another squad.

Depending upon the size and skills of the company and this group, they may also be responsible for creating training materials for the other groups. These are usually based upon existing Pulumi training materials, but customized for the specific needs and culture of the company.

Reusable components

This central automation team also often provides reusable components that might be shared across multiple development teams. This helps the individual teams to not have to “reinvent the wheel” and to bring in these components for their own particular needs.

As a simple example, a component could be created (and published as a library to be consumed by product teams) that defines a Kubernetes deployment.

import * as deploy from "@mycorp/deploy";

const kuard = new deploy.Deployment("kuard", {
 image: "gcr.io/kuard-project/kuard:latest",
 replicas: 1,
});

What’s happened here in our (incomplete!) example is that the published package only exposes the parts that the team needs to change; the rest is provided by the automation team.

The application team don’t need to duplicate values (such as ports) because the abstractions cater for them. The application team also didn’t need to apply any metadata, health checks, and environment variables because it can be inferred through convention, increasing velocity across the organization. When values are provided by the application team, they override convention and they can still iterate and experiment, providing feedback to the automation team.

Final tips

One of the more important recommendations for this pattern is to avoid trying to create a “mandate” for all teams for adoption. The automation team provides capabilities, such as templated CI/CD pipelines, reusable components, and training materials, but it is still up to the individual teams to decide how to use them.

Conclusion

This is a pattern that works well at a larger organization, where it would be beyond the scope of a single group to be able to wrap their understanding around hundreds of services. However, it is a pattern that can be applied to smaller organizations as well!

Watch for the next post in this series, where we will dig into some other patterns! Or revisit the first post, Organizational Patterns - A Single Infra Repo.

Organizational Patterns: A Single Infra Repo

Matty Stratton — Fri, 17 Dec 2021 05:46:04 -0600

Using Pulumi is more than just writing code and components. In addition to common software development practices, there are also a number of success patterns related to how your company or team builds and deploys Pulumi programs to successfully build, deploy, and manage your infrastructure and applications. In this first post of a series, I will explore one of these patterns - the centralized platform infrastructure repository.

An emergent organizational pattern these days is that of a centralized “platform” team, which has various product and service teams (or squads) as internal customers. In this approach, the platform team takes responsibility for the tooling and infrastructure - if it’s not directly product related, it usually falls under the responsibility of the platform team. The platform team provides functionality and platforms to be consumed by the product teams.

For many of the examples I’ll be using to illustrate this pattern, I refer to a conversation I had recently with Jacob Foard, who is the Tech Lead for the Platform Team at GreenPark Sports. This pattern is used at GreenPark Sports, and he was very clear about the benefits of the pattern.

One of the key concepts to keep in mind is that when providing a platform, it is made up of more than just the compute and other resources provided by AWS, GCP, Azure, or even your own Kubernetes implementations. The platform also includes the infrastructure that is shared between the various teams, such as monitoring and observability tooling, version control/pipeline services, as well as secret and key management.

In this pattern, your main infrastructure repository is made up of directories for each product/service that your teams use, in addition to directories for each higher level shared service. Each of these directories is itself a Pulumi program. So it would look something like this:

├── bluth-apps
│   ├── apps
│   │   ├── apps.go
│   │   ├── bananastand.go
│   │   ├── suddenvalley.go
│   ├── main.go
│   ├── Pulumi.dev.yaml
│   └── Pulumi.prod.yaml
├── datadog
│   ├── main.go
│   └── Pulumi.prod.yaml
├── github
│   ├── main.go
│   └── Pulumi.prod.yaml
├── pkg
│   ├── datadog
│   ├── pagerduty
│   └── vault
└── .etc

In the above (fictional, but inspired by the GreenPark Sports pattern) example, the Bluth Company has two main services that are used in all of its environments (“Banana Stand” and “Sudden Valley”). The main apps.go file is the entry point that simply calls functions from each of the various apps to “set up” those apps, as well as the common infrastructure that an environment might require (networks, storage, etc). Note that the way you structure your code is up to you, and likely will vary depending upon the runtime for your particular Pulumi program, but this is the general idea.

Similarly, the github and datadog directories are Pulumi programs that are responsible for the “core” infrastructure for those services (perhaps creating roles, etc). The pkg directory is a directory that contains packages that are used by the other programs to implement that infrastructure. Again, the pkg convention is used by Go, but other runtimes will have a similar approach.

Examples

These examples are not complete runnable code, but used to illustrate the pattern. While these examples are using Go, they are written in a way that is compatible with any language that supports the Pulumi language.

main.go

package main

import (
 "github.com/bluthcompany/infra/bluth-apps/apps"
 "github.com/bluthcompany/infra/pkg/datadog"
 "github.com/bluthcompany/infra/pkg/github"
 "github.com/pulumi/pulumi/sdk/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // Create the apps.
 apps.SetupApps(ctx, bananastand, suddenvalley)

 // Create the infrastructure.
 datadog.CreateInfrastructure(ctx)
 github.CreateInfrastructure(ctx)

 return nil
 })
}

apps.go

package apps
import (
 "github.com/pulumi/pulumi/sdk/go/pulumi"
)

func SetupApps(
 ctx *pulumi.Context,

 setupBananaStand(ctx)
 setupSuddenValley(ctx)

)

bananastand.go

package apps
// imports, etc
func setupBananaStand(ctx *pulumi.Context) {
 // Create the banana stand.
}

Setting up a new service in the platform repository

If a service/product team has a new service they want infrastructure for, they simply add a new myapp.go file to the apps directory for their service, and add it to the apps.go file to make sure it is called. This is then submitted as a pull request for the platform team to review.

One important part of this pattern is that the platform team does not want to be a “blocker” for the product and service teams. It’s key to make sure that you have more than one person able to review and merge these pull requests, and to add sufficient testing into your CI/CD pipeline for this infrastructure repository.

Variations on this pattern

In the fictional Bluth example, there is one Pulumi program that is used regardless of environment, and the different configurations are handled by the use of stacks. However, there are situations where you might have complex enough differences between your environments where the amount of conditionals you require in your code to handle this would make for very challenging maintenance and understanding of the code! This is the case with GreenPark Sports, so in their implementation, instead of a single bluth-apps directory at the root of the repo, you would instead have bluth-prod, bluth-dev, etc.

This approach does generate duplication of code, and it can provide challenges at scale, but it is up to you and your teams to determine the tradeoffs of the branching/conditional logic vs separate programs.

Conclusion

This pattern works well depending upon the makeup of your teams and services. It is a pattern that facilitates collaboration between teams and focuses on having a central platform team that enables product teams, rather than getting in their way.

Watch for the next posts in this series, where we will dig into some other patterns as well!

Building a Development Environment for Cloud Engineering

Sophia Parafina — Wed, 03 Feb 2021 00:00:00 +0000

Starting can be daunting. Before you take your first step, there’s a lot to consider, but you can prepare your development environment ahead of time to make your first steps in cloud engineering smooth and productive. In this article, we’ll cover how to set up your development environment to work across cloud providers, multiple languages, and different operating systems.

Building your toolbox

You might have heard of the French culinary term “mise-en-place,” which means laying out all the ingredients and cookware before starting to cook. That’s what we’re going to do here; layout all the accounts, authorizations, and software that you need to be a successful cloud engineer. We are going to need the following:

A package manager: Every operating system has a package manager for installing software. Unlike binary installers, a package manager lets you manage all software packages, including updates. Package managers can help resolve dependencies, saving you from frustration.
Cloud provider accounts: You can choose to set up one or multiple accounts. The important thing is how to configure credentials in your development environment,
Programming languages: You can choose one or many, but they all have different versions and dependencies.
Code editor: This is personal preference but make sure that it can perform code completion, error checking, and use enums. These features can be the difference between hitting Tab and searching through online documentation for a function.
Pulumi: You can install Pulumi with a package manager and configure

Decisions

Before proceeding, you need to answer three questions:

Which operating system to use for building cloud resources: macOS, Windows, or Linux?
Which cloud provider are you using? This guide covers AWS, Azure, and Google Cloud.
Which programming language will you use? Pulumi supports Node.js (JavaScript and Typescript), Python 3.7 or higher, Golang, and .NET (C#, F#, and VB).

Once you’ve made these choices, you can follow this guide in a choose-your-own-adventure style.

Package manager

Let’s start with the package manager. We’ll use it to install and manage all the software we need, including cloud provider CLIs, programming languages, editors, and Pulumi. Choose your operating system below.

Homebrew is the most popular package manager for macOS. The Command Line Tools (CLT) for XCode is required to install and build Homebrew. Install the XCode Tools first, then install Homebrew from the command line.

$ xcode-select --install
$ /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

If you are uncomfortable with downloading and running an online shell script (with good reason), Homebrew provides an alternate installation method.

Chocolatey is a popular package manager for Windows. It should be installed using an administrative shell.

Make sure that Get-ExecutionPolicy is not restricted.

> Get-ExecutionPolicy

If the command returns Restricted, we can use Bypass to install Chocolatey with the following command:

> Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))

To check if it’s installed type:

> choco

Linux distributions include a package manager, apt for Debian based distributions or yum for Red Hat-based Linux systems. However, they are primarily used for application and system management.

Homebrew is a popular package manager for utilities, Software Development Kits, and programming tools. Install Homebrew from the command line.

$ /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

To add Homebrew to your PATH and your bash shell profile script (~/.profile on Debian or ~/.bash_profile on Red Hat), follow the Next steps instructions to install and configure brew. To check if it’s installed, install a package;

$ brew install hello

Setting up a cloud account

The first task is signing up for an account. Once you have that out of the way, the next steps are installing the CLI and configuring your credentials.

macOS

We’ll use brew to install AWS CLI version 2 and verify if it’s installed by checking the version.

$ brew update && brew install awscli
$ aws --version

The next step is to create and download your AWS access keys and configure your environment to make them available to both the AWS CLI. To create your access keys, follow these directions for programmatic access.

Pulumi uses the AWS SDK to authenticate requests. Your AWS access keys are never sent to pulumi.com.

To configure the AWS CLI with your credentials, use aws configure:

$ aws configure
AWS Access Key ID [None]: <YOUR_ACCESS_KEY_ID>
AWS Secret Access Key [None]: <YOUR_SECRET_ACCESS_KEY>
Default region name [None]:
Default output format [None]:

This creates a ~/.aws/credentials file used by the AWS CLI to authenticate requests.

We’ll use brew to install Azure CLI and check if it’s installed.

$ brew update && brew install azure-cli
$ az --version

To log into Azure, run the `login` command, which opens a browser to log into [Azure](https://login.microsoftonline.com/common/oauth2/authorize).

```bash
$ az login
The default web browser has been opened at https://login.microsoftonline.com/common/oauth2/authorize. Please continue the login in the web browser. If no web browser is available or if the web browser fails to open, use device code flow with `az login --use-device-code`.
You have logged in. Now let us find all the subscriptions to which you have access...

The Azure client is authenticated and ready to use.

Pulumi uses the Azure SDK to authenticate requests. Your AWS access keys are never sent to pulumi.com.

We’ll use brew to install the Google Cloud SDK and CLI.

$ brew update && brew install --cask google-cloud-sdk
Update done!
==> Source [/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/completion.bash.inc] in your profile to enable shell command completion for gcloud.
==> Source [/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/path.bash.inc] in your profile to add the Google Cloud SDK command line tools to your $PATH.

Add the gcloud SDK to $PATH in you ~/.bash_profile

source /usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/completion.bash.inc
source /usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/path.bash.inc

Open a new shell or source ~/.profile for Debian or ~/.bash_profile for Red Hat and check to see if it’s installed and pathed.

$ source ~/.bash_profile
$ gcloud version
Google Cloud SDK 325.0.0
bq 2.0.64
core 2021.01.22
gsutil 4.58

Authenticate using the gcloud CLI.

gcloud init

Pulumi uses the Google Cloud SDK to authenticate requests. Your AWS access keys are never sent to pulumi.com.

Windows

We’ll use chocolatey to install AWS CLI version 2 and check if it’s installed. Open a cmd Command Prompt as Administrator:

> choco install awscli
> aws --version

The install adds the AWS CLI client to the $PATH, so either open a new cmd window or use the refreshenv command to update the window’s environment variables and use the AWS CLI.

Pulumi uses the AWS SDK to authenticate requests. Your AWS access keys are never sent to pulumi.com.

To configure the AWS CLI with your credentials, use aws configure:

> aws configure
AWS Access Key ID [None]: <YOUR_ACCESS_KEY_ID>
AWS Secret Access Key [None]: <YOUR_SECRET_ACCESS_KEY>
Default region name [None]:
Default output format [None]:

This will create the ~/.aws/credentials file used by the AWS CLI to authenticate requests.

We’ll use chocolatey to install Azure CLI and check if it’s installed. Open a cmd Command Prompt as Administrator:

> choco install azure-cli

The Azure CLI client was added to the path, so either open a new cmd window or use the refreshenv command to update the window’s environment variables. To log into Azure, run the login command, which opens a browser to log into Azure.

> refreshenv
> az login

The Azure client will open a browser window and prompt you to sign-in to your account. Once signed-in, the Azure CLI is authenticated and ready to use.

Pulumi uses the Azure SDK to authenticate requests. Your AWS access keys are never sent to pulumi.com.

We’ll use chocolatey to install the Google Cloud SDK and CLI.

> choco install gcloudsdk --ignore-checksums

The gcloud CLI was added to the path, so either open a new cmd window or use the refreshenv command to update the window’s environment variables. Check to see if gcloud CLI is installed and pathed correctly.

> refreshenv
> gcloud version
Google Cloud SDK 325.0.0
bq 2.0.64
core 2021.01.22
gsutil 4.58

Authenticate using the gcloud CLI

> gcloud init
Welcome! This command will take you through the configuration of gcloud.

Your current configuration has been set to: [default]

You can skip diagnostics next time by using the following flag:
 gcloud init --skip-diagnostics

Network diagnostic detects and fixes local network connection issues.
Checking network connection...done.
Reachability Check passed.
Network diagnostic passed (1/1 checks passed).

You must log in to continue. Would you like to log in (Y/n)? Y

Your browser has been opened to visit:
 https://accounts.google.com/o/oauth2/...

A browser window will open to authenticate your client.

Pulumi uses the Google Cloud SDK to authenticate requests. Your AWS access keys are never sent to pulumi.com.

Linux

We’ll use brew to install AWS CLI version 2 and check if it’s installed.

$ brew update && brew install awscli
$ aws --version

Pulumi uses the AWS SDK to authenticate requests. Your AWS access keys are never sent to pulumi.com.

To configure the AWS CLI with your credentials, use aws configure:

$ aws configure
AWS Access Key ID [None]: <YOUR_ACCESS_KEY_ID>
AWS Secret Access Key [None]: <YOUR_SECRET_ACCESS_KEY>
Default region name [None]:
Default output format [None]:

This will create the ~/.aws/credentials file used by the AWS CLI to authenticate requests.

We’ll use brew to install Azure CLI and check if it’s installed.

$ brew update && brew install azure-cli
$ az --version

To log into Azure, run the `login` command, which opens a browser to log into [Azure](https://login.microsoftonline.com/common/oauth2/authorize).

```bash
$ az login
The default web browser has been opened at https://login.microsoftonline.com/common/oauth2/authorize. Please continue the login in the web browser. If no web browser is available or if the web browser fails to open, use device code flow with `az login --use-device-code`.
You have logged in. Now let us find all the subscriptions to which you have access...

The Azure client is authenticated and ready to use.

Pulumi uses the Azure SDK to authenticate requests. Your AWS access keys are never sent to pulumi.com.

We’ll use brew to install the Google Cloud SDK and CLI.

$ brew update && brew install --cask google-cloud-sdk

Update done!

==> Source [/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/completion.bash.inc] in your profile to enable shell command completion for gcloud.
==> Source [/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/path.bash.inc] in your profile to add the Google Cloud SDK command line tools to your $PATH.

Add the gcloud SDK to $PATH to your ~/.bash_profile or ~/.profile.

source /usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/completion.bash.inc
source /usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk/path.bash.inc

Open a new shell or source ~/.profile for Debian or ~/.bash_profile for Red Hat and check to see if it’s installed and pathed.

$ source ~/.bash_profile
$ gcloud version
Google Cloud SDK 325.0.0
bq 2.0.64
core 2021.01.22
gsutil 4.58

Authenticate using the gcloud CLI

gcloud init

Pulumi uses the Google Cloud SDK to authenticate requests. Your AWS access keys are never sent to pulumi.com.

Choose a Programming Language

Pulumi supports Node.js (JavaScript and Typescript), Python 3, Golang, and .NET Core (C#, VB, and F#) languages.

Node.js

Use brew to install Node.js for JavaScript and Typescript.

$ brew install node

Use chocolatey to install Node.js for JavaScript and Typescript.

> choco install nodejs

Use brew to install Node.js for JavaScript and Typescript.

$ brew install node

Python

MacOS includes Python; however, versions shipped before December 2019 have Python 2.7 installed, which is deprecated. Pulumi requires Python 3.7 or higher. Use brew to install Python, which installs it at /usr/bin/local/python3.

$ brew install python

In macOS versions with Python 2 installed, calling python uses the 2.7 binary. To ensure that you will always use Python 3, you can add an alias to your .bash_profile.

alias python=/usr/local/bin/python3

It is a best practice to create a virtual environment and activate it for a Python project. Pulumi creates a new venv environment when you start a new Project with pulumi new.

You can install Python in several ways, including typing ‘python` in the command prompt, which brings up the Microsoft Store application. In this article, we’ll continue using chocolatey to install Python 3.

> choco install python --pre

It is a best practice to create a virtual environment and activate it for a Python project. Pulumi creates a new venv environment when you start a new Project with pulumi new.

Use brew to install Python3.

$ brew install python

Use python3 to call Python, but you can add an alias to your .bash_profile or .profile to use just python to run scripts.

alias python=/usr/local/bin/python3

It is a best practice to create a virtual environment and activate it for a Python project. Pulumi creates a new venv environment when you start a new Project with pulumi new.

Golang

Use brew to install golang.

$ brew install golang

It is a best practice to create a local programming environment and set GOPATH as an environment variable, although it is not required for golang versions after 1.8. Pulumi creates a new local environment and go.mod to work with modules each time you start a new Project with pulumi new.

Use chocolatey to install golang.

> choco install golang

It is a best practice to create a local programming environment and set GOPATH as an environment variable, although it is not required for golang versions after 1.16. Pulumi creates a new local environment and go.mod to work with modules each time you start a new Project with pulumi new.

Use brew to install golang.

$ brew install golang

.NET

Install .NET with brew, Pulumi requires .NET Core 3.1 or higher.

$ brew install dotnet

Install .NET with chocolatey. Pulumi requires .NET Core 3.1 or higher.

> choco install dotnet-sdk

Install .NET with brew, Pulumi requires .NET Core 3.1 or higher.

$ brew install dotnet

Get a code editor

While knowing how to use vim or similar text editors is a worthwhile skill, you should use a modern code editor. Here’s a non-exhaustive list why you should use a code editor.

Quickly navigating to a type
Autocompletion when you can’t remember the names of all members by heart
Automatic code generation
Refactoring
Organise imports
Warnings as you type.
Hovering over something to see the docs
Keeping a view of files, errors/warnings/console/unit-tests and source code on the screen
Running unit tests from the same window
Integrated debugging
Integrated source control
Navigating to where a compile-time error or run-time exception occurred directly from the error details.

You can use an IDE (Integrated Development Environment) such as Microsoft Visual Studio, Xcode, or any one of JetBrains’ language-specific IDEs. Alternatively, you can use a lightweight solution with many of the features of an IDE. Popular code editors include Visual Studio Code, Sublime Text, and Atom.

You can install some editors with brew.

$ brew install --cask visual-studio-code
$ brew install --cask sublime-text
$ brew install --cask atom
$ brew install --cask pycharm-ce
$ brew install --cask goland

Install Pulumi

macOS Ventura (13) or later is required.

Homebrew

You can install Pulumi through the Homebrew package manager:

$ brew install pulumi/tap/pulumi

This will install the pulumi CLI to the usual place (often /usr/local/bin/pulumi) and add it to your path.

Subsequent updates can be installed in the usual way:

$ brew upgrade pulumi

Installation Script

To install, run our installation script:

$ curl -fsSL https://get.pulumi.com | sh

This will install the pulumi CLI to ~/.pulumi/bin and add it to your path. When it can’t automatically add pulumi to your path, you will be prompted to add it manually. See How to permanently set $PATH on Unix for guidance.

Windows 8 and 10 are supported.

Chocolatey

You can install Pulumi using elevated permissions through the Chocolatey package manager:

> choco install pulumi

This will install the pulumi CLI to the usual place (often $($env:ChocolateyInstall)\lib\pulumi) and generate the shims (usually $($env:ChocolateyInstall)\bin) to add Pulumi your path.

Subsequent updates can be installed in the usual way:

> choco upgrade pulumi

Verifying your Installation

After installing Pulumi, verify everything is in working order by running the pulumi CLI:

$ pulumi version
v3.255.0

$ pulumi version
v3.255.0

> pulumi version
v3.255.0

Next steps

Congratulations! You have a fully configured environment, and you’re ready to jump into cloud engineering. You can maintain your development environment using the package manager to add or update your toolset. Your code editor provides a modern development platform that takes advantage of all the advances in software engineering. Your coding experience will be more productive and less frustrating.

What are the next steps? Begin with Pulumi’s Getting Started. You can skip the configuration sections and jump straight into your first project. Once you’re done with your first project, try out example projects on Github. You can start with simple projects using the pulumi CLI, such as deploying a web server on AWS with python.

$ pulumi new https://github.com/pulumi/examples/tree/master/aws-py-webserver

This command will download the project from Github, create a virtual environment and activate it, and download all the python package dependencies. You’re ready to go and deploy with pulumi up. Want more? How about deploying Kubernetes on Azure with python?

$ pulumi new https://github.com/pulumi/examples/tree/master/azure-py-aks

You can use the examples as a starting point for building your cloud infrastructure and add resources documented on Pulumi’s Registry and Guides.

Build a Pulumi VS Code Devcontainer Environment

Sophia Parafina — Tue, 14 Jul 2020 00:00:00 +0000

One of the major advantages of using containers for development is reducing the need to install software and associated dependencies. Developers can start writing code without configuring a development environment that emulates production. The Visual Studio Code Remote - Containers extension lets you develop inside a container. If you want to use Pulumi’s infrastructure as code engine without installing the Pulumi CLI, this blog post is for you!

Pre-requisites

To set up your development environment, you’ll need Visual Studio Code, Docker Desktop, and the Remote - Containers extension.

Check to see if Docker is running:

$ docker --version
Docker version 19.03.8, build afacb8b

Check to see if the Remote - Containers is installed:

Click on the green icon in the Remote Status Bar to display the commands:

Building the Pulumi container

Now we can start!

Create a directory for your project.
```
$ mkdir my_devcontainer
$ cd my_devcontainer
```
Open the directory in VS Code.
Next, we’ll create the devcontainer configuration files. Click on the Remote Status button and select Add Development Container Configuration Files ….
Select the latest version for Alpine. We’ll replace the Dockerfile with the Pulumi Dockerfile.

VS Code will display a popup asking to build the container. Dismiss the popup, we will replace the Dockerfile with the Pulumi Dockerfile.

In your project directory, you’ll see a .devcontainer directory with a Dockerfile that builds the container and a devcontainer.json for configuring the container environment.

Dockerfile

For this example, we’ll use the standard Pulumi Dockerfile, which includes the SDK for all the supported languages. Download the Dockerfile and save it to the .devcontainer directory.

$ cd .devcontainer
$ curl "https://raw.githubusercontent.com/pulumi/pulumi/master/.devcontainer/Dockerfile" -o Dockerfile

Container configuration

Next, we’ll clean up the .devcontainer/devcontainer.json file and set up the necessary credentials.

First edit devcontainer.json as follows:

Replace the “name” field with Pulumi
Remove the “args” line
Replace the “remoteUser” value with user

Regardless of your cloud provider, you will have to provide your credentials to deploy resources. If you want to save your project to Github or another repository, you don’t want your credentials stored in the repository. There are several ways to do this securely. Using AWS as an example, you can export them inside the container, e.g.,

Linux/macOS:

export AWS_ACCESS_KEY_ID=AKIAI44QH8DHBEXAMPLE
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Windows:

SET AWS_ACCESS_KEY_ID=AKIAI44QH8DHBEXAMPLE
SET AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

The disadvantage of this method is that you will have to set these environment variables every time you start the devcontainer.

Use local environment variables

One alternative is to set the environment variables on your local machine and pass them to the container through the devcontainer.json file. This method requires that you have VS Code in your path and launch it from the shell.

Use CTRL+SHIFT+P or CMD+SHIFT+P to bring up the command palette and type Shell Command: Install ‘code’ command in PATH. You can set your credentials as environmental variables and launch the VS Code from the shell
```
$ export AWS_ACCESS_KEY_ID=AKIAI44QH8DHBEXAMPLE
$ export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
$ code .
```

Next, set your provider credentials on your machine and add the following to devcontainer.json.

"containerEnv": {
 "AWS_ACCESS_KEY_ID": "${localEnv:AWS_ACCESS_KEY_ID}",
 "AWS_SECRET_ACCESS_KEY": "${localEnv:AWS_SECRET_ACCESS_KEY}"
},

Use an .env file

Another alternative is to use a devcontainer.env file, which contains your credentials.

Create a file with your credentials.

AWS_ACCESS_KEY_ID=AKIAI44QH8DHBEXAMPLE
AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Then add it to the devcontainer.json file.

"runArgs": ["--env-file","devcontainer.env"],

To prevent it from exposing your credentials on a public repository, add the file to .gitignore or similar for you repository, e.g.,
```
echo devcontainer.json > .gitignore
```

There are many more configuration options available, including mounts and remote configurations.

Try it out

Again, use the command palette to start the container. Use the Open Folder in Container… command, and select the project folder. If you’ve already got the folder open locally, you could also use the Reopen in Container command.

Check to see if your credentials are set. Open a terminal in VS Code if it isn’t already open.

$ user@973a612c1555:/workspaces/my_devcontainer# env | grep AWS
AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
AWS_ACCESS_KEY_ID=AKIAI44QH8DHBEXAMPLE

Now we ready to use Pulumi in our VS Code development environment! Let’s try it out by creating an S3 bucket. First, we’ll make a directory for our project and use pulumi new to create a default project.

$ mkdir my_project
$ cd my_project
$ pulumi new aws-typescript

Follow the prompts, then run pulumi up to deploy.

$ pulumi up
Previewing update (dev):
 Type Name Plan
 + pulumi:pulumi:Stack my_project-dev create
 + └─ aws:s3:Bucket my-bucket create

Resources:
 + 2 to create

Do you want to perform this update? yes
Updating (dev):
 Type Name Status
 + pulumi:pulumi:Stack my_project-dev created
 + └─ aws:s3:Bucket my-bucket created

Outputs:
 bucketName: "my-bucket-cbdec06"

Resources:
 + 2 created

Duration: 31s

Permalink: https://app.pulumi.com/spara/my_project/dev/updates/1

You can CMD + click on the link to open a browser and display the status of your project on the Pulumi Service.

Build a Pulumi examples devcontainer

Pulumi has a repository of examples, and we can create a devcontainer from the repository so you can try them out.

Clone the repository, then follow the steps above to create and configure the Pulumi container image.
```
$ git clone https://github.com/pulumi/examples.git
```
Create the configuration files, add the Pulumi Docker file, and add your credentials.
Let’s deploy a Kubernetes cluster! With pulumi stack init, we can create a stack, but we will need to install the project dependencies and set configuration variables such as the region.
```
$ cd aws-ts-eks
$ pulumi stack init
$ npm install
$ pulumi config set aws:region us-west-2
$ pulumi up
```

Here’s the kubeconfig file’s output, which you can use with kubectl to manage the cluster.

Outputs:
 kubeconfig: {
 apiVersion : "v1"
 clusters : [
 [0]: {
 cluster: {
 certificate-authority-data: "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJd01EY3hNVEl5TXpZMU9Wb1hEVE13TURjd09USXlNelkxT1Zvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTHFvCk9wTDBiQWo2TFFMNlh2anRsMFZ6TUN5WTJmYlo4cXEvNGd3bkJxcVI2RFV6Q1RwbkJmOHJVRjVrUXhBQXRUNysKTTFZblp5VEFBYndOZ3NIL3lObk5sMnFCQmRSQ0JZTVVlN1I2MVBxcGdabFZpN0JaNHRlek9DalRqeGV0SXlaMwoxRW9TdTNBYnhzTmh6dDVBTjVYZklnZlBuM3B2eEtwYTA1MEg3aWhFaUptSEl3M0Z5TVovUHpxY2JHczh3RHVCCk1jYjNzOHMwZDBIVExsb2dCL2svOVBLbGVpZDgyWFkxMEdBN0YxeFM2cFdZODkwQ1BwNXpWZ0pRWGhuRGR2S1gKVlFValdCTDZQYU14U0lBZEZQM1BZYk1FdXk5Vyt0c2lqc2VydThVNTVpUDlWSUdMdTZzdEdycTNuQUhtUk40UQpqeXRCRWd4RjN1R0dtZ0dMOStFQ0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFEcUtyVGVnZmhWbElNUklQNjFVZEtvaGtnS3MKblUyZjUvTHBzU3ozbmliUCt3Y2VpNktYL3RSNjAzTHhFMlVXUTlmR0ZycTBCMTlJR1JaK0VVQjl2Vy9JdWowTApVK2UwUUtWc01qd2dMWm4rS3IyN0VQNUNGay8vZEVCRUZqa01PMmNlTFFRaFcvUXFzSFhJb2ZjZ0UvZnJ3YzErCmQwdGROWmxJa3JRdzJwYnVhcVI0VmRlWlpVSitqbW8rdEo1SlZ5a2Rwa29JYml1ZEZ6Q0N5YVB1LzhJc09CTUsKcWtvTFpNazdBMzh5SWxDK2RCTnJwdkQvRTF5U1lhcXlDaE5CdVBkd1VqcWZaWmJmaWR5N050emd2VjMvUWFtWgpuZUJJcWs0c1AzSWU0TUN0QmV4bU9iSWowdVBLTnVCQ01UTStxSVI2UWdnYXdEbWxXcXE4V0ZiUlNJMD0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo="
 server : "https://3FE47B904E57AF9E16E0B7F00C7EEABA.yl4.us-west-2.eks.amazonaws.com"
 }
 name : "kubernetes"
 }
 ]
 contexts : [
 [0]: {
 context: {
 cluster: "kubernetes"
 user : "aws"
 }
 name : "aws"
 }
 ]
 current-context: "aws"
 kind : "Config"
 users : [
 [0]: {
 name: "aws"
 user: {
 exec: {
 apiVersion: "client.authentication.k8s.io/v1alpha1"
 args : [
 [0]: "eks"
 [1]: "get-token"
 [2]: "--cluster-name"
 [3]: "cluster-eksCluster-367b66b"
 ]
 command : "aws"
 }
 }
 }
 ]
 }

You can now try out any of our infrastructure as code examples with this devcontainer.

Conclusion

As you can see, we’ve built our development with VS Code and Docker. Using a Pulumi devcontainer lets you build and deploy infrastructure without installing Pulumi or any associated dependencies on your computer. Moreover, you can share your development environment by creating a repository. Give it a try and let us know what you’re building on Slack or Twitter.