1. Docs
  2. Using Pulumi
  3. Continuous Delivery
  4. Azure DevOps

Pulumi CI/CD & Azure DevOps

    This page details how to use Azure DevOps to manage deploying stacks based on commits to specific Git branches, and based on the build reason. You may also choose to introduce a Manual Intervention task to control the preview vs. update step for any Pulumi stack.

    Pulumi doesn’t require any particular arrangement of stacks or workflow to work in a continuous integration / continuous deployment system. So the steps described here can be altered to fit into any existing type of deployment setup.

    This page shows you how to use the Pulumi Azure DevOps task extension and the YAML method to configure your DevOps pipeline, but you can easily adapt the steps outlined in the sample YAML file below to the Visual Designer as well.

    Prerequisites

    Pulumi Task Extension for Azure Pipelines

    Pulumi provides a task extension that lets you easily use Pulumi in your CI/CD pipelines. It can be used with the Azure Pipelines wizard UI or the YAML config. The task handles installing the Pulumi CLI and running any commands without the need for any scripts.

    Pulumi Task Extension for Azure Pipelines can be used with any cloud provider that Pulumi supports. You are not limited to using it only with Azure.

    Command Summary

    Details regarding parameters and options supported by the extension can be found in the Pulumi CLI documentation.

    Parameter NameRequired?Parameter Description
    stackYesName of stack being managed. Can be of the form ORG/STACK or ORG/PROJECT/STACK.
    azureSubscriptionNoOptionally reference a service connection. If not used, environment variables can be configured with the credentials needed for the applicable Pulumi providers.
    commandNoThe applicable pulumi cli command (e.g. preview, up, destroy)
    argsNoOption flags (e.g. --yes) that can be passed to the given pulumi command. Use space to separate multiple args.
    cwdNoThe working directory to run the Pulumi commands. Use this if your Pulumi app is in a different directory.
    versionSpecNoThe Pulumi version that should be used. Defaults to the latest version. If you require a specific version then the format is 1.5.0 or if you just need the latest version then latest can be used.
    createStackNoSet to true if the stack should be created if it does not already exist. Defaults to false.
    createPrCommentNoSet to true to add a comment to your Pull Request (PR). Can only be used in pipelines driven by PRs. Defaults to false. See Log Pulumi Output as PR Comments.
    useThreadedPrCommentsNoDefaults to true to always add a comment to the previously-created comments thread. Set to false to have each comment added separately.

    Using the Pulumi Task Extension

    Install the Pulumi task from the Visual Studio Marketplace to your Azure DevOps organization.

    The task requires the use of a service connection, which allows the pipeline to connect to your Azure Subscription. The task also looks for the build variable pulumi.access.token, and automatically maps it to the environment variable PULUMI_ACCESS_TOKEN, that is used by the CLI for non-interactive logins. You may still use the env directive to map any other environment variables you wish to make available to your Pulumi app.

    You can obtain a Pulumi access token from the Pulumi Cloud. Then use the task in your pipeline yaml.

    # Lines omitted for brevity.
    ...
    ...
      - task: Pulumi@1
        condition: or(eq(variables['Build.Reason'], 'PullRequest'), eq(variables['Build.Reason'], 'Manual'))
        inputs:
          azureSubscription: "My Service Connection"
          command: "preview"
          cwd: "infra/"
          stack: "acmeCorp/acmeProject/acme-ui"
    ...
    ...
    

    Log Pulumi Output as PR Comments

    This feature is only supported for builds triggered by pull requests created in git repositories hosted by Azure DevOps. Repositories hosted by external VCS such as Bitbucket, GitHub, GitLab are not supported at this time.

    The Pulumi task supports adding PR comments containing the log output from the Pulumi command that was executed in your build pipeline. Your project’s build service user will need additional permissions to perform that action. Follow these steps to grant the build service user the Contribute to pull requests permission:

    • Navigate to the Project Settings page and select Repositories under the Repos heading.
    • Select the repository where you will be using this feature and then select the Security tab.
    • Now under the Users section find the build service user. If you are using the default build service user, the naming convention is <Project name> Build Service where <Project name> is your project’s name.
    • Change the value of Contribute to pull requests to Allow.

    Using The Pulumi Task Extension With Other Clouds

    To use the Pulumi Task Extension for Azure Pipelines with other clouds, you can specify the necessary environment variables as build variables or link variable groups to your build and release pipelines.

    For example, if you are using the AWS provider, you can set the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

    Using Pulumi in Azure DevOps Pipelines

    More details on how to use Pulumi - with or without the Task Extension - are provided below.

    Stack and Branch Mappings

    The names used below are purely for demonstration purposes only. You may choose a naming convention that best suits your organization.

    The scripts below act on a hypothetical stack: acmeCorp/acmeProject/acme-ui. acme-ui contains the infrastructure code or pulumi program. It also contains an Angular-based SPA. The git repo for this look like this:

    acme-ui
      src/
        app/
          components/
        index.html
      infra/
        index.ts
        Pulumi.yaml
        Pulumi.acme-ui.yaml
      azure-pipelines.yml
    

    Once you login into the pulumi CLI on your machine, you can create a stack by running pulumi stack init. To create a pulumi program using one of the many available templates, you may run pulumi new <template>. You can find a suitable template to run, using the command pulumi new --help. Learn more about the pulumi CLI commands.

    Once your stack has been initialized, the Pulumi.<stack-name>.yaml file will be created with some basic configuration. The yaml file is used just for configuration values. All of your infrastructure will be built using your pulumi program.

    For this walkthrough, we will assume a TypeScript-based pulumi program, which will deploy resources to an Azure Subscription.

    About The pulumi Program

    The code inside infra/index.ts creates a resource group, a storage account and a blob container in the storage account. It then exports three values using the syntax export const <variable_name> = <value>;. Learn more about stack outputs.

    Build Variables

    Build variables are an important aspect of any CI/CD pipeline. We will use some pre-defined system build variables provided by the Azure DevOps pipeline to decide whether or not we should run an update on our infrastructure.

    Build variable formats differ based on the agent in which your job is running. See this to learn more about how you can access a build variable correctly depending on your agent OS.

    User-Defined Output Variables

    You can set job-scoped output variables or multi-job variables. In this article, we demonstrate the use of multi-job variables with job dependencies, using the dependsOn job constraint.

    Environment Variables

    pulumi requires a few environment variables in order to work in a CI/CD environment. More specifically, PULUMI_ACCESS_TOKEN is required to allow the pulumi CLI to perform an unattended login. In addition to this, you will also need to set the cloud provider-specific variables. Azure environment variables.

    If you are using the Pulumi task extension for Azure Pipelines, you don’t need to manually configure the environment variables in your pipeline builds. You can use Service Connections to centralize access to your Azure subscription(s).

    Agents

    Azure DevOps allows you to specify a build agent for each of your jobs in your pipeline. You may have a requirement to run certain jobs on a Ubuntu agent, and some on a Windows agent. If not using the task extension, pulumi can be installed on these agents by following the directions from this page.

    Setup

    For the YAML-driven DevOps pipeline, the repository must contain the azure-pipelines.yml in the root of the repo for Azure DevOps to use it automatically. The following are samples only. You may choose to structure your configuration any way you like.

    Sample azure-pipelines.yml

    # Node.js with Angular
    # Build a Node.js project that uses Angular.
    # Add steps that analyze code, save build artifacts, deploy, and more:
    # https://docs.microsoft.com/azure/devops/pipelines/languages/javascript
    
    jobs:
    - job: infrastructure
      pool:
        vmImage: 'ubuntu-20.04'
      steps:
      - task: Npm@1
        inputs:
          command: install
          workingDir: "infra/"
      - task: Pulumi@1
        condition: or(eq(variables['Build.Reason'], 'PullRequest'), eq(variables['Build.Reason'], 'Manual'))
        inputs:
          # Using a service connection is optional. Specify build variables for your pipeline or link variable groups
          # that contain the necessary environment variables needed by the Pulumi provider your Pulumi app uses.
          azureSubscription: 'My Service Connection'
          command: 'preview'
          cwd: 'infra/'
          stack: 'acmeCorp/acmeProject/acme-ui'
      - task: Pulumi@1
        condition: or(eq(variables['Build.Reason'], 'IndividualCI'), eq(variables['Build.Reason'], 'BatchedCI'))
        inputs:
          # Using a service connection is optional. Specify build variables for your pipeline or link variable groups
          # that contain the necessary environment variables needed by the Pulumi provider your Pulumi app uses.
          azureSubscription: 'My Service Connection'
          command: 'up'
          cwd: 'infra/'
          stack: 'acmeCorp/acmeProject/acme-ui'
          args: '--yes'
      - script: |
          RESOURCE_GROUP_NAME=$(pulumi stack output resourceGroupName)
          STORAGE_ACCOUNT_NAME=$(pulumi stack output storageAccountName)
          CONTAINER_NAME=$(pulumi stack output containerName)
          echo "##vso[task.setvariable variable=resourceGroupName;isOutput=true]$RESOURCE_GROUP_NAME"
          echo "##vso[task.setvariable variable=storageAccountName;isOutput=true]$STORAGE_ACCOUNT_NAME"
          echo "##vso[task.setvariable variable=containerName;isOutput=true]$CONTAINER_NAME"      
        displayName: 'Set stack outputs as variables'
        name: 'pulumi'
    
    # The following job is optional, and shown here for demonstration purposes only.
    - job: build_and_deploy
      condition: ne(dependencies.infrastructure.outputs['pulumi.containerName'], '')
      dependsOn: infrastructure
      variables:
        resourceGroupName: $[ dependencies.infrastructure.outputs['pulumi.resourceGroupName'] ]
        storageAccountName: $[ dependencies.infrastructure.outputs['pulumi.storageAccountName'] ]
        containerName: $[ dependencies.infrastructure.outputs['pulumi.containerName'] ]
      steps:
      - task: NodeTool@0
        inputs:
          versionSpec: '16.x'
        displayName: 'Install Node.js'
      - task: Npm@1
        inputs:
          command: 'install'
      - script: |
          npm run build      
        displayName: 'UI build'
      - task: AzurePowerShell@3
        displayName: 'Upload UI dist bundle to Storage Account'
        inputs:
          azureConnectionType: 'ConnectedServiceNameARM'
          azureSubscription: 'My Service Connection'
          scriptType: 'FilePath'
          # This script file is shown below in the next section.
          scriptPath: 'build-and-deploy.ps1'
          scriptArguments: '-resourceGroupName $(resourceGroupName) -storageAccountName $(storageAccountName) -containerName $(containerName) -isAzurePipelineBuild $true -skipBuild $true -localFolder $(Build.SourcesDirectory)/dist'
          errorActionPreference: 'stop'
          failOnStandardError: true
          azurePowerShellVersion: 'LatestVersion'
    

    Sample build-and-deploy.ps1 (Optional)

    This PowerShell script builds the UI app, and uploads the dist/ folder to an Azure Storage blob container. You don’t have to use a script like this. You can always use the built-in Azure DevOps task to accomplish the steps in this script. This script is just an example of how pulumi can be easily integrated into your existing app.

    Param(
      [string]$storageAccountName,
      [string]$containerName,
      [string]$resourceGroupName,
      [int] $maxAge = 86400,
      [int] $indexMaxAge = 3600,
      [boolean] $isProductionBuild = $false,
      [boolean] $skipBuild = $false,
      [boolean] $isAzurePipelineBuild = $false,
      # Upload files in the dist subfolder to Azure.
      [string] $localFolder = ".\dist"
    )
    
    function Get-MimeType() {
      param([parameter(Mandatory=$true, ValueFromPipeline=$true)][ValidateNotNullorEmpty()][System.IO.FileInfo]$CheckFile)
      begin {
          Add-Type -AssemblyName "System.Web"
          [System.IO.FileInfo]$check_file = $CheckFile
          [string]$mime_type = $null
      }
      process {
          if ($check_file.Exists) {
              $mime_type = [System.Web.MimeMapping]::GetMimeMapping($check_file.FullName)
          }
          else {
              $mime_type = "false"
          }
      }
      end {
        return $mime_type
      }
    }
    
    if ($false -eq $skipBuild) {
      npm ci
    
      if ($false -eq $isProductionBuild) {
        write-host "Building the UI for non-prod"
        npm run build
      } else {
        write-host "Building the UI for PROD"
        npm run build:prod
      }
      write-host "UI build completed"
      write-host "Launching Azure login pop-up.."
    }
    
    # The destination folder. BE SURE TO INCLUDE THE TRAILING SLASH IF YOU ENTER A NON-EMPTY VALUE HERE.
    $destfolder = ""
    $storageAccount = Get-AzureRmStorageAccount -ErrorAction Stop | where-object {$_.StorageAccountName -eq $storageAccountName}
    If($storageAccount)
    {
      $storageAccountKey = Get-AzureRmStorageAccountKey -ResourceGroupName $storageAccount.ResourceGroupName -StorageAccountName $storageAccountName
      $storageContext = New-AzureStorageContext -StorageAccountName $storageAccountName -StorageAccountKey $storageAccountKey[0].Value
      $storageContainer = Get-AzureStorageContainer -Context $storageContext -ErrorAction Stop | where-object {$_.Name -eq "$ContainerName"}
    
      Invoke-Expression -Command "ls $localFolder"
      $files = Get-ChildItem $localFolder  -File -Recurse
      foreach($file in $files)
      {
        $directoryName = $file.Directory.Name
        $filename = $file.Name
        if ($directoryName -ne "dist") {
          $blobName = $destfolder + $file.FullName.Substring($file.FullName.IndexOf("dist\") + 5)
        } else {
          $blobName = $destfolder + $filename
        }
        write-host "copying $directoryName\$filename to $blobName"
        if ($file.Name -like "*mp4*") {
          $mimeType = "video/mp4"
        } elseif ($file.Name -like "*webm*") {
          $mimeType = "video/webm"
        } else {
          $mimeType = $(Get-MimeType -CheckFile $file.FullName)
        }
        # For index.html, use the $indexMaxAge
        if ($file.Name -eq "index.html") {
          $maxAge = $indexMaxAge
        }
        $Properties = @{"CacheControl" = "max-age=$maxAge"; "ContentType" = $mimeType}
        Set-AzureStorageBlobContent -File $file.FullName -Container "$containerName" -Blob $blobName -Context $storageContext -Force -Properties $Properties
      }
      write-host "All files in $localFolder uploaded to $containerName!"
    } else {
      Write-Warning "'$storageAccountName' storage account not found."
    }
    

    Using Scripts (Manual Approach)

    If you prefer to control the installation of the Pulumi CLI and how it runs your Pulumi app, you can use scripts in your pipeline builds. Below are some sample scripts to help you get started in order to install the CLI and run your Pulumi app.

    The run-pulumi.sh script runs pulumi preview for PR builds and the pulumi up --yes command with explicit consent, for master branches.

    The following environment variables are set in the build pipeline using the Azure DevOps portal.

    • pulumi.access.token
    • arm.client.id
    • arm.client.secret
    • arm.subscription.id
    • arm.tenant.id

    The above variables are mapped-in to the job using the env: directive as described in Set secret variables.

    azure-pipelines.yml

    In your pipeline configuration, you need to then call these scripts when appropriate. Here’s an example:

    ...
    ...
      # Some lines omitted for brevity.
      - script: |
          chmod +x infra/scripts/*.sh
          ./infra/scripts/setup.sh
          ./infra/scripts/run-pulumi.sh      
        displayName: 'Install pulumi and run infra code'
        name: pulumi
        env:
          PULUMI_ACCESS_TOKEN: $(pulumi.access.token)
          ARM_CLIENT_SECRET: $(arm.client.secret)
          ARM_SUBSCRIPTION_ID: $(arm.subscription.id)
          ARM_CLIENT_ID: $(arm.client.id)
          ARM_TENANT_ID: $(arm.tenant.id)
    ...
    ...
    

    Sample setup.sh

    #!/bin/bash
    
    # exit if a command returns a non-zero exit code and also print the commands and their args as they are executed
    set -e -x
    # Download and install required tools.
    # pulumi
    curl -fsSL https://get.pulumi.com/ | bash
    export PATH=$PATH:$HOME/.pulumi/bin
    # Login into pulumi. This will require the PULUMI_ACCESS_TOKEN environment variable
    pulumi login
    # update the GitLab Runner's packages
    apt-get update -y
    apt-get install sudo -y
    # nodejs
    curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -
    apt-get install -y nodejs
    # yarn
    npm i -g yarn
    

    Sample run-pulumi.sh

    #!/bin/bash
    
    # exit if a command returns a non-zero exit code and also print the commands and their args as they are executed
    set -e -x
    
    # Add the pulumi CLI to the PATH
    export PATH=$PATH:$HOME/.pulumi/bin
    
    pushd infra/
    
    npm install
    npm run build
    
    pulumi stack select acmeCorp/acmeProject/acme-ui
    
    # https://docs.microsoft.com/en-us/azure/devops/pipelines/build/variables?view=vsts
    case $BUILD_REASON in
      PullRequest)
          pulumi preview
        ;;
      BuildCompletion|BatchedCI)
          pulumi up --yes
        ;;
      *)
    esac
    
    # Save the stack output variables to job variables.
    # Note: Before the `pulumi up` is run for the first time, there are no stack output variables.
    # The pulumi program exports three values: resourceGroupName, storageAccountName and containerName.
    echo "##vso[task.setvariable variable=resourceGroupName;isOutput=true]$(pulumi stack output resourceGroupName)"
    echo "##vso[task.setvariable variable=storageAccountName;isOutput=true]$(pulumi stack output storageAccountName)"
    echo "##vso[task.setvariable variable=containerName;isOutput=true]$(pulumi stack output containerName)"
    
    popd
    
      Pulumi AI - What cloud infrastructure would you like to build? Generate Program