Pulumi Blog: Cleve Littlefield

Announcing AWS Systems Manager - Parameter Store Support for Pulumi ESC

Sean Yeh — Mon, 16 Dec 2024 12:00:00 -0600

We are super excited to announce integration support for AWS Systems Manager - Parameter Store within Pulumi Environments, Secrets, and Configuration (ESC). Parameter Store is a popular managed service by AWS for storing and managing secrets and other configuration, and its integration with ESC has been highly requested among the community.

With the AWS Parameter Store ESC Provider, development teams can now import their AWS Parameter Store parameters as environment variables or as Pulumi Config in their workflows, all in a secure and convenient way.

As we continue to expand our roster of integrations, it’s easier now than ever to manage and use secrets and configuration all in one place by using ESC. If you or your team are users of AWS Parameter Store, say goodbye to managing multiple configuration sources and give this provider a try!

How does it work?

Add parameters to AWS Parameter Store.
Use aws-login to authenticate with AWS.
Define a path name, and enter the parameter name you want to retrieve. This will be imported into the path name that you have defined.

AWS Parameter Store provider syntax:

values:
 params:
 fn::open::aws-parameter-store:
 region: us-west-1
 login: ${aws.login}
 get:
 myKey:
 name: /myNamespace/myKey
 myKeyByVersion:
 name: /myNamespace/myKey:1
 myKeyByVersionLabel:
 name: /myNamespace/myKey:stable
 secureKey:
 name: /myNamespace/secureKey
 decrypt: true

Demo

Now that we know how to use the Pulumi ESC Parameter Store provider, let’s use it to pull some parameters!

Setup OpenID Connect for AWS. We will use OIDC to authenticate with AWS.
Store some parameters in Parameter Store. In the demo, we will store a String /test/api-endpoint and a SecureString /test/api-key
Create a new Pulumi ESC Environment with the following definition. Make sure to replace the OIDC roleArn with your own.

values:
 aws:
 login:
 fn::open::aws-login:
 oidc:
 duration: 1h
 roleArn: arn:aws:iam::123456789:role/esc-oidc
 sessionName: pulumi-environments-session
 params:
 fn::open::aws-parameter-store:
 region: us-west-1
 login: ${aws.login}
 get:
 api-endpoint:
 name: /test/api-endpoint
 api-key:
 name: /test/api-key
 decrypt: true
 environmentVariables:
 API_ENDPOINT: ${params.api-endpoint}
 API_KEY: ${params.api-key}

Open the environment to ensure we are able to successfully pull the credentials from AWS Parameter Store
Run pulumi env open <project/environment> --format shell to view the exported environment variables from this ESC environment.
Use pulumi env run to run any command with these environment variables. For example, we can now run pulumi env run <project/environment> -- <your command>

pulumi env run passes the configuration stored under the environmentVariables section for use with the provided command, without ever storing them locally on your machine.

Conclusion

We are excited to expand our roster of integrations with the new AWS Parameter Store ESC Provider, providing developers more flexibility and choices. We look forward to seeing what you will create next using Pulumi ESC!

Visit the AWS Parameter Store Provider Page for more detailed documentation on this provider.
Visit our Pulumi ESC docs to learn more about Pulumi ESC and its supported integrations

As always, we deeply value your insights. Your feedback is instrumental in helping us refine and enhance our solutions to better align with your needs.

Announcing the Pulumi Visual Studio Code Extension

Meagan Cojocar — Wed, 18 Sep 2024 08:56:00 +0000

At the heart of Pulumi’s approach to cloud infrastructure and secrets management is a belief in empowering engineers to use the best software engineering tools to manage complexity at scale and to be maximally productive building cloud infrastructure and applications for their businesses.

Today, we’re excited to announce a next big step in delivering great software engineering tools for Pulumi users, with the launch of a new Pulumi Visual Studio Code (VS Code) Extension. The Pulumi VS Code Extension brings Pulumi key features for Pulumi IaC and Pulumi ESC directly into the IDE environment that many Pulumi users work in every day. Pulumi IaC users can now debug their applications and get Pulumi YAML language support directly in VS Code. And Pulumi ESC users can now create and manage environments, secrets and configuration directly within the IDE with rich IDE features.

The following are a few examples of new capabilities available for Pulumi users in VS Code today:

Launch your Pulumi program under a debugger
Automatically generate a launch configuration for a Pulumi project
Create and explore Pulumi ESC (Environments, Secrets, and Configuration) environments
Get contextual warnings and errors directly within the editor for Pulumi YAML

Getting Started

Install the Software

Install Pulumi 3.132.0 (or greater) using these instructions.
Install the Pulumi VS Code extension from the Visual Studio Marketplace.

Debugging Pulumi IaC Programs

For simple Pulumi programs, it’s easy enough to see what will happen when you look at the IaC code. But as teams build more complex cloud infrastructure components, fully taking advantage of Pulumi’s support for all of the rich componentization and control flow features in their language of choice, it can become more challenging to diagnose why a particular input value is being used for a cloud infrastructure resource, and what set of decisions were made by the infrastructure components to arrive at that. The pulumi CLI recently added support for a new --attach-debugger flag, which allows attaching a debugger to any language processes that are run as part of a Pulumi operation - regardless of the language. We’ve now added native VS Code debugger extension support for it as well, so that you can debug your Pulumi programs in any language with just a couple of clicks.

Start Debugging

The Pulumi VS Code Extension allows you to run and debug Pulumi programs with ease. Open a new or existing Pulumi project in VS Code and use an automatic or customized launch configuration.

Select the Run and Debug icon.
Choose Show all automatic debug configurations.
Select “Pulumi…” then “pulumi preview” or “pulumi up”.

Pulumi automatically generates a debug configuration for the current stack, making it simple to get started. You can also set breakpoints in your program and use the full functionality of the VS Code debugger.

You can now step through your code, set breakpoints on resource constructors or inside apply callbacks, and see the callstack and the values of variables within the VS Code debugger.

Create a Launch Configuration

For more control, you can create a customized launch configuration:

Click the gear icon when selecting an automatic debug configuration to customize it.
Alternatively, create a launch.json file and use a configuration template.

You can run without debugging by selecting “Run Without Debugging” from the Run menu or by adding "noDebug": "true" to your configuration.

Using Pulumi ESC

Pulumi ESC manages secrets and configuration for your applications and infrastructure within a centralized and composable secrets store. The environments are stored securely within Pulumi Cloud, and can be edited and opened from the Pulumi Cloud console or from the pulumi CLI or REST API.

For developers working with both their application infrastructure code as well as their Pulumi ESC environments, it is convenient to be able to edit ESC environments and the secrets they contain directly within the IDE without switching contexts.

The Pulumi ESC integration in the Pulumi VS Code extensions makes this easy. You can navigate through your environments, see all the projects and versions, and directly edit the source of the environments within VS Code. When you save, the edit to the environment is saved directly back into Pulumi ESC as a new version. And while you have it open in VS Code, you get all the amazing software engineering productivity feature of your IDE - colorization, completion lists, error checking, goto definition, find references and much more!

Open the ESC Explorer

From the primary sidebar, open the “Pulumi ESC Explorer” view and click “Login” to authenticate to Pulumi Cloud. Once logged in, you’ll see a tree of your organizations and environments.

Key Capabilities

Easily create or edit environments by clicking the plus sign on an organization or project node. The extension opens an editor where you can define and save new environment revisions. The IntelliSense functionality will guide you on various values, such as completion for setting up provider configuration.

You can also:

Run your application in the context of your environment by hovering over it and selecting the terminal icon

Delete environments by hovering over them and selecting the delete icon
Decrypt environments for cloning or moving
View environment versions over time
Compare different environments or environment versions
Use the search icon to quickly find environments
Tag specific environment revisions for easy reference.

Manage infrastructure from your editor

Pulumi brings infrastructure as code and Pulumi ESC into the tools you already work in. Build, deploy, and debug your stacks, then manage secrets and configuration in Pulumi Cloud.

Get started

Pulumi YAML Support

Along with support for many popular programming languages including TypeScript, Python, Go, .NET and Java - Pulumi also supports a simple declarative YAML language for defining and managing infrastructure. VS Code already has great support for all the other languages Pulumi supports, but for Pulumi YAML, the Pulumi VS Code Extension offers great support for working in this YAML dialect, including support for type checking, completion lists, colorization and more.

Building on a Pulumi YAML Language Server Protocol (LSP) available for use in any IDE environment, the Pulumi YAML IDE support providers a wide range of editor integrations.

Type Checking

The Pulumi YAML LSP provides contextual warnings and errors directly within the editor.

Warnings: Alerts when a variable is defined but never referenced.
Errors:
- Invalid YAML document detection.
- Reference errors for non-existent variables.
- Duplicate names for variables/resources.

Hover Tooltips

Hover over a resource type token or function type token to see a popup with descriptions, helping you understand your resources and functions better.

Completion Lists

Experience semantic completion for:

Predefined Pulumi YAML keys such as “resources” or “properties”.
Resource properties or function arguments.
Type tokens for resources or functions.
Structured variables (e.g., referencing properties within complex objects).

Wrapping up

This extension is currently in public beta, and we’re eager to hear from the Pulumi community. Your feedback is crucial in helping us shape the future of the extension. If you have suggestions for new features or encounter any bugs, please open an issue.

What’s Next

We’re committed to making Pulumi development as efficient and enjoyable as possible. In the coming months, expect even more features and improvements, all aimed at helping you work smarter and faster. Stay tuned for updates as we continue to build out this powerful extension and bring new capabilities to the Pulumi platform.

Introducing Pulumi ESC SDK: Streamline Your Application Secrets Management

Arun Loganathan — Wed, 05 Jun 2024 00:00:00 -0400

Managing secrets and application configurations effectively is crucial for building secure and maintainable software. However, developers often face challenges such as hardcoded credentials, configuration inconsistencies, and security risks. Pulumi Environments Secrets and Configuration (ESC) simplifies the management of sensitive data and configuration across your entire application lifecycle. Today, we’re thrilled to introduce the official Pulumi ESC SDK in TypeScript/JavaScript, Go, and Python, making it even easier to integrate ESC directly into your applications.

Primer on Pulumi ESC

Pulumi ESC (Environments, Secrets, and Configuration) provides a developer-first solution to simplify how you manage sensitive data and configuration across your entire application lifecycle. It’s a fully managed solution allowing teams to generate dynamic cloud provider credentials, aggregate secrets and configuration from multiple sources, and manage them through composable collections called “environments.” These environments can be consumed from anywhere, making Pulumi ESC ideal for any application and development workflow. Additionally, while Pulumi ESC works independently to eliminate duplication and reduce drift and sprawl of secrets and configuration for all your applications, it also integrates smoothly with Pulumi Infrastructure as Code (IaC) to enhance these capabilities within the Pulumi ecosystem.

Introducing the Pulumi ESC SDK

We’re excited to unveil the official Pulumi ESC SDK, making it even easier to harness the power of ESC directly within your applications using your favorite programming languages. The SDK provides a simple programmatic interface to all of Pulumi ESC’s robust features, allowing you to:

Manage the Entire Lifecycle of Your Environments: Create new environments, list existing ones, and easily update or delete them as your needs evolve. You can even add version tags to your environments, making it simple to track changes and roll back to previous states if needed.
Seamlessly Integrate Secrets and Configurations: Securely access and utilize secrets and configurations within your applications, ensuring consistency of configuration across environments. The SDK provides a streamlined way to fetch the information you need, whether it’s dynamic cloud credentials, database connection strings, feature flags, or any other sensitive data.

The Pulumi ESC SDK streamlines secret and configuration management, promoting secure handling best practices. It handles the secure storage and retrieval of your sensitive data at runtime, eliminating the need to store credentials long-term and minimizing the risk of accidental exposure. The SDK is a natural extension of your development workflow, offering familiar objects and methods, smooth integration with your IDE, and the benefits of type safety such as compile-time checks and helpful code suggestions that reduce errors and accelerate development.

How to Get Started

Here are a few examples of how to use the ESC SDK for various languages:

import * as esc from "@pulumi/esc-sdk";

async function main() {
 const PULUMI_ACCESS_TOKEN = process.env.PULUMI_ACCESS_TOKEN!;
 const orgName = process.env.PULUMI_ORG!;
 const config = new esc.Configuration({ accessToken: PULUMI_ACCESS_TOKEN });
 const client = new esc.EscApi(config);

 const envName = "sdk-typescript-example";

 // Create a new environment
 await client.createEnvironment(orgName, envName);

 const envDef: esc.EnvironmentDefinition = {
 values: {
 my_secret: {
 "fn::secret": "shh! don't tell anyone",
 },
 },
 };

 // Update the environment with the new definition
 await client.updateEnvironment(orgName, envName, envDef);

 // Open and read the environment
 const openEnv = await client.openAndReadEnvironment(orgName, envName);

 if (!openEnv) {
 console.error("Failed to open and read the environment");
 return;
 }

 // Access the value of the secret
 const secretValue = openEnv.values?.my_secret;
 console.log(`Secret value: ${secretValue}\n`);

 // List all the environments in the organization
 const orgEnvs = await client.listEnvironments(orgName);
 if (!orgEnvs || !orgEnvs.environments) {
 console.log("No environments found");
 return;
 }

 for (const env of orgEnvs.environments) {
 console.log(`Environment: ${env.name}`);
 }
}

(async ()=>{
 await main();
})();

import pulumi_esc_sdk as esc
import os

accessToken = os.getenv("PULUMI_ACCESS_TOKEN")

orgName = os.getenv("PULUMI_ORG")

configuration = esc.Configuration(access_token=accessToken)
client = esc.EscClient(configuration)

envName = "sdk-python-example"

# Create environment
client.create_environment(orgName, envName)

# create a new EnvironmentDefinition with "my_secret" as a secret in values additional_properties
envDef = esc.EnvironmentDefinition(
 imports=[],
 values=esc.EnvironmentDefinitionValues(
 additional_properties={
 "my_secret": {
 "fn::secret": "shh! don't tell anyone"
 }
 }
 )
)

# Update environment
client.update_environment(orgName, envName, envDef)

# Open and read the environment

env, values, yaml = client.open_and_read_environment(orgName, envName)

secret = values["my_secret"]
print(f'Secret: {secret}\n')

# List environments

environments = client.list_environments(orgName)

for env in environments.environments:
 print(f'Environment: {env.name}')

package main

import (
 "log"
 "os"

 esc "github.com/pulumi/esc-sdk/sdk/go"
)

func main() {
 accessToken := os.Getenv("PULUMI_ACCESS_TOKEN")
 orgName := os.Getenv("PULUMI_ORG")
 configuration := esc.NewConfiguration()
 escClient := esc.NewClient(configuration)
 authCtx := esc.NewAuthContext(accessToken)

 envName := "sdk-go-example"
 // Create Environment
 err := escClient.CreateEnvironment(authCtx, orgName, envName)
 if err != nil {
 log.Fatalf("Failed to create environment: %v", err)
 }

 // Update Environment with Secrets
 updatePayload := &esc.EnvironmentDefinition{
 Values: &esc.EnvironmentDefinitionValues{
 AdditionalProperties: map[string]interface{}{
 "my_secret": map[string]string{
 "fn::secret": "shh! don't tell anyone",
 },
 },
 },
 }

 _, err = escClient.UpdateEnvironment(authCtx, orgName, envName, updatePayload)
 if err != nil {
 log.Fatalf("Failed to update environment: %v", err)
 }

 // Open and View Secrets
 _, values, err := escClient.OpenAndReadEnvironment(authCtx, orgName, envName)
 if err != nil {
 log.Fatalf("Failed to open environment: %v", err)
 }

 mySecret, ok := values["my_secret"]
 if !ok {
 log.Fatalf("Secret 'my_secret' not found in environment %s", envName)
 }

 log.Printf("my_secret: %v\n", mySecret)

 orgEnvs, err := escClient.ListEnvironments(authCtx, orgName, nil)
 if err != nil {
 log.Fatalf("Failed to list environments: %v", err)
 }

 for _, orgEnv := range orgEnvs.Environments {
 log.Printf("Environment: %v\n", orgEnv.Name)
 }

}

Example Scenarios: Pulumi ESC SDK in Action

Here are some real-world scenarios showcasing how you can leverage the Pulumi ESC SDK:

Database Credentials: Applications can use the ESC SDK to fetch database credentials with a specified TTL, ensuring they always use valid credentials, even as the credentials are rotated in the background. This eliminates service interruptions due to expired credentials.
Secure Serverless Deployments: A Google Cloud Function requiring access to an AWS S3 bucket can leverage the ESC SDK to obtain temporary AWS credentials during runtime. This enables secure cross-cloud interactions without embedding long-lived AWS credentials directly within the function’s code.
Microservices Configuration: With ESC, development teams can manage diverse configurations for their microservices, including API endpoints, retry policies, and logging levels, specific to each environment (development, staging, production). The ESC SDK allows each microservice to retrieve the correct configuration values during runtime, simplifying environment-based customizations.

Get Started Today!

The Pulumi ESC SDK is available now! To learn more and start simplifying your application secrets workflow, check out the Pulumi ESC SDK Documentation. We’re confident that this powerful new tool will streamline your development process, enhance your application security, and empower you to build innovative solutions with confidence. Join us on this journey towards a more secure and manageable future for application secrets and configurations!

OpenID Connect Trust Relationships for Pulumi Cloud

Germán Lena — Fri, 19 Apr 2024 21:45:13 +0000

We are excited to introduce a powerful addition to Pulumi’s authentication capabilities: OpenID Connect (OIDC) Trust Relationships. This feature makes it easy to integrate Pulumi securely into any ecosystem that supports OIDC. By incorporating OIDC, Pulumi is not only extending its compatibility with a broader range of environments but also reinforcing its commitment to delivering top-tier, secure, and scalable solutions to developers and enterprises alike. Whether you are working within CI/CD pipelines or engaging directly with cloud services, this new feature ensures that your infrastructure management is more secure, efficient, and aligned with industry best practices.

Addressing the “Secret Zero” Challenge

A lot of platforms deal with the “secret zero” challenge, which have a weak link where they require a static, long-lived access token. Managing long-lived access tokens has always been a challenge for developers. Often those secrets are set and forgotten, leaving the team to scramble when it comes time to rotate them. This struggle often leads to those secrets not being rotated very often. This long-term access token becomes a weak point in the overall security posture as well as a hassle for developers. With OIDC Trust Relationships, we’re tackling the “secret zero” challenge head-on by introducing exchanging a secure platform token for a short-term Pulumi token. This enhances security and simplifies token management.

Simplified Authentication

OIDC Trust simplifies the authentication process by allowing you to securely request dynamic credentials for Pulumi using your preferred OIDC provider. OIDC is supported across many popular CI/CD systems, such as GitHub, GitLab, Circle CI, and more. In addition, OIDC can be used from within most cloud providers, such as AWS, Azure, GCP, and more.

Enhanced Security with Policy Controls

When you set up an OIDC Trust in Pulumi Cloud, you can set policies to deny or approve token exchanges based on issuer subject or additional claims. We support wildcard matching to create simple policies that support complex authorization scenarios. Based on your specific policy requirements, these policies can be used to issue a token scoped to an organization, team, or personal access.

Seamless Integration with Pulumi ESC

OIDC Trust seamlessly integrate with Pulumi ESC, providing a comprehensive solution for managing infrastructure and access to secrets and configuration. You can now use your native GitHub app token to exchange it for a short-lived Pulumi Token, and ESC will seamlessly exchange it for a cloud token through an ESC environment.

Demo

In this demo, we are going to use Github Actions to retrieve Pulumi credentials and use them to list all the Pulumi ESC Environments in our organization.

Go to the OIDC Issuers Page.
Register Issuer and give a name and (optional) max expiration. Enter the GitHub actions URL https://token.actions.githubusercontent.com.
Add a policy to allow OIDC and configure the sub and audience for your organization and repositories. In the demo, we are using:

Aud: urn:pulumi:org:organization
Sub: repo:organization/repo:*

Create a GitHub action. Here is a sample code. Make sure to substitute your organization in the pulumi/auth-actions organization parameter.

name: Pulumi ESC List Environments
on:
 workflow_dispatch:

permissions:
 id-token: write
 contents: read

jobs:
 run_cron_job:
 runs-on: ubuntu-20.04
 timeout-minutes: 30

 steps:
 - name: Checkout repo
 uses: actions/checkout@v3

 - name: Install pulumi
 uses: pulumi/actions@v4

 - name: Install deps
 run: yarn

 - uses: pulumi/auth-actions@v1
 with:
 organization: organization
 requested-token-type: urn:pulumi:token-type:access_token:organization

 - name: Login to Pulumi
 run: pulumi login
 env:
 PULUMI_ACCESS_TOKEN: ${{ env.PULUMI_ACCESS_TOKEN }}

 - name: List all Pulumi ESC Environments
 run: pulumi env ls

Go to GitHub Actions page, and run the workflow you just created.

Pulumi Blog: Cleve Littlefield

Announcing AWS Systems Manager - Parameter Store Support for Pulumi ESC

How does it work?

Demo

Conclusion

Announcing the Pulumi Visual Studio Code Extension

Getting Started

Install the Software

Debugging Pulumi IaC Programs

Start Debugging

Create a Launch Configuration

Using Pulumi ESC

Open the ESC Explorer

Key Capabilities

Pulumi YAML Support

Type Checking

Hover Tooltips

Completion Lists

Wrapping up

What’s Next

Introducing Pulumi ESC SDK: Streamline Your Application Secrets Management

Primer on Pulumi ESC

Introducing the Pulumi ESC SDK

How to Get Started

Example Scenarios: Pulumi ESC SDK in Action

Get Started Today!

OpenID Connect Trust Relationships for Pulumi Cloud

Addressing the “Secret Zero” Challenge

Simplified Authentication

Enhanced Security with Policy Controls

Seamless Integration with Pulumi ESC

Demo

Next steps