Pulumi Blog: Github

Expanded Version Control Support in Pulumi Cloud

Luke Ward — Mon, 09 Mar 2026 00:00:00 +0000

Your version control provider shouldn’t limit your infrastructure workflows. Pulumi Cloud now works with GitHub, GitHub Enterprise Server, Azure DevOps, and GitLab. Every team gets the same deployment pipelines, PR previews, and AI-powered change summaries regardless of where their code lives.

Connect multiple providers and accounts

You can connect multiple VCS providers to a single Pulumi organization simultaneously, like GitHub, GitLab, and Azure DevOps all at once. You can also connect multiple accounts of the same provider, such as two separate GitHub organizations or two GitLab groups. This means teams that work across different repositories, providers, or organizational boundaries can manage everything from one place.

GitHub Enterprise Server is currently limited to one connection per Pulumi organization.

What your team can do

Deploy on every push

Connect a repository to a stack, and infrastructure deploys automatically when you push. Configure path filters to trigger only when relevant files change, and manage environment variables and secrets directly in Pulumi Cloud. No external CI/CD pipeline required.

Preview changes on pull requests

Every pull request gets an infrastructure preview so reviewers can see exactly what will change before merging. The preview runs the same Pulumi operations your deployment would, giving your team confidence that a merge won’t break anything.

Neo explains your changes

Neo posts AI-generated summaries on your pull requests explaining what infrastructure changes mean in plain language. Reviewers who aren’t Pulumi experts can still understand the impact of a change without reading resource diffs.

Let Neo open pull requests for you

Ask Neo to make infrastructure changes and it opens pull requests directly against your connected repositories. Describe what you want in natural language, and Neo writes the code, opens the PR, and kicks off a preview, all without leaving Pulumi Cloud.

Detect and fix drift

Schedule drift detection to catch out-of-band changes automatically. When someone modifies infrastructure outside of your Pulumi programs, drift detection flags the difference so your team can remediate before it causes issues.

Secure authentication

Pulumi Cloud authenticates with your VCS provider using OIDC or OAuth so no long-lived credentials need to be stored. Short-lived tokens keep your deployment pipelines secure without manual secret rotation.

Set up new projects from your VCS

The new project wizard discovers your organizations, repositories, and branches so you can scaffold and deploy a new stack without leaving Pulumi Cloud. Pick your repo, choose a branch, and you’re ready to deploy.

Getting started

An org admin configures the integration under Settings > Version Control.
Authorize with your VCS provider.
Deploy infrastructure with first-class workflows.

For setup details, see the docs for GitHub, GitHub Enterprise Server, Azure DevOps, and GitLab.

Connect your VCS

How We Load Data into Snowflake in Seconds with Pulumi

Pablo Seibelt — Mon, 23 Feb 2026 00:00:00 +0000

When you manage dozens of data-loading pipelines, copying and pasting IaC configurations between them is a recipe for mishap. IAM policies can drift, naming conventions diverge, and every new source is a new opportunity to make a mistake — not to mention compound the problem of duplication. In this post, we’ll show you how you can identify and encapsulate common patterns into composable components and walk through the production lessons we’ve learned running 25+ pipelines for over three years.

What we’ll cover

If you’re loading data into Snowflake and want reusable, composable infrastructure, this post is for you. Here’s what we’ll cover:

Handling and validating GitHub webhooks with AWS Lambda
Streaming webhook payloads directly into Snowflake with Amazon Data Firehose
Wiring it all up with a reusable Pulumi ComponentResource

The companion template also includes S3 auto-ingest and batch loading patterns, which we’ll cover in upcoming posts. We also use Pulumi ESC to handle authentication to both AWS and Snowflake using OpenID Connect.

Our own Josh Kodroff wrote an excellent introduction to Snowpipe with Pulumi. This post builds on his work using the newest Snowflake and AWS provider APIs and the direct Firehose-to-Snowflake destination, which wasn’t available when Josh wrote his post. Some resource names and grant patterns will also differ if you’re comparing the two.

Architecture overview

The following diagram shows the architecture in more detail:

GitHub sends webhook events to a Lambda Function URL.
Lambda validates the HMAC (Hash-based Message Authentication Code) signature and forwards the payload to Amazon Data Firehose.
Firehose streams records directly into Snowflake via the Snowpipe Streaming API. Data appears in Snowflake within seconds.
S3 is used only as a backup destination for failed records.

The direct Firehose-to-Snowflake destination is an AWS-native feature that works with any Snowflake account.

Project setup

Start a new Pulumi Python project and choose uv for dependency management when prompted:

mkdir snowpipe-data-loading && cd snowpipe-data-loading
pulumi new python

Notice Pulumi.yaml shows uv as your selected toolchain:

name: snowpipe-data-loading
runtime:
 name: python
 options:
 toolchain: uv

Add the provider dependencies for AWS, Snowflake, GitHub, and the Random and TLS providers:

uv add pulumi-aws pulumi-snowflake pulumi-github pulumi-random pulumi-tls

That’s it. uv creates the virtual environment and lockfile automatically, and Pulumi uses uv run under the hood to execute your program.

All examples in this post are in Python, but Pulumi supports multiple languages. You can implement the same components in TypeScript, Go, .NET, Java, or YAML.

Managing credentials with Pulumi ESC

This project needs credentials for AWS, Snowflake, and GitHub. Rather than managing long-lived secrets locally, we can use Pulumi ESC to obtain dynamic, short-lived OIDC credentials for AWS and Snowflake at runtime. When you run pulumi up, ESC exchanges a Pulumi-issued OIDC token for temporary credentials from each provider and injects them into your stack config automatically. If you prefer not to use ESC, you can set credentials directly with pulumi config set --secret.

Here is a single ESC environment that handles all three providers:

values:
 aws:
 login:
 fn::open::aws-login:
 oidc:
 duration: 1h
 roleArn: arn:aws:iam::123456789012:role/pulumi-esc-oidc
 sessionName: pulumi-snowpipe-demo
 snowflake:
 login:
 fn::open::snowflake-login:
 oidc:
 account: <your-snowflake-account>
 user: ESC_SERVICE_USER
 organizationName: <your-org-name>
 accountName: <your-account-name>
 environmentVariables:
 AWS_ACCESS_KEY_ID: ${aws.login.accessKeyId}
 AWS_SECRET_ACCESS_KEY: ${aws.login.secretAccessKey}
 AWS_SESSION_TOKEN: ${aws.login.sessionToken}
 SNOWFLAKE_USER: ${snowflake.login.user}
 SNOWFLAKE_TOKEN: ${snowflake.login.token}
 pulumiConfig:
 aws:region: us-west-2
 snowflake:organizationName: ${snowflake.organizationName}
 snowflake:accountName: ${snowflake.accountName}
 snowflake:authenticator: OAUTH
 snowflake:role: PULUMI_DEPLOYER
 github:token:
 fn::secret: <your-github-pat>
 github:owner: <your-github-org-or-user>

Then reference the environment from your stack config:

# Pulumi.<stack>.yaml
environment:
 - <project>/my-snowpipe-env
config:
 snowpipe-data-loading:database: LANDING_ZONE_WEBHOOKS
 snowpipe-data-loading:environment: dev
 snowpipe-data-loading:webhook-repo: <your-test-repo-name>
 snowflake:previewFeaturesEnabled:
 - snowflake_table_resource

Depending on your preferences, you can split credentials into separate per-provider environments and compose them with imports and reuse across stacks.

To set up OIDC trust for each provider, see the AWS OIDC guide and the Snowflake OIDC login guide. For GitHub authentication options (fine-grained PATs, classic PATs, or GitHub Apps), see the pulumi-github provider docs.

Shared infrastructure

The direct streaming pipeline needs an S3 bucket for backup/errors, a Snowflake database, and a schema.

Building the direct ingestion ComponentResource

Amazon Data Firehose supports Snowflake as a native destination via the Snowpipe Streaming API. Firehose streams records directly into Snowflake.

The Lambda handler

The Lambda function is the entry point for GitHub webhooks. It validates the HMAC-SHA256 signature, wraps the payload in an envelope with the event type, and forwards it to Firehose. Create this as lambda/webhook_handler.py:

import hashlib
import hmac
import json
import os

import boto3

firehose = boto3.client("firehose")

STREAM_NAME = os.environ["FIREHOSE_STREAM_NAME"]
WEBHOOK_SECRET = os.environ["WEBHOOK_SECRET"]


def handler(event, context):
 body = event.get("body", "")
 signature = (event.get("headers") or {}).get("x-hub-signature-256", "")

 # Validate HMAC-SHA256 signature
 expected = "sha256=" + hmac.new(
 WEBHOOK_SECRET.encode(), body.encode(), hashlib.sha256
 ).hexdigest()

 if not hmac.compare_digest(expected, signature):
 return {"statusCode": 401, "body": "Invalid signature"}

 github_event = (event.get("headers") or {}).get("x-github-event", "unknown")

 # Wrap in envelope - newline-delimited for S3 backup where Firehose concatenates records
 record = json.dumps({
 "github_event": github_event,
 "payload": json.loads(body),
 }) + "\n"

 firehose.put_record(
 DeliveryStreamName=STREAM_NAME,
 Record={"Data": record.encode()},
 )

 return {"statusCode": 200, "body": "OK"}

The envelope format {"github_event": "<type>", "payload": {...}}\n is important. The github_event field (e.g., push, pull_request, star) comes from the x-github-event header and lets downstream queries filter by event type. The trailing newline delimits records in the S3 backup destination, where Firehose concatenates them into files.

Another interesting part of this snippet is that instead of manually creating a secret which would have to be copy-pasted into both your webhook configuration and your Lambda environment, we are using random.RandomPassword to generate and store it securely it in Pulumi state.

The secret is automatically wired to both the Lambda env var and the GitHub webhook config, and it rotates cleanly if you ever need to replace it.

Why direct Firehose to Snowflake?

Data like this is normally written to and loaded from Amazon S3. But with an S3 intermediate path, you must wait for Firehose to buffer records (60 seconds), then for Snowpipe to detect the new file and load it. Total latency: about two minutes. With the direct Snowflake destination, Firehose uses the Snowpipe Streaming API to insert records as soon as they arrive, in seconds.

The direct path also removes the need for S3 event notifications, SQS queues, external stages, and pipe resources. S3 is still used, but only as a backup destination for failed records.

The DirectSnowflakeIngestion component

The component creates everything needed for the direct path: a TLS key pair for Snowflake authentication, a Snowflake service user with least-privilege grants, the landing table, a Firehose delivery stream with destination="snowflake", and a Lambda function with a public URL. Create an empty components/__init__.py so that Python treats the directory as a package.

The full component lives in components/direct_snowflake_ingestion.py:

import json
from dataclasses import dataclass

import pulumi
import pulumi_aws as aws
import pulumi_snowflake as snowflake
import pulumi_tls as tls

# In the example repository, you will find this class imported from a common library file instead
@dataclass
class ColumnDef:
 """Column definition for a Snowflake table."""

 name: str
 type: str
 nullable: bool = True

def strip_pem_headers(pem: str) -> str:
 """Remove PEM header/footer lines, returning only the base64 content."""
 lines = pem.strip().split("\n")
 return "".join(lines[1:-1])


@dataclass
class DirectSnowflakeIngestionArgs:
 bucket_arn: pulumi.Input[str]
 bucket_name: pulumi.Input[str]
 database: pulumi.Input[str]
 schema_name: pulumi.Input[str]
 table_name: str
 table_columns: list[ColumnDef]
 snowflake_account_url: pulumi.Input[str]
 snowflake_role_name: str
 lambda_code: pulumi.Archive
 lambda_handler: str
 lambda_environment: dict[str, pulumi.Input[str]]
 table_comment: str = ""
 s3_prefix: str = "direct-webhooks"
 s3_backup_mode: str = "FailedDataOnly"
 buffering_interval: int = 0
 buffering_size: int = 1
 retry_duration: int = 60
 data_loading_option: str = "VARIANT_CONTENT_AND_METADATA_MAPPING"
 content_column_name: str = "CONTENT"
 metadata_column_name: str = "METADATA"

class DirectSnowflakeIngestion(pulumi.ComponentResource):
 function_url: pulumi.Output[str]
 firehose_stream_name: pulumi.Output[str]
 snowflake_user_name: pulumi.Output[str]

 def __init__(
 self,
 name: str,
 args: DirectSnowflakeIngestionArgs,
 opts: pulumi.ResourceOptions | None = None,
 ):
 super().__init__(
 "snowpipe:direct:DirectSnowflakeIngestion", name, {}, opts
 )

 # --- TLS key pair for Snowflake auth ---
 key_pair = tls.PrivateKey(
 f"{name}-keypair",
 algorithm="RSA",
 rsa_bits=2048,
 opts=pulumi.ResourceOptions(parent=self),
 )

 # --- Snowflake role, user, and grants ---
 sf_role = snowflake.AccountRole(
 f"{name}-sf-role",
 name=args.snowflake_role_name,
 opts=pulumi.ResourceOptions(parent=self),
 )

 user_name = f"FIREHOSE_{name.upper().replace('-', '_')}_USER"
 sf_user = snowflake.ServiceUser(
 f"{name}-sf-user",
 name=user_name,
 login_name=user_name,
 default_role=sf_role.name,
 rsa_public_key=key_pair.public_key_pem.apply(strip_pem_headers),
 opts=pulumi.ResourceOptions(parent=self),
 )

 # Landing table
 table = snowflake.Table(
 f"{name}-table",
 name=args.table_name,
 database=args.database,
 schema=args.schema_name,
 comment=args.table_comment,
 columns=[
 snowflake.TableColumnArgs(
 name=col.name, type=col.type, nullable=col.nullable,
 )
 for col in args.table_columns
 ],
 opts=pulumi.ResourceOptions(parent=self),
 )

 # Grants: DB USAGE, schema USAGE, table INSERT+SELECT
 snowflake.GrantPrivilegesToAccountRole(
 f"{name}-grant-db-usage",
 account_role_name=sf_role.name,
 privileges=["USAGE"],
 on_account_object=snowflake.GrantPrivilegesToAccountRoleOnAccountObjectArgs(
 object_type="DATABASE",
 object_name=args.database,
 ),
 opts=pulumi.ResourceOptions(parent=self),
 )

 snowflake.GrantPrivilegesToAccountRole(
 f"{name}-grant-schema-usage",
 account_role_name=sf_role.name,
 privileges=["USAGE"],
 on_schema=snowflake.GrantPrivilegesToAccountRoleOnSchemaArgs(
 schema_name=pulumi.Output.all(
 args.database, args.schema_name
 ).apply(lambda parts: f'"{parts[0]}"."{parts[1]}"'),
 ),
 opts=pulumi.ResourceOptions(parent=self),
 )

 table_name = args.table_name
 snowflake.GrantPrivilegesToAccountRole(
 f"{name}-grant-table",
 account_role_name=sf_role.name,
 privileges=["INSERT", "SELECT"],
 on_schema_object=snowflake.GrantPrivilegesToAccountRoleOnSchemaObjectArgs(
 object_type="TABLE",
 object_name=pulumi.Output.all(
 args.database, args.schema_name
 ).apply(
 lambda parts: f'"{parts[0]}"."{parts[1]}"."{table_name}"'
 ),
 ),
 opts=pulumi.ResourceOptions(parent=self, depends_on=[table]),
 )

 # --- Firehose IAM role (S3 backup write) ---
 firehose_role = aws.iam.Role(
 f"{name}-firehose-role",
 assume_role_policy=json.dumps({
 "Version": "2012-10-17",
 "Statement": [{
 "Effect": "Allow",
 "Action": "sts:AssumeRole",
 "Principal": {"Service": "firehose.amazonaws.com"},
 }],
 }),
 opts=pulumi.ResourceOptions(parent=self),
 )

 aws.iam.RolePolicy(
 f"{name}-firehose-s3-policy",
 role=firehose_role.id,
 policy=args.bucket_arn.apply(
 lambda arn: json.dumps({
 "Version": "2012-10-17",
 "Statement": [{
 "Effect": "Allow",
 "Action": [
 "s3:AbortMultipartUpload",
 "s3:GetBucketLocation",
 "s3:GetObject",
 "s3:ListBucket",
 "s3:ListBucketMultipartUploads",
 "s3:PutObject",
 ],
 "Resource": [arn, f"{arn}/*"],
 }],
 })
 ),
 opts=pulumi.ResourceOptions(parent=self),
 )

 # --- Firehose delivery stream (Snowflake destination) ---
 stream = aws.kinesis.FirehoseDeliveryStream(
 f"{name}-firehose",
 destination="snowflake",
 snowflake_configuration=aws.kinesis.FirehoseDeliveryStreamSnowflakeConfigurationArgs(
 account_url=args.snowflake_account_url,
 database=args.database,
 schema=args.schema_name,
 table=args.table_name,
 role_arn=firehose_role.arn,
 user=sf_user.name,
 private_key=key_pair.private_key_pem_pkcs8.apply(
 strip_pem_headers
 ),
 data_loading_option=args.data_loading_option,
 content_column_name=args.content_column_name,
 metadata_column_name=args.metadata_column_name,
 s3_backup_mode=args.s3_backup_mode,
 buffering_size=args.buffering_size,
 buffering_interval=args.buffering_interval,
 retry_duration=args.retry_duration,
 snowflake_role_configuration=aws.kinesis.FirehoseDeliveryStreamSnowflakeConfigurationSnowflakeRoleConfigurationArgs(
 enabled=True,
 snowflake_role=args.snowflake_role_name,
 ),
 s3_configuration=aws.kinesis.FirehoseDeliveryStreamSnowflakeConfigurationS3ConfigurationArgs(
 bucket_arn=args.bucket_arn,
 role_arn=firehose_role.arn,
 prefix=f"{args.s3_prefix}/backup/",
 error_output_prefix=f"{args.s3_prefix}/errors/",
 ),
 ),
 opts=pulumi.ResourceOptions(parent=self, depends_on=[table]),
 )

 # --- Lambda function + Function URL ---
 lambda_role = aws.iam.Role(
 f"{name}-lambda-role",
 assume_role_policy=json.dumps({
 "Version": "2012-10-17",
 "Statement": [{
 "Effect": "Allow",
 "Action": "sts:AssumeRole",
 "Principal": {"Service": "lambda.amazonaws.com"},
 }],
 }),
 opts=pulumi.ResourceOptions(parent=self),
 )

 aws.iam.RolePolicyAttachment(
 f"{name}-lambda-basic-execution",
 role=lambda_role.name,
 policy_arn="arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole",
 opts=pulumi.ResourceOptions(parent=self),
 )

 aws.iam.RolePolicy(
 f"{name}-lambda-firehose-policy",
 role=lambda_role.id,
 policy=stream.arn.apply(
 lambda arn: json.dumps({
 "Version": "2012-10-17",
 "Statement": [{
 "Effect": "Allow",
 "Action": ["firehose:PutRecord"],
 "Resource": [arn],
 }],
 })
 ),
 opts=pulumi.ResourceOptions(parent=self),
 )

 env_vars = {
 **args.lambda_environment,
 "FIREHOSE_STREAM_NAME": stream.name,
 }

 fn = aws.lambda_.Function(
 f"{name}-handler",
 runtime="python3.11",
 handler=args.lambda_handler,
 role=lambda_role.arn,
 timeout=30,
 code=args.lambda_code,
 environment=aws.lambda_.FunctionEnvironmentArgs(
 variables=env_vars,
 ),
 opts=pulumi.ResourceOptions(parent=self),
 )

 fn_url = aws.lambda_.FunctionUrl(
 f"{name}-function-url",
 function_name=fn.name,
 authorization_type="NONE",
 opts=pulumi.ResourceOptions(parent=self),
 )

 aws.lambda_.Permission(
 f"{name}-function-url-permission",
 action="lambda:InvokeFunctionUrl",
 function=fn.name,
 principal="*",
 function_url_auth_type="NONE",
 opts=pulumi.ResourceOptions(parent=self),
 )

 # --- Outputs ---
 self.function_url = fn_url.function_url
 self.firehose_stream_name = stream.name
 self.snowflake_user_name = sf_user.name

 self.register_outputs({
 "function_url": self.function_url,
 "firehose_stream_name": self.firehose_stream_name,
 "snowflake_user_name": self.snowflake_user_name,
 })

A few things to note:

TLS key pair for authentication. The component generates an RSA key pair using the pulumi-tls provider. The public key is assigned to the Snowflake service user; the private key (PKCS#8 format, base64-encoded) is passed to Firehose. No passwords or OAuth tokens are stored.
ServiceUser instead of User. Snowflake service users can’t log in interactively. They authenticate only via key pair, which is exactly what Firehose needs.
destination="snowflake" on Firehose. This tells Firehose to use the Snowpipe Streaming API rather than writing to S3. The s3_configuration block is still required, but only for backup/error records.
Immediate flushing. buffering_interval=0 and buffering_size=1 ensure records are sent to Snowflake as soon as they arrive, minimizing latency. Tune according to your needs.

Amazon Data Firehose does not connect from fixed IP addresses, so you cannot use Snowflake network policies to restrict access by IP. If your Snowflake account uses network policies, you have three options: use AWS PrivateLink (requires Snowflake Business Critical edition), allow public internet access for the Firehose service user, or switch to S3 auto-ingest via Snowpipe which does not require direct network access to Snowflake from Firehose.

Wiring it together

With the component defined, __main__.py wires the direct ingestion pipeline:

import pulumi
import pulumi_aws as aws
import pulumi_github as github
import pulumi_random as random
import pulumi_snowflake as snowflake
from components.direct_snowflake_ingestion import (
 DirectSnowflakeIngestion, DirectSnowflakeIngestionArgs,
)
from components.snowpipe_pipeline import ColumnDef

config = pulumi.Config()
database_name = config.get("database") or "LANDING_ZONE_WEBHOOKS"
environment = config.get("environment") or "dev"

# --- Shared infrastructure ---

# S3 bucket for backup/errors
bucket = aws.s3.Bucket(
 "data-landing-bucket",
)

# Snowflake database and schema
database = snowflake.Database("demo-database", name=database_name)
schema = snowflake.Schema(
 "demo-schema",
 name="GITHUB",
 database=database.name,
)

# --- Direct ingestion pipeline ---

webhook_repo = config.require("webhook-repo")

# Snowflake account URL for Firehose configuration
snowflake_config = pulumi.Config("snowflake")
snowflake_account_url = (
 f"https://{snowflake_config.require('organizationName')}"
 f"-{snowflake_config.require('accountName')}"
 f".snowflakecomputing.com"
)

# Landing table columns: CONTENT (webhook JSON) + METADATA (Firehose metadata)
DIRECT_COLUMNS = [
 ColumnDef(name="CONTENT", type="VARIANT", nullable=True),
 ColumnDef(name="METADATA", type="VARIANT", nullable=True),
]

# Step 1: Generate webhook secret for HMAC validation
direct_webhook_secret = random.RandomPassword(
 "github-direct-webhook-secret", length=32, special=False
)

# Step 2: Direct ingestion pipeline - Lambda validates, Firehose streams to Snowflake
direct = DirectSnowflakeIngestion(
 "github-webhooks-direct",
 DirectSnowflakeIngestionArgs(
 bucket_arn=bucket.arn,
 bucket_name=bucket.bucket,
 database=database.name,
 schema_name=schema.name,
 table_name="REPOSITORY_EVENTS_DIRECT",
 table_columns=DIRECT_COLUMNS,
 table_comment="GitHub webhook events loaded via direct Firehose to Snowflake",
 snowflake_account_url=snowflake_account_url,
 snowflake_role_name="FIREHOSE_DIRECT_LOADER",
 lambda_code=pulumi.AssetArchive({
 "webhook_handler.py": pulumi.FileAsset("lambda/webhook_handler.py"),
 }),
 lambda_handler="webhook_handler.handler",
 lambda_environment={"WEBHOOK_SECRET": direct_webhook_secret.result},
 ),
)

# Step 3: GitHub webhook - sends events to the Lambda Function URL
github.RepositoryWebhook(
 "github-direct-webhook",
 repository=webhook_repo,
 configuration=github.RepositoryWebhookConfigurationArgs(
 url=direct.function_url,
 content_type="json",
 secret=direct_webhook_secret.result,
 ),
 events=["push", "pull_request", "issues", "star"],
)

# Exports
pulumi.export("webhook_url", direct.function_url)
pulumi.export("firehose_stream", direct.firehose_stream_name)

That’s the entire pipeline. One component, one GitHub webhook, one secret. The DirectSnowflakeIngestion component handles the TLS key pair, Snowflake service user, landing table, Firehose stream, and Lambda function internally, and now you can reuse this component for as many pipelines as you need.

The full code for this example is available on GitHub:

github.com/pulumi-demos/examples/tree/main/python/aws-snowflake-data-loading-real-time

Testing the pipeline

Deploy the stack:

pulumi up

The entire stack deploys in about two minutes. Immediately after deployment, you’ll start seeing GitHub events flowing into Snowflake.

Before querying, grant the DATA_READER role to your Snowflake user:

GRANT ROLE DATA_READER TO USER <your-user>;

In production, you can manage this grant through Pulumi, manually, or automatically via SCIM provisioning from your identity provider.

No need to craft test payloads. Just interact with the test repo. Star it, push a commit, or open an issue, then wait about 30 seconds and query Snowflake using the least-privilege reader role:

USE ROLE DATA_READER;

SELECT CONTENT:github_event::STRING AS event_type,
 CONTENT:payload:repository:full_name::STRING AS repo,
 METADATA:IngestionTime::TIMESTAMP AS ingested_at
FROM LANDING_ZONE_WEBHOOKS.GITHUB.REPOSITORY_EVENTS_DIRECT
ORDER BY ingested_at DESC;

You should see rows with event types like star, push, or issues, real GitHub events flowing through the entire pipeline. The METADATA column includes Firehose metadata like IngestionTime, which you can use to track end-to-end latency.

Other loading patterns

Direct streaming is the fastest path, but two other patterns are available in the companion template for different requirements:

S3 auto-ingest via Snowpipe. Firehose buffers to S3, and Snowpipe auto-ingests new files. Latency is about two minutes. Best when you need S3 as the system of record or can’t use direct Snowpipe Streaming.
Batch loading. Your orchestrator (Airflow, Prefect, cron, etc.) runs COPY INTO on a schedule. Best for full control over timing and deduplication.

We’ll walk through both patterns in detail in upcoming posts.

Stamp out data pipelines with components

Encapsulate your Snowflake loading logic in a Pulumi component, then create new pipelines with a handful of configuration values instead of copied files.

Get started

Publishing as reusable components

Once your components are battle-tested, you can share them across teams and projects instead of copying files around.

The most straightforward approach: push your components to a Git repository with a PulumiPlugin.yaml file at the root:

runtime: python
name: snowpipe-components
version: 1.0.0

Consumers add the package to their project with pulumi package add:

pulumi package add github.com/your-org/pulumi-snowpipe@v1.0.0

Pulumi downloads the package and generates typed SDKs automatically. The consumer’s __main__.py imports your components as if they were local, but they’re versioned and pinned.

Pulumi Cloud Private Registry

For organization-level discoverability, publish to the Pulumi Cloud Private Registry with pulumi package publish:

pulumi package publish ./schema.json

This gives you auto-generated API docs, usage tracking across teams, and cross-language SDK generation. Your Python components become usable from TypeScript, Go, and C# without any extra work. Teams browse available components in the Pulumi Cloud console, see who’s using what, and get notified when new versions are published. As an additional benefit, Neo will be able to use these components and build new pipelines in minutes from a natural language request.

Conclusion

ComponentResource is the key abstraction that makes this architecture scale. Instead of copying and pasting resources for each new data source, you instantiate a component with a handful of configuration parameters.

The DirectSnowflakeIngestion component in this post delivers data from GitHub webhooks into Snowflake in seconds: Lambda validates the HMAC signature, Firehose streams directly to Snowflake via the Snowpipe Streaming API, and the TLS key pair is managed entirely within Pulumi. No S3 intermediate, no SQS queues.

The component accepts pluggable Lambda handlers, so swapping GitHub for Stripe webhooks or any other source is just a matter of providing different lambda_code and lambda_environment arguments. We’ve been running this pattern in production for over three years across dozens of pipelines without significant changes to the infrastructure code.

You’ll find the complete example in the GitHub repository.

Automatic Diagram Generation for Always-Accurate Diagrams

Elisabeth Lichtie — Mon, 22 Sep 2025 00:00:00 +0000

Architecture diagrams are one of the most tedious aspects of infrastructure management. Teams spend countless hours manually creating and updating visual representations of their systems, only to see them become outdated the moment infrastructure changes.

When you manage your infrastructure as code, you can automatically generate accurate, up-to-date diagrams directly from your IaC definitions and state. Pulumi’s preview capabilities let you anticipate exactly what changes will result from code modifications before deployment, providing the perfect foundation for automated diagramming workflows.

In this post, we’ll explore three practical approaches to automating diagram generation in your CI/CD pipeline:

Native Pulumi diagramming: Using Pulumi’s built-in diagramming capabilities to generate basic architecture diagrams that you can retrieve from GitHub Actions’ artifact archive
AI-powered diagramming: Leveraging the Claude app with Mermaid and Pulumi preview to generate detailed diagrams and automatically add them to pull requests for infrastructure changes
Pulumi Neo for enterprise diagramming: Using Pulumi’s AI-powered platform for organization-wide diagramming patterns, project-level and organization-level views, and intelligent customization for different stakeholders

These approaches eliminate manual diagramming work while ensuring your documentation stays current with your actual infrastructure, from simple automation to enterprise-scale intelligent diagramming.

Native Pulumi diagramming with GitHub Actions

Pulumi’s built-in pulumi stack graph command generates architecture diagrams directly from your deployed infrastructure state. This approach provides a reliable foundation for automated diagramming since it reflects your actual deployed resources rather than just the code.

Complete example: View the full working example on GitHub.

Here’s a complete GitHub Actions workflow that deploys infrastructure and automatically generates diagrams:

name: Pulumi Deploy & Diagram

on:
 push:
 branches: [main]

jobs:
 deploy:
 name: Deploy & Generate Diagram
 runs-on: ubuntu-latest
 steps:
 - uses: actions/checkout@v4

 - name: Setup Node.js
 uses: actions/setup-node@v4
 with:
 node-version: "20"
 cache: "npm"

 - name: Install dependencies
 run: npm ci

 - uses: pulumi/actions@v5
 with:
 command: up
 stack-name: dev
 env:
 PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}

 - name: Install Pulumi CLI
 run: |
 curl -fsSL https://get.pulumi.com | sh
 echo "$HOME/.pulumi/bin" >> $GITHUB_PATH

 - name: Generate Architecture Diagram
 run: pulumi stack graph architecture.dot --stack dev
 env:
 PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}

 - name: Install Graphviz
 run: sudo apt-get update && sudo apt-get install -y graphviz

 - name: Generate PNG from DOT file
 run: dot -Tpng architecture.dot -o architecture.png

 - name: Upload Architecture Diagrams
 uses: actions/upload-artifact@v4
 with:
 name: architecture-diagrams
 path: |
 architecture.dot
 architecture.png

How the workflow works

This workflow follows a straightforward pattern that ensures your diagrams always reflect your deployed infrastructure:

Deploy first: The workflow uses the official pulumi/actions@v5 action to deploy your infrastructure changes. This ensures the state reflects your latest code.
Generate diagram from state: The pulumi stack graph command reads your deployed stack’s state and generates a DOT format file containing the resource relationships and dependencies.
Convert to visual format: Graphviz converts the DOT file into a PNG image that’s easy to view and share.
Archive as artifacts: Both the raw DOT file and the rendered PNG are uploaded as GitHub Actions artifacts, making them downloadable from the workflow run.

Key advantages and limitations

Highest accuracy: Since diagrams are generated from deployed state rather than code, they reflect your actual infrastructure including any drift or manual changes. This is the biggest advantage of the native approach.

Zero maintenance: Once set up, diagrams update automatically with every deployment without any manual intervention.

Multiple formats: The DOT format can be converted to various output formats (PNG, SVG, PDF) depending on your needs.

Audit trail: Each workflow run preserves the diagram as an artifact, creating a visual history of your infrastructure changes.

However, this approach has some limitations:

Fixed styling: You’re locked into Pulumi’s diagram style and layout algorithms, with limited customization options.

Basic aesthetics: The generated diagrams are functional but not always aesthetically polished compared to hand-crafted or AI-generated alternatives.

Technical focus: Diagrams show all resources and relationships, which can be overwhelming for non-technical stakeholders who need higher-level architectural views.

Accessing your diagrams

After each workflow run, you can download the generated diagrams from the Actions tab in your repository. Click on any workflow run, scroll to the “Artifacts” section, and download the architecture-diagrams package.

For a more sophisticated setup, you could extend this workflow to automatically commit diagrams back to your repository, post them to Slack, or integrate them with your documentation system.

AI-powered diagramming with Claude and Mermaid

The second approach leverages Claude’s ability to analyze Pulumi preview output and generate aesthetically pleasing, contextually relevant diagrams using Mermaid syntax. This method addresses the styling limitations of native Pulumi diagrams while maintaining accuracy through pulumi preview integration.

Complete example: View the full working example on GitHub.

The automated workflow

This approach uses two GitHub Actions workflows working together:

Auto-comment workflow (auto-pr-comment.yml):

name: Auto PR Comment

on:
 pull_request:
 types: [opened]
 paths:
 - 'index.ts'
 - 'Pulumi.yaml'
 - 'Pulumi.*.yaml'

jobs:
 comment:
 if: github.actor != 'claude-code[bot]'
 runs-on: ubuntu-latest
 permissions:
 contents: read
 pull-requests: write
 steps:
 - name: Comment on PR
 uses: actions/github-script@v7
 with:
 github-token: ${{ secrets.PERSONAL_ACCESS_TOKEN }}
 script: |
 github.rest.issues.createComment({
 issue_number: context.issue.number,
 owner: context.repo.owner,
 repo: context.repo.repo,
 body: '@claude update the diagrams to reflect any infrastructure changes.'
 })

Claude integration workflow (claude.yml):

name: Claude Code

on:
 issue_comment:
 types: [created]
 pull_request_review_comment:
 types: [created]

jobs:
 claude:
 if: contains(github.event.comment.body, '@claude')
 runs-on: ubuntu-latest
 permissions:
 contents: write
 pull-requests: write
 issues: write
 steps:
 - name: Checkout repository
 uses: actions/checkout@v4

 - name: Run Claude Code
 uses: anthropics/claude-code-action@beta
 env:
 PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}
 with:
 anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
 allowed_tools: "Bash(npm install),Bash(pulumi stack select prod),Bash(pulumi preview --json),Bash(npx mmdc)"
 custom_instructions: |
 ## Pulumi Infrastructure Diagramming Assistant

 ### Step 1: Analyze Infrastructure State
 - Run `npm install` and `pulumi stack select prod`
 - Execute `pulumi preview --json` to get detailed resource plan
 - Parse JSON output to understand resource relationships and changes

 ### Step 2: Generate/Update Diagrams
 Update Mermaid diagram files in `/diagrams/`:
 - aws-infrastructure.mmd: Technical AWS resource diagram
 - c4-context.mmd: System context diagram
 - c4-container.mmd: Container-level view
 - c4-component.mmd: Component-level architecture

 ### Step 3: Generate SVG Files
 Convert .mmd files to .svg using mmdc CLI

How the AI approach works

Automatic triggering: When a pull request with infrastructure changes is opened, the auto-comment workflow adds a @claude mention, triggering Claude’s analysis.
Infrastructure analysis: Claude runs pulumi preview --json to get detailed information about planned infrastructure changes, including resource relationships and dependencies.
Intelligent diagram generation: Claude analyzes the preview output and updates multiple Mermaid diagrams:
- AWS infrastructure diagram: Technical view showing resources, networking, and data flow
- C4 diagrams: Context, container, and component views for different stakeholder needs
Visual rendering: The mmdc CLI converts Mermaid syntax to SVG files for immediate viewing.
PR integration: Updated diagrams are committed directly to the pull request branch, making them visible in code review.

Example generated diagram

Here’s a sample of the AWS infrastructure diagram that Claude generates:

graph TB
subgraph VPC["myVpc - CIDR: 10.0.0.0/16"]
subgraph PublicSubnet["myPublicSubnet - CIDR: 10.0.1.0/24"]
EC2["myInstance<br/>Type: t2.micro<br/>AMI: Latest Amazon Linux 2"]
end
IGW["myInternetGateway<br/>Internet connectivity"]
SG["mySecurityGroup<br/>HTTP (80) + SSH (22) access"]
end
WEB_USER["Web Users"] -->|HTTP requests| IGW
IGW --> PublicSubnet
SG --> EC2

Key advantages and limitations

Intelligent styling: Claude generates aesthetically pleasing diagrams with consistent formatting, colors, and layout that’s much more polished than native Pulumi output.

Multiple perspectives: Creates different diagram types (technical infrastructure, C4 architecture views) suitable for different audiences and stakeholder needs.

Context awareness: Claude understands the purpose and relationships of resources, creating meaningful groupings and flow representations.

Automated maintenance: Diagrams update automatically with infrastructure changes, staying current without manual intervention.

Integration with code review: Diagrams appear directly in pull requests, making infrastructure changes visible during review.

However, this approach also has limitations:

Potential for inaccuracy: Unlike native Pulumi diagrams that are generated directly from infrastructure state, AI-generated diagrams depend on Claude’s interpretation of preview output. While generally reliable, there’s inherent risk of misinterpretation or missed details that could lead to diagram inaccuracies.

AI dependency: Diagram accuracy is only as good as Claude’s understanding of complex infrastructure relationships, which may occasionally miss nuances or make incorrect assumptions.

Setup complexity: Requires more configuration including API keys, GitHub app installation, and workflow permissions.

Cost considerations: Each diagram update consumes Claude API credits, though costs are typically minimal for most projects.

Preview limitations: Diagrams reflect planned changes from pulumi preview rather than actual deployed state, which may differ if deployments fail.

This approach works best for teams that value diagram aesthetics and need multiple architectural views for different stakeholders, while accepting slightly more complexity and potential accuracy trade-offs in exchange for significantly better visual presentation.

Generate diagrams from your code

You can produce accurate architecture diagrams directly from your Pulumi state and preview, then wire the workflow into any coding agent through the Pulumi MCP server. Start a project to keep your diagrams in sync with your infrastructure.

Get started

Pulumi Neo: Enterprise-scale intelligent diagramming

Pulumi Neo brings the next evolution in infrastructure diagramming, with AI-powered intelligence that enables organization-wide diagram generation. Unlike the previous approaches that focus on individual stacks, Pulumi Neo provides enterprise-scale diagramming capabilities with intelligent customization for different stakeholders.

How the workflow works

Pulumi Neo streamlines enterprise diagramming through an intelligent, organization-aware approach that scales across multiple projects and stakeholders:

Repository setup: Create a dedicated repository to host your architecture diagrams and ensure your GitHub App has access. The repository structure helps Neo understand your diagramming patterns:
- overview/ folder for organization-wide infrastructure views
- stacks/ folder for individual stack diagrams
- Template files that Neo can model new diagrams from
- README documentation explaining the repository’s purpose and conventions

Neo prompt engineering: Use structured prompts to guide Neo’s diagram generation with enterprise-grade specificity. You can also select your architecture diagram repository to tell Neo where to store his work:

1. Scope: Create diagrams for all resources in my prod Pulumi stacks
1. Diagram Types:
- Individual stack diagrams showing resources within each stack
- Master overview diagram showing cross-stack dependencies and data flow
1. Visual Standards:
- Use official AWS/Azure/GCP service icons from their architecture icon sets
- Show resource relationships with labeled connections (security groups, subnets, etc.)
- Group related resources visually (VPC boundaries, availability zones)
1. Content Focus:
- Infrastructure resources only (compute, storage, networking, databases)
- Resource dependencies and data flow directions
- Cross-stack references and shared resources
1. Format: Generate diagrams as Mermaid code that can be rendered in documentation

Automated pull request generation: Neo analyzes your entire organization’s infrastructure state, identifies changes, and automatically creates pull requests with updated diagrams for review and approval.

Complete example: You can see my resulting architecture diagrams repository on GitHub.

Key advantages and limitations

Pulumi Neo offers the most sophisticated approach to infrastructure diagramming, particularly for enterprise environments:

Enterprise-scale intelligence: Neo understands your entire organization’s infrastructure landscape, generating diagrams that show cross-project dependencies and organization-wide resource relationships that individual stack-based approaches cannot capture.

Style customization: Customize your prompts to request different styles of diagrams.

Organizational consistency: Maintains consistent diagramming standards across teams and projects, ensuring visual coherence in large organizations with multiple infrastructure teams.

Cross-stack visibility: Unlike single-stack approaches, Neo provides organization-wide infrastructure views showing how different projects and teams’ resources interconnect.

Template-driven generation: Learns from your existing diagram patterns and templates to maintain consistency with your established visual standards and conventions.

AI-powered context understanding: Goes beyond simple resource mapping to understand business context, automatically highlighting critical paths, security boundaries, and architectural patterns.

However, Neo also has limitations:

AI interpretation risks: Like the Claude approach, AI-generated diagrams may miss nuances or make incorrect assumptions about complex enterprise infrastructure relationships.

Repository access requirements: Neo requires permissions to open pull requests against your repository.

Learning investment: Getting maximum value from Neo requires developing sophisticated prompt engineering skills and understanding how to structure repositories for optimal AI comprehension.

Conclusion

Automating diagram generation transforms one of the most tedious aspects of infrastructure management into a seamless part of your development workflow. All three approaches eliminate the manual overhead of keeping architecture diagrams current while solving the core problem of documentation drift.

Choose native Pulumi diagramming when accuracy is paramount and you need diagrams that perfectly reflect your deployed infrastructure. This approach offers the highest reliability with minimal setup complexity.

Choose AI-powered diagramming when you need polished, stakeholder-friendly diagrams and can accept slightly more complexity for significantly better aesthetics and multiple architectural perspectives.

Choose Pulumi Neo when you need enterprise-scale diagramming with organization-wide consistency, multi-level views across projects and stacks, and intelligent customization for different stakeholders with varying technical backgrounds.

Regardless of which approach you choose, you’ll eliminate hours of manual work while ensuring your documentation stays synchronized with your infrastructure changes. With Pulumi Neo’s AI capabilities now available, intelligent, context-aware diagramming has moved from possibility to reality, making infrastructure documentation both effortless and incredibly powerful.

The future of infrastructure documentation is here: automated, intelligent, and always accurate. With these examples as your foundation, you’re ready to transform how your team approaches architecture visualization at any scale.

Announcing the Pulumi ESC GitHub Action

Komal Ali — Wed, 05 Mar 2025 00:00:00 +0000

We’re excited to share our latest addition to the Pulumi Ecosystem: the Pulumi ESC GitHub Action. This Action lets you inject secrets and configuration securely into your GitHub Actions workflows as they are needed, rather than storing them as static, long-lived secrets.

Why we built it

Secret management continues to be a challenge for organizations of all sizes. Storing long-lived credentials in multiple places (e.g., GitHub, third-party CI systems, and internal servers) often leads to “secret sprawl” and potential security gaps. With Pulumi ESC, you can migrate to dynamic credentials, and ensure secrets don’t persist beyond their necessary lifetime. If your process requires longer-lived static credentials, with ESC you can manage credentials in a central place and rotate them automatically.

By packaging ESC functionality into a GitHub Action, we’re enabling teams to inject secrets and run ESC commands directly within their GitHub workflows. This allows for a streamlined approach to environment management and secrets rotation—without leaving GitHub.

Key benefits

Secure Secrets Injection - Inject secrets into your GitHub Actions workflows as they are needed, rather than storing them as static, long-lived secrets. This dynamic approach reduces the risk of secrets leaking in logs or pulled from unauthorized sources.
Automatic Secret Rotation - Pulumi ESC handles rotation for you behind the scenes, ensuring that any secrets or credentials you inject into GitHub Actions are valid at the time of use, but rotated on a regular schedule to avoid the risks that come with long-lived static secrets.
Streamlined CI/CD Pipeline - In addition to injecting environment variables, you can run any Pulumi ESC command directly in your GitHub Actions workflows. Automate the creation, update, or teardown of ESC Environments as part of your CI/CD process.
Seamless Integration with Pulumi Cloud - Combine the Pulumi ESC GitHub Action with pulumi/auth-actions to authenticate securely with Pulumi Cloud. No need for long-lived tokens — simply leverage short-lived tokens provided at runtime.

How it works

At a high level, the Pulumi ESC GitHub Action can do three things:

Install the Pulumi CLI - By default, the action ensures the Pulumi CLI is installed in your GitHub Actions environment. If you provide a specific version, that version is installed.
Inject all environment variables from an ESC environment - If you specify an environment (for example, tinyco/someProject/myEnv), the action will inject all environment variables and projected files associated with that environment into the GitHub Actions context.
Inject specific environment variables from an ESC environment - If you only need certain keys, you can specify them in the action’s inputs to pick and choose what gets injected, giving you fine-grained control.

Example workflow

Below is a minimal workflow that showcases injecting an entire environment’s variables into your GitHub Actions job:

on:
 - pull_request

permissions:
 id-token: write
 contents: read

jobs:
 test-env-injection:
 runs-on: ubuntu-latest
 steps:
 - name: Check out repository
 uses: actions/checkout@v4
 - name: Authenticate with Pulumi Cloud
 uses: pulumi/auth-actions@v1
 with:
 organization: pulumi
 requested-token-type: urn:pulumi:token-type:access_token:organization
 - name: Install and inject ESC environment variables
 uses: pulumi/esc-action@v3
 with:
 environment: 'tinyco/someProject/myEnv@stable'
 - name: Verify environment variables
 run: |
 echo "FOO=$FOO"
 echo "SOME_IMPORTANT_KEY=$SOME_IMPORTANT_KEY"
 echo "TEST_ENV=$TEST_ENV"

The GitHub Action workflow above does the following:

Authenticate with Pulumi Cloud using pulumi/auth-actions. Instead of storing a permanent token, we use a short-lived token to authenticate with Pulumi Cloud.
Inject the environment variables from tinyco/someProject/myEnv into the current GitHub Actions environment, making them accessible to subsequent steps in the workflow.

What’s Next?

Explore More ESC Features: Dive deeper into ESC’s environment management, secret rotation, and dynamic credentials.
Contribute to the GitHub Action: The Pulumi ESC GitHub Action is open source. Feel free to open issues, submit PRs, or suggest enhancements.
Join the Pulumi Community: Connect with other Pulumi users and share best practices in our Community Slack.

We’re thrilled to bring you the Pulumi ESC GitHub Action, allowing you to automate your infrastructure and streamline secrets management from within GitHub. This integration makes it simpler than ever to adopt best-in-class secrets management and environment configuration practices, all while speeding up your CI/CD pipelines.

Have questions or feedback? Feel free to reach out to us in our Community Slack or submit an issue on GitHub. We can’t wait to see what you build!

The Pulumi 'Push to start' GitOps Experience

Kat Morgan — Tue, 23 Jan 2024 19:41:13 +0000

As a skeptic of “quick starts” myself, I approach most marketing promises with a measure of cautious excitement. If the great and powerful algorithm, friends, or a peer brought your attention here, then I invite you to take this one seriously.

Pulumi, with its full support of many general-purpose programming languages, can appear like a chore to get started with. The feeling can haunt seasoned developers as much as practitioners new to infrastructure code.

However, I’ll show you that finding the proverbial easy street is easier than you might believe. The pulumi new developer story just gets sweeter when combined with a few other nice-to-have conveniences.

This is a “bring your own GitHub login” event, so refresh that login. You will want to grab a Pulumi Cloud login as well. With credentials in hand, you are set with everything that you need to start. Let’s go!

Hint: all of the steps in this demo are ready to run yourself from the Pulumi Devcontainer using just your web browser and a Github account. Even better, they are real activities that Pulumians use every day to orchestrate the cloud too!

New technology learning advice

When it comes to ice cream, sometimes just selecting your flavor can be too much. Similarly, when choosing where to start with cloud tooling the onset of “choice paralysis” can be chronic. Pulumi’s powerful toolset is both a worthy endeavor to learn and a rewarding challenge to master. There is no better time to learn than the present!

“If you are short on time and have a bike, do you stop to hop on the bike, or choose to walk because there is no time to stop and get on?”

Freezing, while great for ice cream, is not quite as satisfying when building the next cloud solution and putting it off for next week or next year can be about as useful as walking fast and pushing a perfectly good bike because there is no time to stop and get on.

How can we overcome the learning curve? Picking up and mastering a new tool can be easy if we choose to start small.

Tech learning hacks:

🏆 Do: Start
🔨 Use: Helpful Tools
🥅 Make: Achievable Goals
🎉 Win: Celebrate Successes

Recognizing achievable goals can be tricky when you are learning something new. Always over-estimate the complexity of each step. If you overestimate how complicated a task might be, the biggest risk is accomplishing the goal faster than you think! Either way, the learning journey should be enjoyable and rewarding by following this rule.

Helpful tools are more abundant than ever. I recommend Pulumi as one of the most helpful tools but that is my bias showing through. Consider how intuitive, durable, and well-integrated any tool selection you make will be. Today we’re using multiple features and services of both the Pulumi and Pulumi Cloud offering, as well as demonstrating the benefits of a tightly integrated Git, Github, Devcontainers, and VSCode workflow to make light work of a traditionally considerably complicated achievement.

The resulting developer stack left me feeling like I would be successful automating anything. Be sure to share your experience with us in the Pulumi Community Slack or find us on Twitter/X @PulumiCorp too!

A DevOps journey

The best solutions are often buried in the details. The only way to reach them is to dig.

In true DevOps fashion, development for this blog post featured a roller coaster of chronic scope creep, pushing deadlines, discovering details that scrapped the original plan, and during the final finish line approach too! The worry and doubt was palpable.

Have we taken on too much? Did project ambitions exceed the achievable? The solution emerged while hacking on a side quest with a friend. A single vision came into focus, tying everything together into a simple “Push to Start” Pulumi developer workflow.

In this story, it took two tries to achieve the goal. Often it can take several tries. With that backstory let’s begin this next learning adventure together.

DevOps day 0

Let’s track our DevOps journey in typical “DevOps Day” style. Because we’re counting like computers, of course, we start with zero. In other words, we need a goal, and we will be starting from scratch.

Day zero is going to be your design, planning, and prep phase. Sometimes this includes various stages of research and other times it is a matter of just getting started with hands-on investigation.

Every journey includes a beginning and an end. To demonstrate the Pulumi Push to Start developer workflow, we need a project to develop! For this adventure let’s use the Pulumi Kubernetes Provider from the Pulumi Providers registry to deploy an app on Kubernetes. This provider is one of our most popular providers, as you might guess.

In this case, we can practice with a new project from the Community Engineering team. The GitHub pulumi/devcontainer repository is a GitHub template repository containing all the fundamentals of a new Pulumi infrastructure as code GitOps project. Let’s begin with a quick tour of the GitOps repo boilerplate.

“What belongs in a GitOps git repository?”

I have seen this question raised several times during the past year. Most recently it was brought up during a great customer conversation at AWS re:Invent.

Filtering through the conflicting opinions and advice about proper GitOps repo hygiene, a few fundamentals remain consistent.

Ops Environment
- Configuration
- Dependencies
- Artifacts
Documentation
- Who it is for
- What it does
- How to Operate
- Why it is important
Actions
- Testing: Continuous Integration
- Deployment: Continuous Deployment
- Safety: Version Controls

You may have assembled projects piece by piece before. In the pulumi/devcontainer repository, we provide a turnkey “known good” starter kit which will even improve over time. As a community resource, we welcome bugs and contributions!

And now for the feature presentation, let’s get to dev’ing!

What’s in a DevOps day 0?

Day 0 usually includes aspects of planning and preparing. For this exercise, we have just a few requirements to get started. Let’s knock these out and get to the fun stuff!

Prerequisites

Now, even the fanciest push-to-start car still requires a fob, so let’s cover those bases super quick.

Web browser
GitHub account
10 minutes (approx.)

Next, let’s review the tools we will depend on for developing and deploying our project.

Commandline Utilities:

*Dependencies supplied in the Pulumi devcontainer ghcr.io/pulumi/devcontainer

Tool	Description
A Programming Language	The code part of IaC
Pulumi CLI	Infrastructure as Code Tools
Pulumi ESC	Secrets and Configuration Store
Kubectl	Kubernetes CLI
Direnv	Automated Environment Variables
KinD	Local Kubernetes Infrastructure
Github CLI	Github & Git Repository Binaries
curl, jq, act, awk, etc	Other useful tools

Cloud-native … cloud developer environment?

Besides our command line tools and the programming language of your choice, we need an editor to write our code with. Many practitioners are fluent with Vim or Emacs; others turn to editors like VSCode, IntelliJ, and others. We are going to choose the GitHub Codespaces VSCode IDE for its deep integration with GitHub, and its convenience of requiring nothing from the user other than a web browser to use.

If you are already familiar with GitHub Codespaces, then you might know what to expect next. If not, you are in for a treat, and yes that is right, we are developing for the cloud, in the cloud, with the cloud.

Alright, let’s jump in and get started by launching a new GitHub Codespaces.

👇 Click this button to begin.👇

If you prefer a more informed approach you can launch the same codespaces by navigating to the Pulumi Devcontainer Repository or you can learn more about Codespaces at this Github Codespaces Link.

Clicking the ‘Open in GitHub Codespaces’ button should take you directly into a new Codespaces session.

Note: This demonstration is deeply integrated into the Web based Codespaces environment offered by Github. Alternatively, VSCode also supports running Devcontainers with Docker locally allowing you the benefit of Pulumi Devcontainer on your own hardware.

With a new codespaces session launched you should see the VSCode IDE in your browser. If you are not already familiar with VSCode, you can learn more about it in the VSCode Docs.

Congratulations! Your developer environment is ready for some Pulumi action.

For our final Day 0 step, we still need to decide what app to deploy with our Infrastructure-as-Code (IaC) program.

Focusing on something fun, let’s deploy Minecraft as the app for our Kubernetes exercise. It’s a fun way to get started with Pulumi, a great way to show off your cloud skills to friends, and maybe even share with your favorite young future technologists! After all, Open Source DevOps is for everyone!

DevOps day 1

We have an environment! We have a plan! Now we need to code … or do we?

Don’t just take my word for it. The big brains of Pulumi’s AI Engineering team have delivered a brilliant Pulumi AI. Your Pulumi IaC developer sidekick makes learning and developing Pulumi just that much easier.

Q: What can Pulumi AI do? A: Today’s AI is all experimental. As an industry, we are only just beginning to discover the possibilities, but you are invited to try it yourself with the Pulumi Neo WebUI or with the pulumi new cli as demonstrated below.

`~$ pulumi new --ai`

Pulumi Neo is available via WebUI much like the ChatGPT console. Besides the familiar browser-based interface, Pulumi AI is also built into the pulumi new CLI too! Now you can prompt Pulumi AI interactively from the convenience of the command line tools you will already be using, conveniently configured in your new Codespaces IDE.

Let’s try it! (Reminder: Perform all these steps from within the Codespaces IDE.)

# First Login to Pulumi Cloud for a few features we will use in upcoming GitOps content.
pulumi login

Step 2. Create a new program

# Let's create a new pulumi with the Pulumi CLI
pulumi new

Step 3. YOLO for fun and profit!

Let’s go all in on this one and try Pulumi AI. We can always enhance the prompt with additional dialog back and forth during the pulumi new chat if it doesn’t meet our expectations right away.

It is good to keep in mind that even as advanced as AI is today, the practice is still very much in its infancy. We will save more deep dives into the differences Traditional Ops (Imperative Ops), GitOps (Automated Declarative Ops), and AI Ops (Probability Augmented GitOps) in another post, for now let’s see how close the Pulumi AI can take us.

Woohoo! We selected the AI option on CLI, let’s follow the user prompts and feed the AI a simple prompt of what we want our Pulumi project to create.

Let’s try something like this:

Write a pulumi infrastructure as code program which creates the namespce 'minecraft' and using the pulumi_kubernetes.helm.v3.Release to deploy the https://itzg.github.io/minecraft-server-charts 'minecraft-server' helm chart, sets EULA to true, deploys onto Kubernetes in the minecraft namespace."

How did we do? Let’s see what the AI came up with.

It’s looking pretty good for a simple deployment. In DevOps, nothing works till the CI testing turns green. Since fortune favors the bold let’s proceed. For my developer workflow I ran kind create cluster in the Codespaces terminal. You can do the same or add your own kubeconfig to deploy on your choice of cluster.

Of course, proceed through the remaining prompts from the pulumi new interactive CLI.

I chose Python for this project, but you can choose any of the supported languages. We can see that pulumi new creates a new Python virtual environment for us programmatically. In addition to the boilerplate supporting our new project, Pulumi AI wrote the first iteration of our Python Minecraft Deployment IaC! The next steps should be printed on the screen. Assuming your Kubernetes cluster kubeconfig is ready to roll we just have one step left.

Hint, this is where we get the name for our annual “Pulumi Up conference.

Follow the link for the full playlist of 2023 Pulumi Up sessions

pulumi up -y

DevOps day 2

Finally, it is day two. This is the beginning of a choose-your-own-adventure wonderland and the opportunities go almost as far as your imagination. This is where we commit our code and plan our next change requests and enhancements.

We can commit this code to git. Build a Github Actions Workflow to automate the CI testing of our IaC with every git pr or commit. Once you have a production-ready IaC, you can add another to deploy to production clusters. If you run make act you will see how comfortable testing new GitHub actions workflows can be in the convenience of your new enhanced IDE workflow. You can see the dev container workflow running yourself right away by executing make act as a starting point.

To conclude, Pulumi is not a single tool. Pulumi is a complete cloud developer ecosystem. Tools, SDKs, and cloud services come together forming the most accelerated way to create your own cloud success legends.

Conclusion

Did it deploy?

Let’s check on the status of our deployment.

# Check for your new Minecraft pod
kubectl get po

With any luck, our Pulumi Neo gave you a working program right out of the gate. For references on Kubernetes Pulumi package resources to expand your infrastructure from here head to the Pulumi Registry Kubernetes Provider page for more.

Sharing Pulumi with people is a genuine passion found among Pulumians all around the globe but we all know evidence reigns king among engineers who, like any good practitioner, expect proof. Evolving your developer workflow with Pulumi gets even better than just easy, it’s remarkably powerful and ready for you to adopt in your daily workflows right away too.

Follow PulumiTV for upcoming examples built on this solution and to see how Pulumi ESC brings safety to your secrets and more.

What’s in the recipe?

Because no hero’s journey is complete without a gruesome account of what came before, I have included many of the manual steps required to reach this state one command at a time for the ~~morbidly~~ curious as well.

click ‘details’ to expand

Git Code Repository

Now create a git code repository to version control this project infrastructure code in Github.

# Authenticate with Github before proceeding.
gh auth login --web --git-protocol https --scopes "repo,gist,read:packages,admin:org,delete_repo,codespace"

# Create new Git Repository
gh repo create workshop --public \
 --gitignore Python --license apache-2.0 \
 --description "pulumi iac developer workflow workshop"

# Initialize the git repository
git clone --recurse-submodules https://github.com/${GITHUB_USER}/workshop .

# Configure `git` cli to use the `gh` cli for authentication with Github
gh auth setup-git

# Configure the username and email associated with your Github account.
git config --global user.email ${GIT_COMMITTER_EMAIL}
git config --global user.name ${GITHUB_USER}

# List files being tracked by git and their current status.
git status

Direnv: Automatic Environment Variables

Various non-secret environment variables may be worth maintaining in code locally as well. There are many ways to do this, but here we are going to use Direnv to automatically load environment variables from a .envrc file.

# Add environment variables useful during development to .envrc
cat <<EOF >> .envrc
export NO_COLOR=true
export PULUMI_HOME=\${HOME}/.pulumi
export PULUMI_SKIP_UPDATE_CHECK=true
export PULUMI_SKIP_CONFIRMATIONS=true
export KUBECONFIG=\${PWD}/.kube/config
source venv/bin/activate 2>/dev/null || true
EOF

Now you can enable direnv on this directory.

# Enable direnv in this directory.
direnv allow

Note:

Env variables shown are for educational purposes only and should be used with care. Read more about available Pulumi Environment variables here: https://www.pulumi.com/docs/cli/environment-variables

Pulumi Devcontainer

Now, to make sure we have all of our CLI dependencies, let’s grab the Pulumi Devcontainer.

# Add the Pulumi Devcontainer git submodule
git submodule add https://github.com/pulumi/devcontainer .devcontainer
git submodule update --init --recursive .devcontainer

# Copy the devcontainer config of your choice into the top level of your project
# so it will auto select by default when you open Codespaces in the future.
cp -f .devcontainer/devcontainer.json .devcontainer.json

# Rebuild the codespaces using the newly added Pulumi Devcontainer
gh codespace rebuild --codespace ${CODESPACE_NAME}

Pulumi Cloud

Let’s login to Pulumi Cloud and initialize a new ESC environment to store our environment variables, secrets, and configuration which should be preserved and distributed securely from your Pulumi Cloud account.

# Login to Pulumi Cloud
pulumi login

# Create a new ESC Environment (optionally use another already existing environment)
pulumi env init workshop

KinD: Kubernetes-in-docker

We are going to use Kubernetes to demonstrate our Pulumi IaC. Let’s go ahead and create a new cluster now.

# Create the directory for your new kind kubeconfig
mkdir .kube

# Create a Kind Cluster
kind create cluster

# Let's make sure our kubeconfig works and pods are starting up
kubectl get pods -A

# Load Kubeconfig into Pulumi ESC as an encrypted secret for safe keeping.
# This is an example use case for storing kubeconfig which would not usually be practical
# for disposable kind kubernetes clusters like this however the approach is practical in many
# other scenarios which you may come across.
pulumi env set workshop secrets.kubeconfig.kind --secret "$(jq . -R -s < $KUBECONFIG)"
pulumi env set workshop files.KUBECONFIG --plaintext \${secrets.kubeconfig.kind}

In our environment, we maintain our secrets including API personal access tokens, kubeconfigs, and such all with Pulumi’s Environments, Secrets, and Configuration cloud service.

# Load the newly updated Pulumi ESC Environment in the local shell
eval $(pulumi env open workshop --format shell)

Now let’s create the deployment code!

If fortune favors the bold, let’s be bold on this next step and let AI write our sample code on the fly!

As this is a “hello world” style demonstration intended to showcase the developer workflow more than any specific cloud technologies, let’s deploy something fun like a Minecraft server! Be sure to watch the video demonstration of this exercise on Pulumi’s PulumiTV YouTube channel for an extra easter egg bonus step!

# Write a new Pulumi Python IaC program to deploy Minecraft on Kubernetes
pulumi new \
 --ai "Write a program using pulumi kubernetes helm v3 Release to deploy the itzg/minecraft-server helm chart on Kubernetes, and include a pulumi output to show the helm release status." \
 --description "A pulumi infrastructure as code (iac) program for deploying and serving minecraft on kubernetes" \
 --name "minecraft-on-kubernetes" \
 --language python \
 --stack "workshop" \
 --force \
 --dir .

Now let’s see if our new Infrastructure as Code Pulumi Python Codebase will deploy!

# Deploy your new Pulumi IaC on Kubernetes
pulumi stack select workshop
pulumi up

Cleanup

pulumi destroy -y --skip-preview
pulumi stack rm workshop
kind delete cluster --name kind
gh repo delete ${GITHUB_USER}/workshop

GitHub & Pulumi Join Forces To Ensure Pulumi Tokens Are Safe

Praneet Loke — Mon, 19 Aug 2019 00:00:00 +0000

We are very excited to announce that we have partnered with GitHub to offer our users better protection for their Pulumi Access Tokens.

By default, Pulumi users manage the state of their cloud infrastructure deployments using https://app.pulumi.com. This service provides state storage, concurrency control, audit history and access controls for both individuals and teams working with Pulumi. Each user and service account can generate one or more Pulumi Access Tokens to be used to authenticate with this service. These access tokens can be used on both local development machines, as well as in CI/CD systems for automated infrastructure deployments. These access tokens are sensitive secrets which should never be shared publicly, and in particular should never be committed to source control.

How GitHub Token Scanning Works for Pulumi Users

With the recent changes to integrate Pulumi with GitHub Token Scanning, GitHub now scans each commit you push to a public repo on github.com for any files containing these tokens. When GitHub’s Token Scanning Service finds a potential match for our tokens, the service will call ours to have us verify the token. Only Pulumi Access Tokens generated after June 28, 2019 that contain the prefix pul- will be matched by the scanning service.

During the processing of the requests from the Token Scanning Service, if we find that the token belongs to a Pulumi account, we will use the primary email on the account to notify the user. We will not, however, revoke the token automatically. So ensure that your account’s email address is up-to-date by going to your profile in the Pulumi Service.

The whole process, from the check-in to the email notification, typically takes just a few seconds, minimizing the window of an accidental breach. Of course, best practice is to avoid checking in the token at all!

Avoid Checking-In The Token In The First Place

Pulumi Access Tokens grant access to your stacks. They allow you to login into the CLI non-interactively and thus allow the CLI to access a stack’s configuration, which may in-turn contain other secrets. The token is typically used inside CI/CD environments. So you should never have to check-in a token to your source tree (public or otherwise.)

Most CI/CD services have a way for you to specify secret environment variables that allow you to safely inject them into your build environment. We have written-up quite a few guides on how to integrate Pulumi into your CI/CD. If you don’t see a guide for a CI/CD system that you are using, feel free to create an issue here, or even better, raise a PR for it. We love community contributions! :)

Pulumi Blog: Github

Expanded Version Control Support in Pulumi Cloud

Connect multiple providers and accounts

What your team can do

Deploy on every push

Preview changes on pull requests

Neo explains your changes

Let Neo open pull requests for you

Detect and fix drift

Secure authentication

Set up new projects from your VCS

Getting started

How We Load Data into Snowflake in Seconds with Pulumi

What we’ll cover

Architecture overview

Project setup

Managing credentials with Pulumi ESC

Shared infrastructure

Building the direct ingestion ComponentResource

The Lambda handler

Why direct Firehose to Snowflake?

The DirectSnowflakeIngestion component

Wiring it together

Testing the pipeline

Other loading patterns

Publishing as reusable components

Git-based sharing

Pulumi Cloud Private Registry

Conclusion

Automatic Diagram Generation for Always-Accurate Diagrams

Native Pulumi diagramming with GitHub Actions

How the workflow works

Key advantages and limitations

Accessing your diagrams

AI-powered diagramming with Claude and Mermaid

The automated workflow

How the AI approach works

Example generated diagram

Key advantages and limitations

Pulumi Neo: Enterprise-scale intelligent diagramming

How the workflow works

Key advantages and limitations

Conclusion

Announcing the Pulumi ESC GitHub Action

Why we built it

Key benefits

How it works

Example workflow

What’s Next?

The Pulumi 'Push to start' GitOps Experience

Index

New technology learning advice

Tech learning hacks:

A DevOps journey

DevOps day 0

What’s in a DevOps day 0?

Prerequisites

Commandline Utilities:

Cloud-native … cloud developer environment?

DevOps day 1

~$ pulumi new --ai

Step 1. Login to Pulumi Cloud

Step 2. Create a new program

Step 3. YOLO for fun and profit!

DevOps day 2

Conclusion

Did it deploy?

What’s in the recipe?

Git Code Repository

Direnv: Automatic Environment Variables

Pulumi Devcontainer

Pulumi Cloud

KinD: Kubernetes-in-docker

Now let’s create the deployment code!

Cleanup

GitHub & Pulumi Join Forces To Ensure Pulumi Tokens Are Safe

How GitHub Token Scanning Works for Pulumi Users

Avoid Checking-In The Token In The First Place

`~$ pulumi new --ai`