Pulumi Blog: Data and analytics

How We Load Data into Snowflake in Seconds with Pulumi

Pablo Seibelt — Mon, 23 Feb 2026 00:00:00 +0000

When you manage dozens of data-loading pipelines, copying and pasting IaC configurations between them is a recipe for mishap. IAM policies can drift, naming conventions diverge, and every new source is a new opportunity to make a mistake — not to mention compound the problem of duplication. In this post, we’ll show you how you can identify and encapsulate common patterns into composable components and walk through the production lessons we’ve learned running 25+ pipelines for over three years.

What we’ll cover

If you’re loading data into Snowflake and want reusable, composable infrastructure, this post is for you. Here’s what we’ll cover:

Handling and validating GitHub webhooks with AWS Lambda
Streaming webhook payloads directly into Snowflake with Amazon Data Firehose
Wiring it all up with a reusable Pulumi ComponentResource

The companion template also includes S3 auto-ingest and batch loading patterns, which we’ll cover in upcoming posts. We also use Pulumi ESC to handle authentication to both AWS and Snowflake using OpenID Connect.

Our own Josh Kodroff wrote an excellent introduction to Snowpipe with Pulumi. This post builds on his work using the newest Snowflake and AWS provider APIs and the direct Firehose-to-Snowflake destination, which wasn’t available when Josh wrote his post. Some resource names and grant patterns will also differ if you’re comparing the two.

Architecture overview

The following diagram shows the architecture in more detail:

GitHub sends webhook events to a Lambda Function URL.
Lambda validates the HMAC (Hash-based Message Authentication Code) signature and forwards the payload to Amazon Data Firehose.
Firehose streams records directly into Snowflake via the Snowpipe Streaming API. Data appears in Snowflake within seconds.
S3 is used only as a backup destination for failed records.

The direct Firehose-to-Snowflake destination is an AWS-native feature that works with any Snowflake account.

Project setup

Start a new Pulumi Python project and choose uv for dependency management when prompted:

mkdir snowpipe-data-loading && cd snowpipe-data-loading
pulumi new python

Notice Pulumi.yaml shows uv as your selected toolchain:

name: snowpipe-data-loading
runtime:
 name: python
 options:
 toolchain: uv

Add the provider dependencies for AWS, Snowflake, GitHub, and the Random and TLS providers:

uv add pulumi-aws pulumi-snowflake pulumi-github pulumi-random pulumi-tls

That’s it. uv creates the virtual environment and lockfile automatically, and Pulumi uses uv run under the hood to execute your program.

All examples in this post are in Python, but Pulumi supports multiple languages. You can implement the same components in TypeScript, Go, .NET, Java, or YAML.

Managing credentials with Pulumi ESC

This project needs credentials for AWS, Snowflake, and GitHub. Rather than managing long-lived secrets locally, we can use Pulumi ESC to obtain dynamic, short-lived OIDC credentials for AWS and Snowflake at runtime. When you run pulumi up, ESC exchanges a Pulumi-issued OIDC token for temporary credentials from each provider and injects them into your stack config automatically. If you prefer not to use ESC, you can set credentials directly with pulumi config set --secret.

Here is a single ESC environment that handles all three providers:

values:
 aws:
 login:
 fn::open::aws-login:
 oidc:
 duration: 1h
 roleArn: arn:aws:iam::123456789012:role/pulumi-esc-oidc
 sessionName: pulumi-snowpipe-demo
 snowflake:
 login:
 fn::open::snowflake-login:
 oidc:
 account: <your-snowflake-account>
 user: ESC_SERVICE_USER
 organizationName: <your-org-name>
 accountName: <your-account-name>
 environmentVariables:
 AWS_ACCESS_KEY_ID: ${aws.login.accessKeyId}
 AWS_SECRET_ACCESS_KEY: ${aws.login.secretAccessKey}
 AWS_SESSION_TOKEN: ${aws.login.sessionToken}
 SNOWFLAKE_USER: ${snowflake.login.user}
 SNOWFLAKE_TOKEN: ${snowflake.login.token}
 pulumiConfig:
 aws:region: us-west-2
 snowflake:organizationName: ${snowflake.organizationName}
 snowflake:accountName: ${snowflake.accountName}
 snowflake:authenticator: OAUTH
 snowflake:role: PULUMI_DEPLOYER
 github:token:
 fn::secret: <your-github-pat>
 github:owner: <your-github-org-or-user>

Then reference the environment from your stack config:

# Pulumi.<stack>.yaml
environment:
 - <project>/my-snowpipe-env
config:
 snowpipe-data-loading:database: LANDING_ZONE_WEBHOOKS
 snowpipe-data-loading:environment: dev
 snowpipe-data-loading:webhook-repo: <your-test-repo-name>
 snowflake:previewFeaturesEnabled:
 - snowflake_table_resource

Depending on your preferences, you can split credentials into separate per-provider environments and compose them with imports and reuse across stacks.

To set up OIDC trust for each provider, see the AWS OIDC guide and the Snowflake OIDC login guide. For GitHub authentication options (fine-grained PATs, classic PATs, or GitHub Apps), see the pulumi-github provider docs.

Shared infrastructure

The direct streaming pipeline needs an S3 bucket for backup/errors, a Snowflake database, and a schema.

Building the direct ingestion ComponentResource

Amazon Data Firehose supports Snowflake as a native destination via the Snowpipe Streaming API. Firehose streams records directly into Snowflake.

The Lambda handler

The Lambda function is the entry point for GitHub webhooks. It validates the HMAC-SHA256 signature, wraps the payload in an envelope with the event type, and forwards it to Firehose. Create this as lambda/webhook_handler.py:

import hashlib
import hmac
import json
import os

import boto3

firehose = boto3.client("firehose")

STREAM_NAME = os.environ["FIREHOSE_STREAM_NAME"]
WEBHOOK_SECRET = os.environ["WEBHOOK_SECRET"]


def handler(event, context):
 body = event.get("body", "")
 signature = (event.get("headers") or {}).get("x-hub-signature-256", "")

 # Validate HMAC-SHA256 signature
 expected = "sha256=" + hmac.new(
 WEBHOOK_SECRET.encode(), body.encode(), hashlib.sha256
 ).hexdigest()

 if not hmac.compare_digest(expected, signature):
 return {"statusCode": 401, "body": "Invalid signature"}

 github_event = (event.get("headers") or {}).get("x-github-event", "unknown")

 # Wrap in envelope - newline-delimited for S3 backup where Firehose concatenates records
 record = json.dumps({
 "github_event": github_event,
 "payload": json.loads(body),
 }) + "\n"

 firehose.put_record(
 DeliveryStreamName=STREAM_NAME,
 Record={"Data": record.encode()},
 )

 return {"statusCode": 200, "body": "OK"}

The envelope format {"github_event": "<type>", "payload": {...}}\n is important. The github_event field (e.g., push, pull_request, star) comes from the x-github-event header and lets downstream queries filter by event type. The trailing newline delimits records in the S3 backup destination, where Firehose concatenates them into files.

Another interesting part of this snippet is that instead of manually creating a secret which would have to be copy-pasted into both your webhook configuration and your Lambda environment, we are using random.RandomPassword to generate and store it securely it in Pulumi state.

The secret is automatically wired to both the Lambda env var and the GitHub webhook config, and it rotates cleanly if you ever need to replace it.

Why direct Firehose to Snowflake?

Data like this is normally written to and loaded from Amazon S3. But with an S3 intermediate path, you must wait for Firehose to buffer records (60 seconds), then for Snowpipe to detect the new file and load it. Total latency: about two minutes. With the direct Snowflake destination, Firehose uses the Snowpipe Streaming API to insert records as soon as they arrive, in seconds.

The direct path also removes the need for S3 event notifications, SQS queues, external stages, and pipe resources. S3 is still used, but only as a backup destination for failed records.

The DirectSnowflakeIngestion component

The component creates everything needed for the direct path: a TLS key pair for Snowflake authentication, a Snowflake service user with least-privilege grants, the landing table, a Firehose delivery stream with destination="snowflake", and a Lambda function with a public URL. Create an empty components/__init__.py so that Python treats the directory as a package.

The full component lives in components/direct_snowflake_ingestion.py:

import json
from dataclasses import dataclass

import pulumi
import pulumi_aws as aws
import pulumi_snowflake as snowflake
import pulumi_tls as tls

# In the example repository, you will find this class imported from a common library file instead
@dataclass
class ColumnDef:
 """Column definition for a Snowflake table."""

 name: str
 type: str
 nullable: bool = True

def strip_pem_headers(pem: str) -> str:
 """Remove PEM header/footer lines, returning only the base64 content."""
 lines = pem.strip().split("\n")
 return "".join(lines[1:-1])


@dataclass
class DirectSnowflakeIngestionArgs:
 bucket_arn: pulumi.Input[str]
 bucket_name: pulumi.Input[str]
 database: pulumi.Input[str]
 schema_name: pulumi.Input[str]
 table_name: str
 table_columns: list[ColumnDef]
 snowflake_account_url: pulumi.Input[str]
 snowflake_role_name: str
 lambda_code: pulumi.Archive
 lambda_handler: str
 lambda_environment: dict[str, pulumi.Input[str]]
 table_comment: str = ""
 s3_prefix: str = "direct-webhooks"
 s3_backup_mode: str = "FailedDataOnly"
 buffering_interval: int = 0
 buffering_size: int = 1
 retry_duration: int = 60
 data_loading_option: str = "VARIANT_CONTENT_AND_METADATA_MAPPING"
 content_column_name: str = "CONTENT"
 metadata_column_name: str = "METADATA"

class DirectSnowflakeIngestion(pulumi.ComponentResource):
 function_url: pulumi.Output[str]
 firehose_stream_name: pulumi.Output[str]
 snowflake_user_name: pulumi.Output[str]

 def __init__(
 self,
 name: str,
 args: DirectSnowflakeIngestionArgs,
 opts: pulumi.ResourceOptions | None = None,
 ):
 super().__init__(
 "snowpipe:direct:DirectSnowflakeIngestion", name, {}, opts
 )

 # --- TLS key pair for Snowflake auth ---
 key_pair = tls.PrivateKey(
 f"{name}-keypair",
 algorithm="RSA",
 rsa_bits=2048,
 opts=pulumi.ResourceOptions(parent=self),
 )

 # --- Snowflake role, user, and grants ---
 sf_role = snowflake.AccountRole(
 f"{name}-sf-role",
 name=args.snowflake_role_name,
 opts=pulumi.ResourceOptions(parent=self),
 )

 user_name = f"FIREHOSE_{name.upper().replace('-', '_')}_USER"
 sf_user = snowflake.ServiceUser(
 f"{name}-sf-user",
 name=user_name,
 login_name=user_name,
 default_role=sf_role.name,
 rsa_public_key=key_pair.public_key_pem.apply(strip_pem_headers),
 opts=pulumi.ResourceOptions(parent=self),
 )

 # Landing table
 table = snowflake.Table(
 f"{name}-table",
 name=args.table_name,
 database=args.database,
 schema=args.schema_name,
 comment=args.table_comment,
 columns=[
 snowflake.TableColumnArgs(
 name=col.name, type=col.type, nullable=col.nullable,
 )
 for col in args.table_columns
 ],
 opts=pulumi.ResourceOptions(parent=self),
 )

 # Grants: DB USAGE, schema USAGE, table INSERT+SELECT
 snowflake.GrantPrivilegesToAccountRole(
 f"{name}-grant-db-usage",
 account_role_name=sf_role.name,
 privileges=["USAGE"],
 on_account_object=snowflake.GrantPrivilegesToAccountRoleOnAccountObjectArgs(
 object_type="DATABASE",
 object_name=args.database,
 ),
 opts=pulumi.ResourceOptions(parent=self),
 )

 snowflake.GrantPrivilegesToAccountRole(
 f"{name}-grant-schema-usage",
 account_role_name=sf_role.name,
 privileges=["USAGE"],
 on_schema=snowflake.GrantPrivilegesToAccountRoleOnSchemaArgs(
 schema_name=pulumi.Output.all(
 args.database, args.schema_name
 ).apply(lambda parts: f'"{parts[0]}"."{parts[1]}"'),
 ),
 opts=pulumi.ResourceOptions(parent=self),
 )

 table_name = args.table_name
 snowflake.GrantPrivilegesToAccountRole(
 f"{name}-grant-table",
 account_role_name=sf_role.name,
 privileges=["INSERT", "SELECT"],
 on_schema_object=snowflake.GrantPrivilegesToAccountRoleOnSchemaObjectArgs(
 object_type="TABLE",
 object_name=pulumi.Output.all(
 args.database, args.schema_name
 ).apply(
 lambda parts: f'"{parts[0]}"."{parts[1]}"."{table_name}"'
 ),
 ),
 opts=pulumi.ResourceOptions(parent=self, depends_on=[table]),
 )

 # --- Firehose IAM role (S3 backup write) ---
 firehose_role = aws.iam.Role(
 f"{name}-firehose-role",
 assume_role_policy=json.dumps({
 "Version": "2012-10-17",
 "Statement": [{
 "Effect": "Allow",
 "Action": "sts:AssumeRole",
 "Principal": {"Service": "firehose.amazonaws.com"},
 }],
 }),
 opts=pulumi.ResourceOptions(parent=self),
 )

 aws.iam.RolePolicy(
 f"{name}-firehose-s3-policy",
 role=firehose_role.id,
 policy=args.bucket_arn.apply(
 lambda arn: json.dumps({
 "Version": "2012-10-17",
 "Statement": [{
 "Effect": "Allow",
 "Action": [
 "s3:AbortMultipartUpload",
 "s3:GetBucketLocation",
 "s3:GetObject",
 "s3:ListBucket",
 "s3:ListBucketMultipartUploads",
 "s3:PutObject",
 ],
 "Resource": [arn, f"{arn}/*"],
 }],
 })
 ),
 opts=pulumi.ResourceOptions(parent=self),
 )

 # --- Firehose delivery stream (Snowflake destination) ---
 stream = aws.kinesis.FirehoseDeliveryStream(
 f"{name}-firehose",
 destination="snowflake",
 snowflake_configuration=aws.kinesis.FirehoseDeliveryStreamSnowflakeConfigurationArgs(
 account_url=args.snowflake_account_url,
 database=args.database,
 schema=args.schema_name,
 table=args.table_name,
 role_arn=firehose_role.arn,
 user=sf_user.name,
 private_key=key_pair.private_key_pem_pkcs8.apply(
 strip_pem_headers
 ),
 data_loading_option=args.data_loading_option,
 content_column_name=args.content_column_name,
 metadata_column_name=args.metadata_column_name,
 s3_backup_mode=args.s3_backup_mode,
 buffering_size=args.buffering_size,
 buffering_interval=args.buffering_interval,
 retry_duration=args.retry_duration,
 snowflake_role_configuration=aws.kinesis.FirehoseDeliveryStreamSnowflakeConfigurationSnowflakeRoleConfigurationArgs(
 enabled=True,
 snowflake_role=args.snowflake_role_name,
 ),
 s3_configuration=aws.kinesis.FirehoseDeliveryStreamSnowflakeConfigurationS3ConfigurationArgs(
 bucket_arn=args.bucket_arn,
 role_arn=firehose_role.arn,
 prefix=f"{args.s3_prefix}/backup/",
 error_output_prefix=f"{args.s3_prefix}/errors/",
 ),
 ),
 opts=pulumi.ResourceOptions(parent=self, depends_on=[table]),
 )

 # --- Lambda function + Function URL ---
 lambda_role = aws.iam.Role(
 f"{name}-lambda-role",
 assume_role_policy=json.dumps({
 "Version": "2012-10-17",
 "Statement": [{
 "Effect": "Allow",
 "Action": "sts:AssumeRole",
 "Principal": {"Service": "lambda.amazonaws.com"},
 }],
 }),
 opts=pulumi.ResourceOptions(parent=self),
 )

 aws.iam.RolePolicyAttachment(
 f"{name}-lambda-basic-execution",
 role=lambda_role.name,
 policy_arn="arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole",
 opts=pulumi.ResourceOptions(parent=self),
 )

 aws.iam.RolePolicy(
 f"{name}-lambda-firehose-policy",
 role=lambda_role.id,
 policy=stream.arn.apply(
 lambda arn: json.dumps({
 "Version": "2012-10-17",
 "Statement": [{
 "Effect": "Allow",
 "Action": ["firehose:PutRecord"],
 "Resource": [arn],
 }],
 })
 ),
 opts=pulumi.ResourceOptions(parent=self),
 )

 env_vars = {
 **args.lambda_environment,
 "FIREHOSE_STREAM_NAME": stream.name,
 }

 fn = aws.lambda_.Function(
 f"{name}-handler",
 runtime="python3.11",
 handler=args.lambda_handler,
 role=lambda_role.arn,
 timeout=30,
 code=args.lambda_code,
 environment=aws.lambda_.FunctionEnvironmentArgs(
 variables=env_vars,
 ),
 opts=pulumi.ResourceOptions(parent=self),
 )

 fn_url = aws.lambda_.FunctionUrl(
 f"{name}-function-url",
 function_name=fn.name,
 authorization_type="NONE",
 opts=pulumi.ResourceOptions(parent=self),
 )

 aws.lambda_.Permission(
 f"{name}-function-url-permission",
 action="lambda:InvokeFunctionUrl",
 function=fn.name,
 principal="*",
 function_url_auth_type="NONE",
 opts=pulumi.ResourceOptions(parent=self),
 )

 # --- Outputs ---
 self.function_url = fn_url.function_url
 self.firehose_stream_name = stream.name
 self.snowflake_user_name = sf_user.name

 self.register_outputs({
 "function_url": self.function_url,
 "firehose_stream_name": self.firehose_stream_name,
 "snowflake_user_name": self.snowflake_user_name,
 })

A few things to note:

TLS key pair for authentication. The component generates an RSA key pair using the pulumi-tls provider. The public key is assigned to the Snowflake service user; the private key (PKCS#8 format, base64-encoded) is passed to Firehose. No passwords or OAuth tokens are stored.
ServiceUser instead of User. Snowflake service users can’t log in interactively. They authenticate only via key pair, which is exactly what Firehose needs.
destination="snowflake" on Firehose. This tells Firehose to use the Snowpipe Streaming API rather than writing to S3. The s3_configuration block is still required, but only for backup/error records.
Immediate flushing. buffering_interval=0 and buffering_size=1 ensure records are sent to Snowflake as soon as they arrive, minimizing latency. Tune according to your needs.

Amazon Data Firehose does not connect from fixed IP addresses, so you cannot use Snowflake network policies to restrict access by IP. If your Snowflake account uses network policies, you have three options: use AWS PrivateLink (requires Snowflake Business Critical edition), allow public internet access for the Firehose service user, or switch to S3 auto-ingest via Snowpipe which does not require direct network access to Snowflake from Firehose.

Wiring it together

With the component defined, __main__.py wires the direct ingestion pipeline:

import pulumi
import pulumi_aws as aws
import pulumi_github as github
import pulumi_random as random
import pulumi_snowflake as snowflake
from components.direct_snowflake_ingestion import (
 DirectSnowflakeIngestion, DirectSnowflakeIngestionArgs,
)
from components.snowpipe_pipeline import ColumnDef

config = pulumi.Config()
database_name = config.get("database") or "LANDING_ZONE_WEBHOOKS"
environment = config.get("environment") or "dev"

# --- Shared infrastructure ---

# S3 bucket for backup/errors
bucket = aws.s3.Bucket(
 "data-landing-bucket",
)

# Snowflake database and schema
database = snowflake.Database("demo-database", name=database_name)
schema = snowflake.Schema(
 "demo-schema",
 name="GITHUB",
 database=database.name,
)

# --- Direct ingestion pipeline ---

webhook_repo = config.require("webhook-repo")

# Snowflake account URL for Firehose configuration
snowflake_config = pulumi.Config("snowflake")
snowflake_account_url = (
 f"https://{snowflake_config.require('organizationName')}"
 f"-{snowflake_config.require('accountName')}"
 f".snowflakecomputing.com"
)

# Landing table columns: CONTENT (webhook JSON) + METADATA (Firehose metadata)
DIRECT_COLUMNS = [
 ColumnDef(name="CONTENT", type="VARIANT", nullable=True),
 ColumnDef(name="METADATA", type="VARIANT", nullable=True),
]

# Step 1: Generate webhook secret for HMAC validation
direct_webhook_secret = random.RandomPassword(
 "github-direct-webhook-secret", length=32, special=False
)

# Step 2: Direct ingestion pipeline - Lambda validates, Firehose streams to Snowflake
direct = DirectSnowflakeIngestion(
 "github-webhooks-direct",
 DirectSnowflakeIngestionArgs(
 bucket_arn=bucket.arn,
 bucket_name=bucket.bucket,
 database=database.name,
 schema_name=schema.name,
 table_name="REPOSITORY_EVENTS_DIRECT",
 table_columns=DIRECT_COLUMNS,
 table_comment="GitHub webhook events loaded via direct Firehose to Snowflake",
 snowflake_account_url=snowflake_account_url,
 snowflake_role_name="FIREHOSE_DIRECT_LOADER",
 lambda_code=pulumi.AssetArchive({
 "webhook_handler.py": pulumi.FileAsset("lambda/webhook_handler.py"),
 }),
 lambda_handler="webhook_handler.handler",
 lambda_environment={"WEBHOOK_SECRET": direct_webhook_secret.result},
 ),
)

# Step 3: GitHub webhook - sends events to the Lambda Function URL
github.RepositoryWebhook(
 "github-direct-webhook",
 repository=webhook_repo,
 configuration=github.RepositoryWebhookConfigurationArgs(
 url=direct.function_url,
 content_type="json",
 secret=direct_webhook_secret.result,
 ),
 events=["push", "pull_request", "issues", "star"],
)

# Exports
pulumi.export("webhook_url", direct.function_url)
pulumi.export("firehose_stream", direct.firehose_stream_name)

That’s the entire pipeline. One component, one GitHub webhook, one secret. The DirectSnowflakeIngestion component handles the TLS key pair, Snowflake service user, landing table, Firehose stream, and Lambda function internally, and now you can reuse this component for as many pipelines as you need.

The full code for this example is available on GitHub:

github.com/pulumi-demos/examples/tree/main/python/aws-snowflake-data-loading-real-time

Testing the pipeline

Deploy the stack:

pulumi up

The entire stack deploys in about two minutes. Immediately after deployment, you’ll start seeing GitHub events flowing into Snowflake.

Before querying, grant the DATA_READER role to your Snowflake user:

GRANT ROLE DATA_READER TO USER <your-user>;

In production, you can manage this grant through Pulumi, manually, or automatically via SCIM provisioning from your identity provider.

No need to craft test payloads. Just interact with the test repo. Star it, push a commit, or open an issue, then wait about 30 seconds and query Snowflake using the least-privilege reader role:

USE ROLE DATA_READER;

SELECT CONTENT:github_event::STRING AS event_type,
 CONTENT:payload:repository:full_name::STRING AS repo,
 METADATA:IngestionTime::TIMESTAMP AS ingested_at
FROM LANDING_ZONE_WEBHOOKS.GITHUB.REPOSITORY_EVENTS_DIRECT
ORDER BY ingested_at DESC;

You should see rows with event types like star, push, or issues, real GitHub events flowing through the entire pipeline. The METADATA column includes Firehose metadata like IngestionTime, which you can use to track end-to-end latency.

Other loading patterns

Direct streaming is the fastest path, but two other patterns are available in the companion template for different requirements:

S3 auto-ingest via Snowpipe. Firehose buffers to S3, and Snowpipe auto-ingests new files. Latency is about two minutes. Best when you need S3 as the system of record or can’t use direct Snowpipe Streaming.
Batch loading. Your orchestrator (Airflow, Prefect, cron, etc.) runs COPY INTO on a schedule. Best for full control over timing and deduplication.

We’ll walk through both patterns in detail in upcoming posts.

Publishing as reusable components

Once your components are battle-tested, you can share them across teams and projects instead of copying files around.

The most straightforward approach: push your components to a Git repository with a PulumiPlugin.yaml file at the root:

runtime: python
name: snowpipe-components
version: 1.0.0

Consumers add the package to their project with pulumi package add:

pulumi package add github.com/your-org/pulumi-snowpipe@v1.0.0

Pulumi downloads the package and generates typed SDKs automatically. The consumer’s __main__.py imports your components as if they were local, but they’re versioned and pinned.

Pulumi Cloud Private Registry

For organization-level discoverability, publish to the Pulumi Cloud Private Registry with pulumi package publish:

pulumi package publish ./schema.json

This gives you auto-generated API docs, usage tracking across teams, and cross-language SDK generation. Your Python components become usable from TypeScript, Go, and C# without any extra work. Teams browse available components in the Pulumi Cloud console, see who’s using what, and get notified when new versions are published. As an additional benefit, Neo will be able to use these components and build new pipelines in minutes from a natural language request.

Conclusion

ComponentResource is the key abstraction that makes this architecture scale. Instead of copying and pasting resources for each new data source, you instantiate a component with a handful of configuration parameters.

The DirectSnowflakeIngestion component in this post delivers data from GitHub webhooks into Snowflake in seconds: Lambda validates the HMAC signature, Firehose streams directly to Snowflake via the Snowpipe Streaming API, and the TLS key pair is managed entirely within Pulumi. No S3 intermediate, no SQS queues.

The component accepts pluggable Lambda handlers, so swapping GitHub for Stripe webhooks or any other source is just a matter of providing different lambda_code and lambda_environment arguments. We’ve been running this pattern in production for over three years across dozens of pipelines without significant changes to the infrastructure code.

You’ll find the complete example in the GitHub repository.

How We Built Platybot: An AI-Powered Analytics Assistant

Pablo Seibelt — Wed, 11 Feb 2026 00:00:00 +0000

Before Platybot, our #analytics Slack channel was a support queue. Every day, people from every team would ask questions: “Which customers use feature X?”, “What’s our ARR by plan type?”, “Do we have a report for template usage?” Our two-person data team was a bottleneck.

Our #analytics channel, before Platybot (dramatized).

We didn’t want to just throw an LLM at our Snowflake warehouse either. Without guardrails, large language models generate SQL that may work but silently gets the answer wrong. Different join logic, wrong filters, missing snapshot handling, incorrect summarization. We needed something that could answer reliably for most queries, otherwise we’d switch to fixing LLM SQL queries.

So we built Platybot (platypus + bot, named after our mascot), an AI-powered analytics assistant that any Pulumi employee can use to query our Data Warehouse in natural language. It’s available as a Web App, a Slack bot, and a Model Context Protocol (MCP) server. The infrastructure is deployed with Pulumi IaC (Infrastructure as Code). But the most important thing we learned building it is that the AI was the easy part. The semantic layer is what makes it work.

The problem with throwing AI at your Data Warehouse

The naive solution is obvious: connect an LLM to your database and let it write SQL. But this fails in practice, and the failure mode is insidious. Consider a few examples from our warehouse:

ARR is a snapshot metric. If you query ARR (Annual Recurring Revenue) without filtering by end-of-period dates (last day of month or quarter), you get duplicate rows and wildly inflated numbers. An LLM doesn’t automatically know this.
Account queries need exclusions. Most queries should exclude Pulumi’s own internal accounts and deleted ones. Without these filters, you’re counting test data alongside real customers.
User queries need employee filters. Querying active users without excluding Pulumi employees inflates adoption metrics.

The danger shows up when results feel decision-ready before anyone has validated how the numbers were derived. A confidently wrong ARR figure presented to leadership is worse than no answer at all.

Many organizations run into the same constraint once data usage spreads. Dashboards answer yesterday’s questions, not today’s. Ad-hoc LLM queries answer today’s questions, but incorrectly. The gap between “I have a question” and “I have a trustworthy answer” is where data teams can get stuck.

Why we built a semantic layer first

Before writing a single line of AI code, we built a semantic layer using Cube (open source). This was the hardest, least glamorous, and most important part of the entire project.

A semantic layer is a shared, versioned definition of what your business metrics mean.

“Monthly active users” starts with COUNT(DISTINCT user_id), but the aggregation is only the outer layer. It has to be applied to the right table (fct_pulumi_operations), with the right filters (exclude Pulumi employees, exclude deleted organizations), scoped to a calendar month, and counting only users who performed real operations — not just previews. The semantic layer encodes all of this once, and the AI can use it to build queries without needing to guess which tables are related to each other, whether it’s one-to-one or many-to-many, etc.

We organized our data into seven domains: Revenue, Cloud, Core, Clickstream, Community, Support, and People. Each domain contains cubes (think of them as well-defined, composable views) with explicit measures, dimensions, and joins. Here’s a real example from our Cloud domain (trimmed for readability):

cubes:
 - name: fct_pulumi_operations
 sql_table: CLOUD.FCT_PULUMI_OPERATIONS
 description: >
 Pulumi CLI operations (update, preview, destroy, refresh).
 Each row is one operation with resource changes, duration,
 and CLI environment.

 joins:
 - name: dim_organization
 sql: '{CUBE}.ORGANIZATION_HK = {dim_organization}.ORGANIZATION_HK'
 relationship: many_to_one
 - name: dim_stacks
 sql: '{CUBE}.STACK_PROGRAM_HK = {dim_stacks}.STACK_PROGRAM_HK'
 relationship: many_to_one
 - name: dim_user
 sql: '{CUBE}.USER_HK = {dim_user}.USER_HK'
 relationship: many_to_one

 measures:
 - name: count
 type: count
 - name: resource_count
 sql: '{CUBE}.RESOURCE_COUNT'
 type: sum
 description: Number of resources active when the operation finished
 - name: operations_succeeded
 sql: "CASE WHEN {CUBE}.OPERATION_STATE = 'succeeded' THEN 1 ELSE 0 END"
 type: sum
 description: Count of operations that succeeded
 # ... plus other measures

 dimensions:
 - name: operation_type
 sql: '{CUBE}.OPERATION_TYPE'
 type: string
 description: Type of operation (Refresh, Update, Destroy, etc)
 - name: operation_state
 sql: '{CUBE}.OPERATION_STATE'
 type: string
 description: Last known state of this operation
 # ... 20+ more dimensions

The key insight: the semantic layer makes the AI’s job tractable. Instead of generating arbitrary SQL from scratch, where the search space is “any possible SQL query against hundreds of tables,” the AI picks from a defined set of measures, dimensions, and joins. The search space shrinks from almost infinite to a well-bounded set of valid combinations.

A semantic layer is to an AI data assistant what a type system is to a programming language. It doesn’t eliminate errors, but it makes entire categories of mistakes structurally impossible. The AI can’t calculate ARR wrong because it doesn’t calculate ARR at all. It references a pre-defined measure that already encodes the correct logic.

Building the semantic layer was mainly data engineering work. We already had agreed-upon metric definitions across the company. The challenge was encoding those definitions into Cube: specifying the correct joins between tables, wiring up the right filters, and making sure every measure matched the logic our dashboards already used. Tedious, but essential.

Adding the AI layer

With the semantic layer in place, the AI becomes a translation problem: convert natural language into a Cube query.

Platybot supports multiple models: Claude Opus 4.6, Claude Sonnet 4.5, and Gemini 3 Pro. Users can choose which model to use. We found that different models have different strengths. Claude excels at structured data queries, while Gemini performs very well on text-heavy tasks like analyzing call transcriptions.

The system prompt gives the model awareness of available cubes, their measures, dimensions, and joins. When a user asks “What’s the ARR breakdown by plan type?”, the model doesn’t write SQL. Instead, it constructs a Cube query, selecting the total_arr measure from the ARR table and grouping by the sku from the subscriptions dimension. Cube handles the SQL generation, the joins, and the filters. For edge cases the semantic layer doesn’t cover, the model can fall back to direct (read-only) SQL against Snowflake, but it may already have a basic query that is already close to what it needs.

Query generation follows a workflow that maps to the same tools a human analyst would use:

flowchart TB
subgraph Entry["Entry points"]
Web["Web UI"]
Slack["Slack (@platybot)"]
MCP["MCP Server"]
end
subgraph Core["Platybot core"]
BE["Backend (Express + LLM)"]
end
subgraph Data["Data layer"]
Cube["Cube semantic layer"]
SF["Snowflake"]
end
Web --> BE
Slack --> BE
MCP --> BE
BE -->|"Structured queries"| Cube
BE -->|"Fallback SQL (read-only)"| SF
Cube --> SF

The workflow is: discover domains, explore cubes, understand the schema, construct and execute the query. This mirrors how a data analyst would work. You don’t jump straight to SQL; you first understand what data is available and what the metrics mean.

The system’s role is narrower than people expect. It’s a translator between human intent and a well-defined data model, not a general-purpose data scientist. This is a feature, not a limitation. By constraining Platybot to operate within the semantic layer, we get reliability. The creativity goes into understanding the question, not into inventing SQL.

Meeting users where they are

The backend solved query generation. It did not solve usage and if people don’t use it, it doesn’t matter. We launched Platybot across three interfaces, each designed for a different workflow.

The web UI

The web app is the primary interface: a React 19 + TypeScript + Vite + Tailwind conversational UI where employees can explore data through multi-turn conversations. It supports table visualization, data export, and conversation history so you can pick up where you left off.

Every analysis produces a shareable report with a permanent link, protected behind company authentication. The report shows the agent’s reasoning so you can verify its approach, and lists every query and table used along with a sample of the data. Each query includes a direct link to run it in Metabase, our reporting tool, so any useful query can be saved, scheduled, or extended without starting from scratch.

The web UI sees the highest adoption of all three access points. It’s where people go for deeper data exploration: multi-step analyses, follow-up questions, and comparing metrics across different dimensions.

One fun detail we’ve added during our last hackathon: while queries iterate through analysis steps (discovering domains, exploring cubes, executing queries), users are entertained by a platypus-themed runner game. Think Chrome’s dinosaur game, but with our mascot on a skateboard. Sometimes Platybot can take a long time iterating, so you can try to beat your high score in the meantime!

Slack

Platybot is also available as a Slack bot. Users @mention it in any channel, and it replies in a thread with the answer, keeping the channel clean while making results visible to the whole team. It’s best suited for quick lookups — “what’s the ARR for account X?” — where the answer fits in a message rather than a deep analysis. Most usage stayed in the web UI, but the Slack bot fills a different niche: answers that benefit from being shared in context.

MCP: Making Platybot usable by Agents

The Model Context Protocol (MCP) server is the newest addition, launched February 4, 2026. It exposes six tools that any MCP-compatible client can use:

list_domains - discover available data domains
list_cubes - explore cubes within a domain
get_cube_details - get measures, dimensions, and joins for a cube
execute_cube_query - run a structured query
execute_sql_query - raw SQL fallback (read-only)
get_generated_sql - preview SQL without executing

Setup is a single command:

claude mcp add --transport http platybot <platybot-mcp-url>

This is a paradigm shift. Platybot goes from a destination (open the app, ask a question) to a capability that any AI tool can use. An engineer writing a postmortem can pull live metrics without leaving their terminal. An analyst building a report in Claude Code can query data inline. The Data Warehouse becomes ambient, always available, never in the way.

Security uses OAuth 2.0 Device Authorization Flow with PKCE for CLI-friendly authentication, with @pulumi.com domain restriction, read-only enforcement, rate limiting, and full audit logging.

A meta moment: we used the Platybot MCP server to query our Slack messages table to find the real usage data cited above.

Deploying with Pulumi

Platybot’s infrastructure runs on AWS and is managed entirely with Pulumi IaC. The stack includes ECS Fargate for the application, an Application Load Balancer for traffic routing, EFS for persistent storage, ECR for container images, and Route 53 for DNS. The setup is intentionally simple and we use a small instance size since it’s just for internal usage.

Here’s a representative snippet of the Fargate service definition:

const service = new awsx.ecs.FargateService("platybot", {
 cluster: cluster.arn,
 assignPublicIp: false,
 taskDefinitionArgs: {
 container: {
 name: "platybot",
 image: image.imageUri,
 essential: true,
 portMappings: [{
 containerPort: 3000,
 targetGroup: targetGroup,
 }],
 environment: [
 { name: "CUBE_API_URL", value: cubeApiUrl },
 { name: "NODE_ENV", value: "production" },
 ],
 secrets: [
 { name: "ANTHROPIC_API_KEY", valueFrom: anthropicKeySecret.arn },
 { name: "SNOWFLAKE_PASSWORD", valueFrom: snowflakePasswordSecret.arn },
 ],
 logConfiguration: {
 logDriver: "awslogs",
 options: {
 "awslogs-group": logGroup.name,
 "awslogs-region": aws.config.region!,
 "awslogs-stream-prefix": "platybot",
 },
 },
 },
 },
});

Dogfooding Pulumi for our internal tools has real benefits beyond the obvious. Staging environments are trivial: spin up a full copy of the stack with different parameters. Secrets management is built in, so API keys and database credentials never touch a config file. And when something needs to change, the diff-and-preview workflow (pulumi preview) catches mistakes before they hit production.

Results

Since launch in September 2025: over 1,700 questions from 83 employees across every team. Usage grew steadily — from around 8 questions per day in the first month to 18 per day by January 2026, with 51 unique users that month alone. This was real production work, not experimentation. Customer analysis for sales calls, resource breakdowns for account managers, blog performance metrics for marketing, policy adoption research for product, ARR deep-dives for leadership.

The impact on the data team was immediate. Questions that used to land in the #analytics channel and wait for a human now get answered in seconds. The data team shifted from answering routine queries to building better models and improving data quality. We went from being a help desk to being a platform team.

Rumors of our replacement have been greatly exaggerated. Platybot just freed us up for the work that doesn't fit in a Slack reply.

Accuracy is harder to quantify, but the semantic layer gives us confidence. Because Platybot uses pre-defined measures instead of generating arbitrary SQL, entire categories of errors (wrong joins, missing filters, incorrect aggregations) are structurally less likely. When the model does get something wrong, it’s usually in interpreting the question, not in the data — and users can verify that through the report’s reasoning trace.

What we learned

The semantic layer matters more than the model. We spent more time defining metrics in Cube than we did on any AI work. That investment pays for itself: swap the model, and the answers stay correct. Swap the semantic layer, and nothing works.

Meet users where they already are. Different UIs handle different cognitive loads. Slack catches the quick questions, the web UI handles deep exploration, and MCP makes data ambient for AI-native workflows. Each interface serves a different mode of thinking.

Transparency builds trust. Showing the AI’s reasoning, the queries it ran, and linking to Metabase for verification turned skeptics into regular users. People don’t trust a black box, but they’ll trust a tool that shows its work.

Resource Search - AI Assist is Generally Available

Meagan Cojocar — Mon, 07 Aug 2023 08:42:44 -0700

Pulumi Cloud Resource Search AI assist functionality is now generally available to all organizations! In addition we have shipped some improvements to the feature to make it easier to use and more discoverable: a toggle on the search bar, suggested queries and an “I’m Feeling Lucky” button to generate a random query for you.

See it in action below!

AI Assist is Generally Available

Resource Search AI assist is a way to search your Pulumi organization using natural language. We launched it in a private beta in April this year. Previously only customers who requested private beta access have been able to leverage natural language resource search, starting today customers on any edition of Pulumi Cloud, AI assist is generally available. You can provide a plain English (or Spanish, or French, or Japanese!) request, to express queries where you might not know the exact syntax, type tokens, or package names. AI Assist makes it easier to gain insight over infrastructure, using queries like:

show me all s3 buckets not tagged in production
show azure and azure native security groups
show all AWS VPCs

Suggested Queries and I’m Feeling Lucky

A blank search bar can be intimidating. We now have suggested natural language searches to get you up and running faster. We also added “I’m Feeling Lucky” functionality that generates random queries for you. This can be a good way to quickly understand the types of insights Resource Search can deliver for your organization.

Wrapping Up

As always, submit any feedback on the feature in the Pulumi Cloud Requests repository.

Pulumi Insights: Intelligence for Cloud Infrastructure

Luke Hoban — Thu, 13 Apr 2023 06:00:00 -0800

Note: This post discusses Pulumi Copilot, which Pulumi Neo has replaced. Learn about Neo →

You can now get Insights capabilities on all your cloud infrastructure, even resources not managed by Pulumi IaC. Get started with Insights 2.0

There are new intelligent cloud management capabilities available in Pulumi Copilot. Learn More

We’ve seen incredible acceleration of cloud adoption over the past 5 years. Pulumi’s flagship open source IaC solution gives engineers great tools to scale up their cloud infrastructure using the same programming languages and tools they already know and love. As a result, thousands of companies of every size and scale have adopted Pulumi as a lynchpin of their cloud infrastructure strategy.

Today we’re excited to announce Pulumi Insights, the next major productivity enhancement for infrastructure as code. Pulumi Insights provides intelligence, search, and analytics over any infrastructure, in any cloud across your organization, leveraging the latest advances in generative AI and Large Language Models (LLMs). Whether you have an AWS VPC, a Kubernetes CRD, or a DataDog alarm definition, Pulumi Insights enables you to intelligently find and interact with all of your resources from within the Pulumi Cloud.

Pulumi Insights was born from partnering with our community and observing their struggles to make sense of millions of cloud resources and resource updates per month, across hundreds of clouds, regions, and accounts. With Pulumi Insights, we now help teams manage cloud footprints of any size with the following key components:

🧠 Intelligence: New AI-powered tool for learning, discovering and building cloud infrastructure with Pulumi, using the power of generative AI and Large Language Models (LLMs).
🔎 Search: Multi-cloud resource search, powered by the Pulumi resource supergraph, which surfaces rich metadata and relationships between cloud infrastructure.
📊 Analytics: Visualize Pulumi resource data with your favorite BI tools, including Excel, Tableau and others, using your own data warehouse for reporting on cost, compliance, and other operational use cases.

All of these features work across Pulumi’s 130+ supported cloud providers, including AWS, Azure, Google Cloud, Cloudflare, and Kubernetes, and can be defined in any programming language, including TypeScript, Python, .NET, Go, Java and YAML.

Pulumi Insights is available for all Pulumi Cloud users today.

Intelligence - Productivity with the Power of AI

The scale of Cloud APIs is often intimidating for new and even seasoned cloud engineers. At Pulumi, we provide access to hundreds of clouds, tens-of-thousands of resources and hundreds-of-thousands of resource properties. Pulumi has many thousands of examples showing how to solve specific problems, and yet developers still sometimes struggle to find the perfect code snippet that addresses their exact set of requirements.

Our new Pulumi Neo is an assistant purpose-built for creating cloud infrastructure using Pulumi. It builds on the power of Large Language Models and GPT to dramatically reduce the time it takes to discover, learn and use new cloud infrastructure APIs. Because Pulumi uses popular programming languages that GPT has already been trained on, Pulumi AI understands how to solve complex Cloud Infrastructure problems.

Using natural language, you can ask to accomplish a specific Cloud Infrastructure goal, and receive a Pulumi program in your favorite language. In addition, Pulumi AI supports iterating on your cloud infrastructure allowing the addition of new features, improving security and performance, fixing correctness issues and clarifying requirements.

The open source pulumi-ai CLI takes these ideas even further, by interactively deploying any cloud infrastructure you request using natural language prompts. You can build arbitrarily complex cloud infrastructure, without requiring detailed knowledge of specific APIs! Of course, since we’re using Pulumi’s Automation API under the hood, you can go to the Pulumi Cloud console to see everything that has been deployed, and even jump straight into your Cloud Console to inspect the results.

Pulumi AI will enhance your team’s ability to go from concept to product. This demo shows Pulumi AI taking natural language input and deploying cloud infrastructure and services which are exposed live on the internet, constantly updating as you refine your request.

Check out Pulumi Neo today at pulumi.com/neo.

Search - Find Anything in Any Cloud

Teams don’t use just a single cloud or a single cloud account. Indeed, many teams manage resources across dozens of clouds and hundreds of accounts! It can be almost impossible to aggregate and search your infrastructure across these accounts, since every cloud and every account is (by design!) a walled garden.

Pulumi Resource Search offers multi-cloud search and analytics across every environment in your organization. You can issue queries that find all of your AWS VPCs, or all of your VPCs in AWS and Azure, or all resources with the “production” tag across all cloud environments. You also have access to statistics about cloud usage, including a breakdown by cloud provider, resource type, and department. Resource search enables you to find the needle within your cloud haystack and visualize cloud consumption trends.

Infrastructure can be discovered interactively with a rich, structured query language, for example:

package:snowflake
data-warehouse stack:production package:aws modified:>=2023-03-21
stack:production type:”github:index/issuelabel:IssueLabel”
package:aws project:pulumi-cloud-import-aws stack:-staging-us-west-2

Search is powered by the Pulumi resource supergraph, which exposes rich metadata around cloud infrastructure resources and models relationships between resources, such as the project and stack they are deployed from, the components they are part of, the provider used to manage them. Pulumi enables faceted search, which allows users to specify criteria by applying multiple filters based on classification of resources.

AI Assist for Search

We are also making available today a private preview of AI Assist for resource search that enables natural language queries. You can provide a plain English (or Spanish, or French, or Japanese!) request, to express queries where you might not know the exact syntax, type tokens, or package names. AI Assist makes it easier to gain insight over infrastructure. You can join the waitlist now for access to this additional functionality for search.

Cloud Import

While all resources managed by Pulumi are part of the resource supergraph, you can also bring resources defined and managed outside of Pulumi into the graph to make them available for Insights. We’re also opening up a private preview of Pulumi Cloud Import, a feature that lets you apply Search and Insights to resources created outside of Pulumi, whether via ClickOps or through other tools like Terraform, CloudFormation, ARM or Kubernetes YAML. You can join the waitlist for access to these new features today.

Analytics - Deeper Insights for Cloud Infrastructure

We are also making the data in the Pulumi supergraph available for export into other Business Intelligence tools for deeper analysis in combination with other key business data. You can visualize resource data with your favorite BI tools, including Excel, Tableau, Looker, PowerBI and others, using your own data warehouse, such as Snowflake, Redshift, Databricks and more, and create reporting on cost, compliance, and many other operational use cases.

The ability to join Pulumi resource data with other key business data in your data warehouse enables users to address a broad range of key business needs - answering questions like:

What infrastructure is stale and possibly orphaned / contributing to our cost?
Where are we spending the most, broken down by department or environment?
Are all my primary data stores marked as protected to avoid unintentional deletion?
What are my most commonly used clouds? What are the most common resource types?
Who has performed the most updates across my organization?

For example, here is a dashboard in Metabase, created by combining Pulumi resource data with cost and department data in a Snowflake data warehouse.

Pulumi resource Data Export is available today both interactively in the console and via REST API . Learn more about these features in the Pulumi Data Export documentation.

Conclusion

We’d like to express our thanks to the entire community that helped us design, develop, test and ship this incredible release. From Intelligence to Search to Analytics, the new features in Pulumi Insights will make all of us more productive. We’re incredibly excited by the use cases we’ve already seen our customers build, and we can’t wait to see what you build on top of these foundations!

To get started with Pulumi Insights today:

Sign up or log in to Pulumi Cloud
Try out Pulumi Neo
Watch the videos for Pulumi Insights, Pulumi AI and Pulumi Search
Check out the Pulumi Insights Documentation
Join the Pulumi Community Slack

Data Science in the Cloud

Sophia Parafina — Thu, 18 Mar 2021 00:00:00 +0000

Data science has advanced because tools like Jupyter Notebook hide complexity by running high level code for the specific problem they are trying to solve. Increasing the level of abstraction lets a data scientist be more productive by reducing the effort to try multiple approaches to near zero, which encourages experimentation and better results.

Data scientists typically work locally, but they often store data for analyses and models in the cloud. There are clear advantages to using cloud resources for these tasks:

Data scientists generally don’t want to manage their storage and databases.
They need to be able to store large data sets cheaply.
They need large capacity swings available on-demand.

SDKs like AWS’ Python library, boto3, can create resources, but they still require domain expertise to manage and properly architect a solution. The Pulumi Automation API improves on raw SDKs by providing high-level abstractions for creating and managing cloud services, letting data scientists concentrate on analyses and models without being well-versed in cloud APIs.

How Pulumi Automation API Improves on Cloud SDKs

Pick any cloud provider, and you will find a cloud SDK in various languages, for example AWS SDKs, Azure SDKs, and Google Cloud SDKs. These SDKs directly access cloud resources, letting data scientists provision resources as required. The drawback to using these libraries is that the data scientist is also responsible for providing resources reliably and securely and performing modeling and analysis.

Pulumi reduces the burden of creating and maintaining repeatable and robust deployments by providing the latest language runtimes, concurrency management, support for diffing updates, and maintaining state with robust checkpoints. Your infrastructure will always be in a well-defined and recoverable state.

Automation API takes this one step further by providing an API overlay to Pulumi’s Infrastructure as Code engine. A data scientist can build infrastructure programmatically within Jupyter to automate building and creating infrastructure as functions and call them as needed. Let’s take a look at an example using a Jupyter notebook.

Infrastructure as a function

As a data scientist, you might want to share data, charts, or graphs with remote team members and make them available through a shared website. You want to keep iterating on your model but don’t want to update the site manually, or worse, let it get out of date.

This example is a simple overview of Automation API that demonstrates a basic deployment of a static website from within a Jupyter notebook. This example uses an inline program to define infrastructure within a function alongside your other code. The function called s3_static_site creates an s3 bucket, sets it up as a basic static website, and exports the URL. This is the equivalent of creating a Pulumi stack using the CLI.

We’ve added code to the example to upload files written to the www directory. Every time you want to update the website, you can write a file to the directory and call the function to upload it to the s3 bucket. Pulumi maintains state, so it will only upload new files when the function is called.

from pulumi_aws import s3

def s3_static_site():
 # Create a bucket and expose a website index document
 site_bucket = s3.Bucket("s3-website-bucket", website=s3.BucketWebsiteArgs(index_document="index.html"))
 index_content = """
 <html>
 <head><title>Hello S3</title><meta charset="UTF-8"></head>
 <body>
 <p>Hello, world!</p>
 <p>Made with ❤️ with <a href="https://pulumi.com">Pulumi</a></p>
 </body>
 </html>
 """

 # Write our index.html into the site bucket
 s3.BucketObject("index",
 bucket=site_bucket.id, # reference to the s3.Bucket object
 content=index_content,
 key="index.html", # set the key of the object
 content_type="text/html; charset=utf-8") # set the MIME type of the file

 # Set the access policy for the bucket so all objects are readable
 s3.BucketPolicy("bucket-policy", bucket=site_bucket.id, policy={
 "Version": "2012-10-17",
 "Statement": {
 "Effect": "Allow",
 "Principal": "*",
 "Action": ["s3:GetObject"],
 # Policy refers to bucket explicitly
 "Resource": [pulumi.Output.concat("arn:aws:s3:::", site_bucket.id, "/*")]
 },
 })

 # Export the website URL
 pulumi.export("website_url", site_bucket.website_endpoint)

We can write additional functions to deploy and destroy the S3 static website. We deploy the stack by calling the deploy_project function and pass it the s3_static_site function, the Pulumi AWS plugin, and the AWS region.

s3_site = deploy_project("my_first_project",
 s3_static_site,
 plugins=[("aws", "v3.23.0")],
 config={"aws:region": auto.ConfigValue(value="us-west-2")})

You can view the resources that the function created in two ways. The standard method is to open the AWS console to see your resources.

You can also view them with the Pulumi Service as a resource graph.

The Pulumi Service also displays updates to the static site and the changes made.

Now that the static site has been deployed, we want to get the site URL. The s3_static_site function exports the website url. Pulumi returns the s3_site as an object with outputs that we use to retrieve the URL. There are several ways to check if the site is working, but a simple way is to open the URL in a web browser.

import webbrowser

outputs = s3_site.outputs()

webbrowser.open(url)

Data Science Your Way

In this simple example, we demonstrated how to create a static website from an S3 bucket, write content into the bucket, and set the access policy as a Python function. We called the function, which created a stack for the website. The stack is returned as a Python object with the site URL as an output. You can manage the site and view updates using the Pulumi Service.

Because it’s code, you can create a separate module containing all the functions to create your infrastructure, import it into your notebooks, and reuse it. Pulumi lets you perform analysis and build models within a familiar environment while reducing the burden of managing infrastructure. You can find the complete example on GitHub including a Jupyter notebook for database migration.

Be sure to watch our video that explains the code in greater detail.

AWS Serverless Analytics

Evan Boyle — Thu, 30 Jan 2020 00:00:00 +0000

Whether it’s an IoT installation, a website, or a mobile app, modern software systems generate a trove of usage and performance data. While it can be daunting to collect and manage, surfacing data empowers the business to make informed product investments. In this article, we’ll explore the following:

An overview of the traditional Redshift analytics stack on AWS, the use cases it excels at, and where it falls apart.
An alternative architecture utilizing serverless and streaming.
How to implement this architecture as code in a reusable library with Pulumi.
How to automate the development loop when writing Pulumi libraries.

If you’d like to follow along, you can clone and run the reference implementation. If you’re new to Pulumi, you can follow this guide to get started.

ACME E-Commerce

ACME is a burgeoning online retailer. They ship goods around the world to millions of happy customers. ACME has recently started an incubation project to test out running ads on its website. They are unsure of how customers will react or whether their merchants will be interested in participating, so they staff a small team of 5 tasked with launching an MVP.

ACME wants to track the performance of its ads, understand how its customers are interacting with them, and look for patterns that can inform business decisions and further investment. With such a small team, they look for the path of least resistance as analytics is just a small slice of the overall work to be done to launch on their aggressive timeline.

The Traditional Analytics Stack

A familiar SQL interface and the ease of loading data through built-in features like S3 COPY have made Redshift the goto choice for bootstrapping an analytics stack on new projects. It’s low friction to get from zero to something, and Redshift scales to petabytes.

Like all tools, Redshift has targeted use cases that it excels at. Broadly speaking, it’s niche is heavy BI and analytics workloads, reading more than you write. Redshift can serve millisecond-level queries on massive datasets through a horizontally scalable, distributed processing architecture. At the core of this is a reliance on sorting and data distribution mechanisms. Schemas are configured by administrators to distribute the data across the cluster in a variety of methods that optimize for different use cases, such as highly denormalized tables, or tables making heavy use of joins. Distributed data is stored in a specified sort order on disk, allowing the query planner to make optimizations.

New data is sorted locally when a COPY command is run, but this doesn’t mean that the global sort order is maintained. A VACUUM and ANALYZE command must be run to re-sort the data, rebuild indexes, and update metadata used by the query planner. It can be a resource-intensive process relative to how much data you’ve loaded, and how it’s distributed amongst your existing sort key range. This limits the effective use of Redshift.

ACME Ads MVP

The team defines its initial product. They’ll begin with testing ads in the search results of ACME’s North American website. When customers search for “Soap”, advertisers can bid to have their product show up first. They’ve picked a cost per click (CPC) billing model, where advertisers pay a small sum whenever a user clicks on their ad. To support this, they’ve designed an initial architecture:

They’ve defined the following key events to track:

Impression: Emitted when the ad is rendered on a customer device.
Click: Emitted when the customer clicks on the product shown in the ad.

The shopping web app has a mechanism to deliver these events into an S3 bucket. On an as-needed basis, engineers on the team can manually run a COPY command to refresh the data in the cluster. They do this on a monthly cadence to generate invoices to send to advertisers, and sometimes more frequently to generate ad-hoc reports on ad performance.

Scaling a Real Business

The team is delighted to find that the ads are performing well. Advertisers see a return on their investment, and data shows that the impact on the consumer experience is minimal. The ACME team has scaled the number of advertisers from five in the beta program, to over a thousand. The team struggled with the ad-hoc nature of its MVP analytics stack and has invested heavily in automation:

Loading of event data (pre-impression, clicks, etc.) into Redshift has been automated into an hourly cron that runs an incremental COPY.
Billing is automated to run on an hourly basis, to limit the amount of time an advertiser can be allowed to serve ads after exceeding their allocated budget.

The automated load, in addition to the data generated by the recently hired Product Manager, Business Intelligence Analyst, and Data Scientist, strained the Redshift deployment. It has quickly become the most expensive part of their system, not just in terms of dollars spent, but operator time as well. An overloaded cluster leads to a variety of operational issues that the team must learn to detect and mitigate:

Data delivery is regularly delayed: COPY commands that used to take minutes are taking over an hour when the cluster is under load. Dependent cron jobs over the data need to be updated to handle delays.
The cluster must be resized regularly to keep up with growth: The team often uses Elastic Resize operations, which require a few minutes of downtime and only allow the cluster to increase in size by factors of two. Upgrading node types to better match the workload requires downtime as a new cluster is stood up, and data is copied over.
Ad-hoc workloads interfere with production: Business users write sophisticated queries that accidentally bring the cluster to a standstill. The team puts a complicated WLM policy in place to try and isolate ad-hoc from production, but this is more of an art than a science.

Designing the V2 Analytics Stack

The team understands how to keep the analytics stack limping along at this point, but the cost in both hardware and developer time is too high. They decide to pursue a redesign with the following goals in mind:

Little to no operator intervention required. Developer time is their most valuable asset, and going forward, they want to focus engineering on delivering new features to advertisers, and new experiences to customers.
Scale the analytics stack to 1B events per day and beyond as the program grows in terms of advertisers, and geography.
Enable workload isolation between production services and ad-hoc business intelligence users.

JSON events are queued in Kinesis, consumed by a Firehose that validates their schema against the definition in Glue, then converted into parquet, and finally stored in S3 in hourly folders based on arrival time. Structured data in S3 and a Glue metastore are flexible primitives that end-users and production systems can build on. Both production systems and ad-hoc users can bring their own compute or take advantage of serverless solutions like Athena (the AWS serverless version of Presto) to query over the data with isolation. The operator overhead is low because it relies almost entirely on serverless offerings. This architecture brings some additional benefits:

Scale the read and write workloads independently. Redshift was overprovisioned to handle the mismatch of a continuous write workload at peak hours. A decrease in operating cost was the result of adopting this architecture.
Pay as you go. Firehose, S3, Lambda, and Athena are all billed based on usage. Kinesis is the exception charging per shard hour, which for most deployments would be provisioned for peak load.
Configurable latency. Firehose can dump batches of data in intervals of 1-15 minutes. If low latency is a concern, you can achieve it (see the end for caveats on compaction). We don’t have to worry about an increased write workload impacting reads.
Malformed records won’t block the data ingestion pipeline. Redshift COPY has a maxerror that allows skipping over bad records, but there isn’t any automation to isolate and alert on those records. Firehose writes unsuccessful events to a separate directory and allows processing to continue.

While our new serverless streaming architecture brings a slew of benefits that enable ease of operation and the next level of scale, there are some drawbacks. For one, there is substantially more infrastructure to configure than the single Redshift cluster. How do we implement this in a repeatable fashion?

Architecture as Code with Pulumi

We want repeatable infrastructure that we can reliably develop, test, and deploy. Creating a single Redshift instance through the console was manageable, but our serverless architecture utilizes multiple instances of five AWS services. Serverless solutions frequently require many more granular resources, and thus the complexity of the infrastructure as code increases, and we need more robust tools to manage this complexity - just like we manage software complexity.

Pulumi takes care of the complexity by bringing software development best practices like abstraction, encapsulation, and modularity to infrastructure. You can use design patterns in your favorite programming language to provision modern cloud infrastructure. Component Resources are our tools for encapsulating low-level cloud infrastructure into Architecture as Code that is shareable across teams.

With Pulumi the code we write looks very similar to our architecture diagram:

import { ServerlessDataWarehouse, StreamingInputTableArgs }
 from "./datawarehouse";

// declare our shared schema for clicks and impressions
const columns = [
 {
 name: "id",
 type: "string",
 },
 {
 name: "session_id",
 type: "string",
 },
 {
 name: "event_time",
 type: "string",
 },
];

const impressionsTableName = "impressions";
const clicksTableName = "clicks";

const tableArgs: StreamingInputTableArgs = {
 columns,
 inputStreamShardCount: 1,
 region: "us-west-2",
};

// create two tables with kinesis input streams,
// writing data into hourly partitions in S3.
const dataWarehouse = new ServerlessDataWarehouse("analytics_dw", { isDev })
 .withStreamingInputTable(impressionsTableName, genericTableArgs)
 .withStreamingInputTable(clicksTableName, genericTableArgs);

// Make input streams available to external programs as stack outputs
export const impressionInputStream = impressionsInputStream.name;
export const clickInputStream = clicksInputStream.name;

In a dozen lines of code, we’ve provisioned two of our desired tables, “clicks” and “impressions”, using our serverless streaming input architecture. It takes care of everything, including implicitly creating an arrival time partition scheme with the key “inserted_at”. We export our Kinesis input streams as stack outputs that can be referenced in other projects, such as the instrumentation within our ad server or consumer-facing web app.

While on the surface this Pulumi component is described imperatively, it produces a declarative output in the form of a state file that can be managed locally, in an object store like S3, or by the Pulumi Service backend. Running a ‘pulumi up’ shows that we’ve created 45 AWS resources, and lists our stack outputs to the console.

After putting some sample events into Kinesis, we can use Athena to query our impressions table:

What does ServerlessDataWarehouse look like under the hood? Just like any other typescript class:

export class ServerlessDataWarehouse extends pulumi.ComponentResource {

 public dataWarehouseBucket: aws.s3.Bucket;
 public queryResultsBucket: aws.s3.Bucket;
 public database: aws.glue.CatalogDatabase;
 private tables: { [key: string]: aws.glue.CatalogTable } = {};
 private inputStreams: { [key: string]: aws.kinesis.Stream } = {};

 constructor(name: string,
 args: DataWarehouseArgs = {},
 opts: pulumi.ComponentResourceOptions = {}
 ) {
 super("serverless:data_warehouse", name, opts);

 const bucketArgs: BucketArgs | undefined = args.isDev ?
 { forceDestroy: true } : undefined;

 const dataWarehouseBucket = new aws.s3.Bucket("datawarehouse-bucket",
 bucketArgs, { parent: this });
 const queryResultsBucket = new aws.s3.Bucket("query-results-bucket",
 bucketArgs, { parent: this });

 const database = args.database || new aws.glue.CatalogDatabase(name, {
 name,
 }, { parent: this });


 this.dataWarehouseBucket = dataWarehouseBucket;
 this.queryResultsBucket = queryResultsBucket;
 this.database = database;
 }
}

The ServerlessDataWarehouse class creates a shared S3 data bucket, Athena query results bucket, and Glue database. We’ve exposed a fluent method, .withStreamingInputTable(), that creates the Glue table, Kinesis input stream, Firehose destination stream, and the Lambda cron that registers the hourly partitions. Many of these pieces themselves are also encapsulated into higher-order components.

public withStreamingInputTable(name: string, args: StreamingInputTableArgs):
ServerlessDataWarehouse {
 const {
 partitionKeyName, columns,
 inputStreamShardCount, fileFlushIntervalSeconds
 } = args;
 const partitionKey = partitionKeyName || "inserted_at";

 const partitionKeys: input.glue.CatalogTablePartitionKey[] = [{
 name: partitionKey,
 type: "string",
 }];

 const tableArgs: TableArgs = {
 columns,
 partitionKeys,
 };

 this.withTable(name, tableArgs);

 const streamArgs: InputStreamArgs = {
 destinationBucket: this.dataWarehouseBucket,
 shardCount: inputStreamShardCount,
 databaseName: this.database.name,
 tableName: name,
 fileFlushIntervalSeconds,
 };

 const { inputStream } = new InputStream(`inputstream-${name}`,
 streamArgs, { parent: this });
 this.inputStreams[name] = inputStream;

 const registrarArgs: PartitionRegistrarArgs = {
 database: this.database,
 partitionKey,
 region: args.region,
 dataWarehouseBucket: this.dataWarehouseBucket,
 athenaResultsBucket: this.queryResultsBucket,
 table: name,
 scheduleExpression: args.partitionScheduleExpression,
 };
 const partitionRegistrar = new HourlyPartitionRegistrar(
 `${name}-partitionregistrar`, registrarArgs, { parent: this });

 return this;
}

Our withStreamingInputTable call makes use of another fluent API, .withTable(), that encapsulates the logic around various storage and SerDe details required for storing different formats in glue (Parquet vs. JSON) in a way that Athena can understand:

public withTable(name: string, args: TableArgs): ServerlessDataWarehouse {
 if (this.tables[name]) {
 throw new Error(`Duplicate table! Name: ${name}`);
 }

 const dataFormat = this.validateFormatAndGetDefault(args.dataFormat);

 const table = this.createTable(name, args.columns,
 dataFormat, args.partitionKeys);
 this.tables[name] = table;

 return this;
}

private createTable(name: string,
 columns: input.glue.CatalogTableStorageDescriptorColumn[],
 dataFormat: DataFormat,
 partitionKeys?: input.glue.CatalogTablePartitionKey[]):
 aws.glue.CatalogTable {
 const location = getS3Location(this.dataWarehouseBucket, name);

 const parquetStorageDescriptor = {
 location,
 inputFormat:
 "org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat",
 outputFormat:
 "org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat",
 serDeInfo: {
 parameters: { "serialization.format": "1" },
 name: "ParquetHiveSerDe",
 serializationLibrary:
 "org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe",
 },
 columns,
 };

 const jsonStorageDescriptor = {
 location,
 inputFormat:
 "org.apache.hadoop.mapred.TextInputFormat",
 outputFormat:
 "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat",
 serDeInfo: {
 name: "OpenXJSONSerDe",
 serializationLibrary: "org.openx.data.jsonserde.JsonSerDe",
 },
 columns,
 };

 const storageDescriptor = dataFormat === "JSON" ?
 jsonStorageDescriptor : parquetStorageDescriptor;
 return new aws.glue.CatalogTable(name, {
 name: name,
 databaseName: this.database.name,
 tableType: "EXTERNAL_TABLE",
 storageDescriptor,
 partitionKeys,
 }, { parent: this });
}

Encapsulating core functionality like this enables other analytics use cases, and can act as an escape hatch for more complex ones. Users can create a data warehouse with component owned and managed streaming tables, use .withTable() to enable the same database to own the table, and then write custom external logic to populate that table.

// create a static fact table
const factTableName = "facts";
const factColumns = [
 {
 name: "thing",
 type: "string",
 },
 {
 name: "color",
 type: "string",
 },
];

const factTableArgs: TableArgs = {
 columns: factColumns,
 dataFormat: "JSON",
};

dataWarehouse.withTable("facts", factTableArgs);

// Load a static facts file into the facts table.
const data = `{"thing": "sky", "color": "blue"}\n` +
 `{ "thing": "seattle sky", "color": "grey"}\n` +
 `{ "thing": "oranges", "color": "orange"}`;

const factJSON = new aws.s3.BucketObject("factsFile", {
 bucket: dataWarehouse.dataWarehouseBucket,
 content: data,
 key: `${factTableName}/facts.json`,
});

Here we create a static fact table that never changes. While this population mechanism is simple, you could imagine something much more sophisticated, like a sequence of EMR jobs instead.

Creating a Batch Table API

We now have a very opinionated streaming table API (withStreamingInputTable), and an escape hatch (withTable). Can we provide something for the general batch use case? It is quite a significant problem, but we can come up with a useful API if we constrain it a bit:

Execute user-specified code on a user-specified interval.
The runtime platform is AWS Lambda. Job must execute within Lambda resource constraints for time, memory, CPU, and disk (15 mins, 3 GB, and 500 MB at the time of writing).
The user must specify policies to attach to the lambda execution role for access to resources needed during the job (S3, Athena, and others).

public withBatchInputTable(name: string, args: BatchInputTableArgs):
ServerlessDataWarehouse {
 const {
 columns, partitionKeys, jobFn, scheduleExpression,
 policyARNsToAttach, dataFormat
 } = args;

 const tableArgs: TableArgs = {
 columns,
 partitionKeys,
 dataFormat,
 };

 // create a table
 this.withTable(name, tableArgs);

 const lambdaCronArgs: LambdaCronJobArgs = {
 jobFn,
 scheduleExpression,
 policyARNsToAttach,
 };

 // configure a lambda to execute on a schedule and
 // run the function to write records to the table
 const batchInputJob = new LambdaCronJob(name, lambdaCronArgs,
 { parent: this });

 return this;
}

Our .withBatchInputTable() API creates a table and schedules a lambda that writes data into the table in batches. In our example below, we create a function that queries our clicks and impressions streaming tables over a given hour and writes a JSON file with aggregations. In fifty lines of code, we can publish higher-level statistics about our streaming tables that can be consumed by business users:

// Configure batch input table 'aggregates'
const aggregateTableName = "aggregates";

const aggregateTableColumns = [
 {
 name: "event_type",
 type: "string",
 },
 {
 name: "count",
 type: "int",
 },
 {
 name: "time",
 type: "string",
 },
];

// function that lambda runs.
// writes records to table that summarize clicks and impressions
const aggregationFunction = async (event: EventRuleEvent) => {
 // configure an athena client for our DW
 const athena = require("athena-client");
 const bucketUri = `s3://${athenaResultsBucket.get()}`;
 const clientConfig = {
 bucketUri,
 };
 const awsConfig = {
 region,
 };
 const athenaClient = athena.createClient(clientConfig, awsConfig);
 const date = moment(event.time);
 // query impressions and clicks tables for hourly event counts
 const partitionKey = date.utc().format("YYYY/MM/DD/HH");
 const getAggregateQuery = (table: string) =>
 `select count(*) from ${databaseName.get()}.${table} ` +
 `where inserted_at='${partitionKey}'`;
 const clicksPromise =
 athenaClient.execute(getAggregateQuery(clicksTableName))
 .toPromise();
 const impressionsPromise =
 athenaClient.execute(getAggregateQuery(impressionsTableName))
 .toPromise();

 const clickRows = await clicksPromise;
 const impressionRows = await impressionsPromise;
 // write the event counts to our table in S3 in JSON format
 const clickCount = clickRows.records[0]["_col0"];
 const impressionsCount = impressionRows.records[0]["_col0"];
 const data = `{ "event_type": "${clicksTableName}", `+
 `"count": ${clickCount}, "time": "${partitionKey}" }\n` +
 `{ "event_type": "${impressionsTableName}", ` +
 `"count": ${impressionsCount}, "time": "${partitionKey}"}`;
 const s3Client = new S3();
 await s3Client.putObject({
 Bucket: dwBucket.get(),
 Key: `${aggregateTableName}/${partitionKey}/results.json`,
 Body: data,
 }).promise();
};

//configure resource access policies required by the function
const policyARNsToAttach: pulumi.Input<ARN>[] = [
 aws.iam.ManagedPolicies.AmazonAthenaFullAccess,
 aws.iam.ManagedPolicies.AmazonS3FullAccess,
];

const aggregateTableArgs: BatchInputTableArgs = {
 columns: aggregateTableColumns,
 jobFn: aggregationFunction,
 scheduleExpression,
 policyARNsToAttach,
 dataFormat: "JSON",
};

dataWarehouse.withBatchInputTable(aggregateTableName, aggregateTableArgs);

While Lambda’s limits restrict the utility of this method, the concept is powerful. Pulumi gives you the building blocks to define your abstractions. Instead of Lambda, the execution plane could be ECS, Kubernetes, or EMR.

Automating the Dev Environment

Developing software is an activity in seeking a specific state. For functions with simple inputs and outputs, this is a matter of well-factored code and unit test coverage. You can and should write this sort of test in your Pulumi applications. But often, the state that you’re seeking can’t be validated in unit tests. It raises many questions such as, “Am I stitching these cloud services together in a valid way? Will data flow from input service through to output destination? If I try to tighten down this IAM role, will everything still work?”

Our workflow is to make a change, run a pulumi up, and then verify that data ingested into Kinesis is read out the other end by Athena. A simple shell script using stack outputs and a query in the AWS console are enough to get started here:

#/bin/sh

aws kinesis put-record \
--stream-name $(pulumi stack output impressionInputStream) \
--region $(pulumi config get aws:region) \
--data '{"session_id": "1", "id": "2", "event_time": "2020-01-23T20:38:42.151Z"}' \
--partition-key '1'

This workflow works well enough, but it has a few problems. First of all, it requires a bunch of manual work. We have to run a pulumi up, wait for it to complete, execute this script, logon to the AWS console, execute an Athena query, and wait for the results to come back. We can automate the data ingestion with more Pulumi in our dev stack.


export class EventGenerator extends pulumi.ComponentResource {
 constructor(name: string,
 args: EventGeneratorArgs,
 opts?: pulumi.CustomResourceOptions
 ) {
 super("serverless:event_generator", name, opts);

 const { eventType, inputStreamName } = args;

 // puts a sample record into our input stream
 const eventGenCallback = (event: EventRuleEvent) => {
 const AWS = require("aws-sdk");
 const uuid = require("uuid/v4");
 const kinesis = new AWS.Kinesis();
 const records: any = [];

 const sessionId = uuid();
 const eventId = uuid();
 const record = {
 Data: JSON.stringify({
 id: eventId,
 session_id: sessionId,
 event_time: event.time,
 }),
 PartitionKey: sessionId,
 };
 records.push(record);

 kinesis.putRecords({
 Records: records,
 StreamName: inputStreamName.get(),
 }, (err: any) => {
 if (err) {
 console.error(err);
 }
 });
 };

 const lambdaCronArgs: LambdaCronJobArgs = {
 jobFn: eventGenCallback,
 scheduleExpression: "rate(1 minute)",
 policyARNsToAttach: [
 aws.iam.ManagedPolicies.AmazonKinesisFullAccess,
 ],
 };

 // schedule a lambda to run our function once a minute.
 const eventGenerator = new LambdaCronJob(`${eventType}-eventGenerator`,
 lambdaCronArgs, { parent: this });
 }
}

This helper creates a Lambda that publishes sample events to Kinesis on a minutely cron. It doesn’t completely automate our validation, but it’s certainly a step closer. Now when we run an incremental update, data automatically propagates through the system. We can use the AWS console to execute Athena queries and observe the output.

Incremental updates in our dev loop are fast, but when dealing with AWS services, entities like IAM roles and Glue schemas can be cached by services like Firehose. It’s entirely possible to make an update that appears to work for a few minutes, just to have it fail minutes later when some implementation detail clears its cache.

To bring 100% automation and certainty, we can write an isolated integration test that stands up new infrastructure, executes our tests, and then tears everything down.

jest.setTimeout(360000);

let runner: PulumiRunner;
const region = "us-west-2";

beforeAll(async () => {
 const config: { [key: string]: string } = {
 "aws:region": region,
 "aws-ts-serverless-datawarehouse:dev": "true",
 };

 const pulumiProjDir = resolve("./");
 runner = new PulumiRunner(config, pulumiProjDir);
 const setupResult = await runner.setup();
 if (!setupResult.success) {
 throw new Error(`Pulumi setup failed, aborting: ${setupResult.error}`);
 }
});

afterAll(async () => {
 const teardownResult = await runner.teardown();
 if (!teardownResult.success) {
 throw new Error(
 `Teardown failed. Test stack has leaked: ${teardownResult.error}`
 );
 }
});

test("WithStreamingInput integrtion test", async () => {
 // assert we see the expected stack outputs
 expect(runner.getStackOutputKeys().length).toBe(6);
 const db = runner.getStackOutput("databaseName");
 const clickTable = runner.getStackOutput("clickTableName");
 const impressionTable = runner.getStackOutput("impressionTableName");
 const bucket = runner.getStackOutput("athenaResultsBucket");

 // query our tables on a retry loop until we find records or timeout and fail
 const clickPromise = verifyRecordsInTable(db, clickTable, bucket);
 const impressionPromise = verifyRecordsInTable(db, impressionTable, bucket);

 const [clickTableHasRecords, impressionTableHasRecords] = await
 Promise.all([clickPromise, impressionPromise]);

 expect(clickTableHasRecords).toBe(true);
 expect(impressionTableHasRecords).toBe(true);
});

This test gives us a great deal of confidence that our changes are valid. While it certainly does a lot, it’s a little slow. We can run this suite with npm run test:int & tput bel to notify us upon completion while we work in the background, but four minutes might not meet the latency bar for core inner dev loop.

Here we’ve seen three different ways to use Pulumi to automate and improve the inner dev loop. Simple scripting with stack outputs to get started, automation of that script using Pulumi, and full blow integration testing to bring 100% confidence in tricky changes. All three are tools in our arsenal to be applied appropriately.

Wrapping Up

ACME re-evaluated their analytics architecture and shipped the next-generation serverless streaming solution that can take them to the next click-stop, and well through the next order of magnitude. Along the way, they learned how to reason about streaming data and examined the underlying assumptions about time and completeness of data in their existing system. ACME developed its streaming architecture using Pulumi and learned how to automate and create reproducible developer environments along the way. The outcome isn’t just an instance of the serverless streaming architecture, but a component library that implements best practices. That investment in time is bundled up and shipped in a format that the next generation of incubators at ACME can pick up and run with.

If you’d like to get started delivering best practices in infrastructure across your organization, get started with Pulumi free today. Clone and run the reference implementation of the Serverless Data Warehouse to build your own serverless analytics stack.

Architectural Caveats

Like every architecture, the serverless streaming model has limitations and pitfalls.

Processing Time vs. Event Time: Our data is now ordered and indexed by when it’s processed as opposed to when the events occur. We need to reason about data arriving late and make sure that our queries and dependent systems account for spikes in latency. For a detailed primer, see Streaming 101.
Data Freshness & Compaction: While it is possible to emit files by the minute to achieve low latency, it can lead to performance issues. In practice, this sort of setup can cause an explosion of tiny files that degrade the performance of query engines. See this article on optimizing file sizes for Athena for more details. A standard solution is to run an hourly compaction job that rewrites data into optimally sized chunks, leaving only the most recent hour or two of data in smaller files.
Cross Account Athena: Due to Glue’s limitations, you cannot grant cross-account access for Athena directly. There are workarounds like temporary credentials and using assume role.
Service Limits: Just as with Redshift, nothing is truly a black box. Athena has a maximum number of concurrent queries, and S3 has default limits on the rate at which you can list and get objects. As you scale, make sure you understand your service limits and reach out to AWS support to proactively increase limits where appropriate.
Query Translation: If you’re migrating an existing system from Redshift to Athena, the queries won’t quite map 1:1. For one, there’s the need to restructure queries to limit first on arrival time, and then narrow down on specific event time. And some of the traditional SQL operations like ORDER BY and GROUP BY will perform differently. See the section on query optimization for more details.
Unbounded Athena Queries: Athena charges $5 per TB scanned during a query. Most business users aren’t used to the concept of pay per query. It’s quite easy to issue a ‘select *’ query that can cost a great deal of money. Guard rails like a middleware that parses queries and ensures usage of a partition key may be necessary to control cost if exposing Athena as an interface to a broader audience.

Create Secure Jupyter Notebooks on Kubernetes using Pulumi

Nishi Davidson — Thu, 30 May 2019 00:00:00 +0000

In this post, we will work through an example that shows how to use Pulumi to create Jupyter Notebooks on Kubernetes. Having worked on Kubernetes since 2015, a couple of critical benefits jump out that may resonate with you as well:

You write everything in code - TypeScript in our example here.
You need not initialize Tiller or Helm to work with existing Helm charts like nginx-ingress-controller that we use here.
The security patterns in Helm and Tiller are no longer concerns, rather you get to focus on the RBAC of the actual service which is Jupyter-notebook in this example.
You accomplish more with less YAML and iteratively work towards your use cases.

Prerequisites

Install Pulumi
Install Node.js version 6 or later
Install a package manager for Node.js, such as npm or Yarn
Install Google Cloud SDK
Install Helm and only initialize with helm init —client-only

We will work this example on a GKE cluster so lets first configure Google Cloud Auth:

$ gcloud auth login
$ gcloud config set project <YOUR_GCP_PROJECT_HERE>
$ gcloud auth application-default login

Step 1: Create a Pulumi Project and Stack with a Pulumi TypeScript Template

$ mkdir gke-jupyter-notebook && cd gke-jupyter-notebook
$ pulumi new typescript
$ npm install --save @pulumi/kubernetes @pulumi/gcp

Step 2: Create a GKE Cluster

To create a GKE cluster, simply update the following code in index.ts file and run pulumi up.

import * as k8s from "@pulumi/kubernetes";
import * as pulumi from "@pulumi/pulumi";
import * as gcp from "@pulumi/gcp";
import { readFileSync, fstat } from "fs";

const name = "jupyter-notebook";

/*
 * STEP 2: Create a GKE Cluster
 */
const engineVersion = gcp.container.getEngineVersions().then(v => v.latestMasterVersion);
const cluster = new gcp.container.Cluster(name, {
 initialNodeCount: 2,
 minMasterVersion: engineVersion,
 nodeVersion: engineVersion,
 nodeConfig: {
 machineType: "n1-standard-1",
 oauthScopes: [
 "https://www.googleapis.com/auth/compute",
 "https://www.googleapis.com/auth/devstorage.read_only",
 "https://www.googleapis.com/auth/logging.write",
 "https://www.googleapis.com/auth/monitoring"
 ],
 },
});

// Export the Cluster name
export const clusterName = cluster.name;

// Manufacture a GKE-style kubeconfig.
export const kubeconfig = pulumi.
 all([ cluster.name, cluster.endpoint, cluster.masterAuth ]).
 apply(([ name, endpoint, masterAuth ]) => {
 const context = `${gcp.config.project}_${gcp.config.zone}_${name}`;
 return `apiVersion: v1
clusters:
- cluster:
 certificate-authority-data: ${masterAuth.clusterCaCertificate}
 server: https://${endpoint}
 name: ${context}
contexts:
- context:
 cluster: ${context}
 user: ${context}
 name: ${context}
current-context: ${context}
kind: Config
preferences: {}
users:
- name: ${context}
 user:
 auth-provider:
 config:
 cmd-args: config config-helper --format=json
 cmd-path: gcloud
 expiry-key: '{.credential.token_expiry}'
 token-key: '{.credential.access_token}'
 name: gcp
`;
 });

// Create a Kubernetes provider instance that uses our cluster from above.
const clusterProvider = new k8s.Provider(name, { kubeconfig: kubeconfig });

Step 3: Create a NGINX-Ingress-Controller to Generate Ingresses

You can use the default L7 load balancer in Google Cloud but here we add an NGINX-Ingress-Controller to the cluster. With Pulumi, you write four lines of typescript to reuse the stable NGINX ingress controller helm chart and have the controller, default backends, RBAC, Service account, Config map all running within seconds.

Add the following lines of code in index.ts file and run pulumi up once again.

/*
 * STEP 3: Create NGINX Ingress Controller in GKE
 */

const nginxingresscntlr = new k8s.helm.v2.Chart("nginxingresscontroller", {
 repo: "stable",
 chart: "nginx-ingress",
 version: "0.24.1",
 values: {},
}, { providers: { kubernetes: clusterProvider } });

Step 4: Create a Jupyter Notebook Deployment and Service with Type NodePort

Bringing up Jupyter notebook deployment and service requires adding the following lines of code in index.ts file and running pulumi up to apply the changes.

/*
 * STEP 4: Create Jupyter notebook deployment and service in the GKE cluster
 */

const appName = "jupyter-notebook";
const appLabels = { app: appName };
const jupyterNotebook = new k8s.apps.v1beta1.Deployment(appName, {
 metadata: { name: appName, labels: appLabels },
 spec: {
 selector: { matchLabels: appLabels },
 replicas: 1,
 template: {
 metadata: { labels: appLabels },
 spec: {
 containers: [
 {
 name: appName,
 image: "jupyter/tensorflow-notebook",
 ports: [{ containerPort: 8888 }],
 command: ["start-notebook.sh"],
 args: ["--NotebookApp.token=''"]
 }]
 }
 }
 }
}, { provider: clusterProvider });

const jupyterService = new k8s.core.v1.Service(appName, {
 metadata: {
 name: appName,
 labels: appLabels
 },
 spec: {
 type: "NodePort",
 selector: appLabels,
 ports: [{ protocol: "TCP", nodePort: 30040, port: 8888, targetPort: 8888 }],
 }
}, { provider: clusterProvider });

Step 5: Create a Secret that is used with your Jupyter Notebook Domain Name

We first create a local auth.txt file with the password using the following command:

htpasswd -c auth.txt jupyter

We then read this file synchronously, convert it to base64 and add it as a secret in the GKE cluster. We use this secret as the TLS password to access the Jupyter notebook ingress endpoint accessible from the domain name defined in the host section of the ingress declaration. The annotations in the ingress declarations are required to enable this behavior on the ingress object.

/*
 * STEP 5: Create a secret to enable "basic-auth" for your Jupyter
 * notebook ingress and add it to the ingress declaration in the
 * GKE cluster
 */

const authContents = (readFileSync("<path-to-auth.txt-file>")).toString()

function toBase64(s: string): string {
 return Buffer.from(s).toString("base64");
}

const authContents_base64 = toBase64(authContents);

const jupyternotebooksecret = new k8s.core.v1.Secret("jupyter-notebook-tls", {
 metadata: { name: "basic-auth", namespace: "default" },
 type: "Opaque",
 data: {
 auth: authContents_base64,
 }
})

export const jupyternotebookingress = new k8s.extensions.v1beta1.Ingress(appName, {
 metadata: {
 name: appName,
 labels: appLabels,
 annotations: {
 "kubernetes.io/tls-acme": "true",
 "kubernetes.io/ingress.class": "nginx",
 "nginx.ingress.kubernetes.io/auth-type": "basic",
 "nginx.ingress.kubernetes.io/auth-secret": "basic-auth",
 "nginx.ingress.kubernetes.io/auth-realm": "Authentication Required - jupyter"
 },
 },
 spec: {
 rules: [{
 host: "nishidavidson.com",
 http: {
 paths: [{
 path: "/",
 backend: { serviceName: "jupyter-notebook", servicePort: 8888 }
 }]
 }
 }],
 tls: [{
 secretName: "jupyter-notebook-tls",
 hosts: ["nishidavidson.com"]
 }],
 }
}, { provider: clusterProvider });

Once complete, you will see the GKE cluster components show up as follows:

Open a browser to access the jupyter-notebook Service using the domain name declared in the host section of your ingress code above. In our example, I used the domain name nishidavidson.com.

When you open your DNS, you will be asked for the “username: jupyter” and “password” for the secret you created in auth.txt file as shown below:

As soon as the password is accepted, You should now be able to go to your website from anywhere in the world and access your password-protected Jupyter notebook running on GKE on secure SSL connection.

Success! We worked through a simple example of creating a GKE cluster, an NGINX ingress controller and stood up our password protected Jupyter notebook Ingress, Service, and Deployment with a simple secret for authentication.

You can get started today using more solutions on Kubernetes from our Pulumi examples repository.

Pulumi Blog: Data and analytics

How We Load Data into Snowflake in Seconds with Pulumi

What we’ll cover

Architecture overview

Project setup

Managing credentials with Pulumi ESC

Shared infrastructure

Building the direct ingestion ComponentResource

The Lambda handler

Why direct Firehose to Snowflake?

The DirectSnowflakeIngestion component

Wiring it together

Testing the pipeline

Other loading patterns

Publishing as reusable components

Git-based sharing

Pulumi Cloud Private Registry

Conclusion

How We Built Platybot: An AI-Powered Analytics Assistant

The problem with throwing AI at your Data Warehouse

Why we built a semantic layer first

Adding the AI layer

Meeting users where they are

The web UI

Slack

MCP: Making Platybot usable by Agents

Deploying with Pulumi

Results

What we learned

Resource Search - AI Assist is Generally Available

AI Assist is Generally Available

Suggested Queries and I’m Feeling Lucky

Wrapping Up

Pulumi Insights: Intelligence for Cloud Infrastructure

Intelligence - Productivity with the Power of AI

Search - Find Anything in Any Cloud

AI Assist for Search

Cloud Import

Analytics - Deeper Insights for Cloud Infrastructure

Conclusion

Data Science in the Cloud

How Pulumi Automation API Improves on Cloud SDKs

Infrastructure as a function

Data Science Your Way

AWS Serverless Analytics

ACME E-Commerce

The Traditional Analytics Stack

ACME Ads MVP

Scaling a Real Business

Designing the V2 Analytics Stack

Architecture as Code with Pulumi

Creating a Batch Table API

Automating the Dev Environment

Wrapping Up

Architectural Caveats

Create Secure Jupyter Notebooks on Kubernetes using Pulumi

Prerequisites

Step 1: Create a Pulumi Project and Stack with a Pulumi TypeScript Template

Step 2: Create a GKE Cluster

Step 3: Create a NGINX-Ingress-Controller to Generate Ingresses

Step 4: Create a Jupyter Notebook Deployment and Service with Type NodePort

Step 5: Create a Secret that is used with your Jupyter Notebook Domain Name