Pulumi Blog: Serverless

Self-Verifying AI Agents: Vercel's Agent-Browser in the Ralph Wiggum Loop

Engin Diri — Tue, 20 Jan 2026 00:00:00 +0000

In my previous post about the Ralph Wiggum technique, Claude Code built a complete serverless URL shortener on AWS. The setup included Playwright MCP for end-to-end testing. It worked. But I kept wondering if there was something better for AI-driven browser automation. Then Vercel released agent-browser, and I had to try it.

Why browser automation matters for AI coding

When an AI coding agent builds a frontend, someone has to verify it works. Without browser automation, that someone is you. The AI finishes and says “done,” but you can’t trust that claim until you open a browser and click around yourself.

Browser automation changes this. The AI verifies its own work. It builds a component, launches a browser, tests the interaction, confirms behavior matches expectations. If something breaks, it fixes the code and tests again. The validation loop runs without you.

This matters more as AI agents take on larger tasks. A quick component fix might not need automated testing. But when an agent builds an entire dashboard from scratch, you need proof that the deployed application actually works.

The context problem with browser automation

Playwright MCP gives you powerful browser control, but it comes with overhead. Every screenshot, every DOM snapshot, every accessibility tree adds tokens to your context window. GitHub issue #889 documents a 6x token increase between versions 0.0.30 and 0.0.32. Users report single screenshots consuming over 15,000 tokens. Some exhausted their entire five-hour token allocation in just a few automation steps.

The problem is verbose output. Full accessibility trees contain every element on the page with all their properties. Console message logging adds more. Each piece of information might be useful for debugging, but together they overwhelm the context window.

This creates a frustrating tradeoff. You want rich browser interaction for thorough testing, but that richness costs tokens. Fewer tokens means fewer iterations before hitting limits.

Vercel’s “less is more” philosophy

Vercel ran into something similar with their D0 text-to-SQL agent. Their research documents what happened when they reduced tool complexity.

The original architecture used 17 specialized tools. Each tool handled one specific operation: create tables, insert rows, run queries, validate schemas. The approach seemed logical. Specialized tools should produce better results than general ones.

The numbers told a different story. With 17 tools, they got 80% success (4 out of 5 queries), averaging 274.8 seconds and ~102,000 tokens. The worst case took 724 seconds, burned 145,463 tokens, and still failed.

Then they rebuilt with just two tools: ExecuteCommand and ExecuteSQL. Success jumped to 100%. Average execution dropped to 77.4 seconds (3.5x faster). Token usage fell to ~61,000 (37% reduction). The worst case took 141 seconds, used 67,483 tokens, and succeeded.

Vercel’s takeaway: “We were constraining reasoning because we didn’t trust the model to reason.” Fewer tools meant the model could think more freely about how to accomplish tasks. The simplicity reduced confusion and context waste.

Applying “less is more” to browser automation

agent-browser takes the same approach. Instead of separate tools for clicking, typing, scrolling, and navigating, it has a unified CLI with one clever idea: the snapshot + refs system.

When you request a page snapshot, agent-browser returns something like this:

- button "Sign In" [ref=e1]
- textbox "Email" [ref=e2]
- textbox "Password" [ref=e3]
- link "Documentation" [ref=e4]

Those @e1, @e2 references are stable element identifiers. To click the sign-in button, you run click @e1. No CSS selectors. No XPath expressions. No waiting for the DOM to stabilize. The reference points to that exact element.

This cuts out the verbosity of full accessibility trees. You get just enough information to understand the page structure and interact with elements. The AI can reason about what it sees without drowning in metadata.

Installing agent-browser

Setup is simple:

npm install -g agent-browser && agent-browser install

This installs the CLI and downloads a bundled browser. You don’t need to install any additional browsers.

For Claude Code integration, install the official skill from the agent-browser package. You can either copy it from node_modules:

cp -r node_modules/agent-browser/skills/agent-browser .claude/skills/

Or download it directly:

mkdir -p .claude/skills/agent-browser
curl -o .claude/skills/agent-browser/SKILL.md \
 https://raw.githubusercontent.com/vercel-labs/agent-browser/main/skills/agent-browser/SKILL.md

The skill gives Claude Code better context than just running agent-browser --help.

Unlike Playwright MCP, there’s no server configuration. Agent-browser runs as a standalone CLI that Claude Code invokes directly through bash.

The experiment: adding analytics

I wanted a realistic test scenario, so I added an analytics dashboard to the url-shortener-saas project from my previous post. The feature tracks click events with metadata (browser, device, country, referrer) and displays them through interactive charts.

This follows the same “Ralph Wiggum” workflow from my previous post: write a feature-prompt.md that describes what you want, then let Claude Code implement, deploy with Pulumi, and verify with browser automation. Here’s a snippet from the feature prompt:

## Backend Requirements

### New DynamoDB Table: Analytics Events

Create a new DynamoDB table `url-shortener-analytics-{stack}` with:
- Partition Key: shortCode (String)
- Sort Key: timestamp (String, ISO 8601 format)
- Attributes: browser, deviceType, country, referrer, etc.

### New Lambda: analytics.ts

GET /api/analytics/{shortCode}
- Returns timeline, browsers, devices, countries, referrers
- Query params: from, to, granularity

## Success Criteria

1. `pulumi up` deploys successfully without errors
2. All existing E2E tests still pass
3. New analytics E2E tests pass
4. Charts render with real data after generating test clicks

Claude Code reads this, writes the Lambda handlers, updates the Pulumi infrastructure code, and deploys. The infrastructure changes are straightforward:

// Analytics events table
const analyticsTable = new aws.dynamodb.Table("analytics-table", {
 name: `${projectName}-analytics-${stack}`,
 billingMode: "PAY_PER_REQUEST",
 hashKey: "shortCode",
 rangeKey: "timestamp",
 attributes: [
 { name: "shortCode", type: "S" },
 { name: "timestamp", type: "S" },
 ],
 globalSecondaryIndexes: [{
 name: "by-date",
 hashKey: "shortCode",
 rangeKey: "timestamp",
 projectionType: "ALL",
 }],
});

After pulumi up succeeds, the agent needs to verify the deployed feature works. That’s where browser automation comes in. The feature prompt specifies verification requirements:

## End-to-End Verification

After deployment, use the `/agent-browser` skill to verify:
1. Analytics link appears in navigation
2. Analytics page loads without errors
3. Charts render (timeline, browser, device, geographic)
4. URL-specific analytics work from dashboard
5. Click data appears after generating test clicks

The test suite covered six scenarios: homepage load, URL shortening, dashboard view, analytics navigation, analytics overview, and date filter functionality. Nothing fancy, just the basics you’d want to verify after deploying.

I ran the exact same tests with both Playwright MCP and agent-browser to see how much context each consumed.

Agent-browser workflow in practice

Here’s what the CLI interaction looked like during my test run.

Navigate to the deployed application:

$ agent-browser open https://d1232drths1aav.cloudfront.net
✓ Snip.ly - Modern URL Shortener
 https://d1232drths1aav.cloudfront.net/

Snapshot to see available elements (the -i flag filters to interactive elements only):

$ agent-browser snapshot -i
- link "S Snip.ly" [ref=e1]
- link "Home" [ref=e2]
- link "Dashboard" [ref=e3]
- link "Analytics" [ref=e4]
- link "Docs" [ref=e5]
- button "Switch to dark mode" [ref=e6]
- link "Get Started" [ref=e7]
- textbox "Paste your long URL here..." [ref=e8]
- button "Shorten URL" [ref=e9]

That entire homepage snapshot is 280 characters. Playwright MCP returned 8,247 characters for the same page.

Fill the URL input and click the shorten button:

$ agent-browser fill @e8 "https://example.com/agent-browser-e2e-test"
✓ Done

$ agent-browser click @e9
✓ Done

Each action confirmation is 6 characters. Playwright MCP returns the full page state after every click - 12,891 characters when I clicked the shorten button.

Navigate to analytics:

$ agent-browser click @e4
✓ Done

$ agent-browser wait --load networkidle && agent-browser snapshot -i
✓ Done
- link "S Snip.ly" [ref=e1]
- link "Home" [ref=e2]
- link "Dashboard" [ref=e3]
- link "Analytics" [ref=e4]
- button "Last 7 days" [ref=e8]
- button "Last 30 days" [ref=e9]
- button "Last 90 days" [ref=e10]
- link "1 analytics-e2e-test https://example.com/analytics-test-verification 5" [ref=e11]
- link "2 test-url https://www.example.com/this-is-a-very-long-url... 4" [ref=e12]

The analytics snapshot shows date filter buttons and top URLs with click counts. 385 characters versus Playwright MCP’s 4,127.

Test the date filter:

$ agent-browser click @e9
✓ Done

$ agent-browser screenshot analytics-30-day-filter.png
✓ Screenshot saved to analytics-30-day-filter.png

The ref-based workflow is deterministic. Each command operates on a specific element from the snapshot. No guessing about selectors.

The numbers: 82.5% context reduction

Same six tests, both tools:

Metric	Playwright MCP	Agent-Browser	Reduction
Total response characters	31,117	5,455	82.5%
Largest single response	12,891	2,847	77.9%
Average response size	3,112	328	89.5%
Homepage snapshot	8,247	280	96.6%
Dashboard snapshot	12,891	2,847	77.9%

The difference is what each tool returns after every action.

Playwright MCP returns the full accessibility tree after every click:

### Page state
- Page URL: https://d1232drths1aav.cloudfront.net/dashboard
- Page Title: Snip.ly - Modern URL Shortener
- Page Snapshot:
- generic [ref=e2]:
 - banner [ref=e3]:
 - generic [ref=e4]:
 - link "S Snip.ly" [ref=e5] [cursor=pointer]:
 - /url: /
 - generic [ref=e6]: S
 - generic [ref=e7]: Snip.ly
 - navigation [ref=e8]:
 - link "Home" [ref=e9] [cursor=pointer]:
 - /url: /
 - text: Home
 ...
[12,891 characters continues...]

Agent-browser returns:

✓ Done

6 characters versus 12,891 for the same button click.

Six tests consumed ~31K characters with Playwright MCP versus ~5.5K with agent-browser. At roughly 4 characters per token, that’s ~7,800 tokens versus ~1,400. An AI agent could run 5.7x more tests in the same context budget.

What I noticed

Past the numbers, working with agent-browser felt different in ways that are hard to quantify.

The compact snapshots changed the rhythm of iteration. A typical page fit in a few hundred tokens instead of thousands, so Claude Code could run more test cycles before hitting limits. I spent less time worrying about context and more time actually testing.

Element refs removed a frustration I’d had with Playwright. CSS selectors break when someone changes a class name or restructures the DOM. With agent-browser, a button is just @e9. It doesn’t care about styling. That predictability was a relief.

What worked well:

Snapshots stay small enough that you don’t think about them
Refs mean no selector debugging
The CLI just works with Claude Code’s bash tool
No MCP server to configure

Agent-browser is early. Documentation is thin, and I read the source more than once to figure out edge cases.

Where I hit friction:

Modals that appear after API calls needed manual wait logic
Playwright’s waiting mechanisms are more mature
Hidden or dynamically loaded elements sometimes didn’t show up without explicit waits

For my URL shortener tests, agent-browser used fewer tokens per cycle, and the ref-based approach was more predictable than selector-driven automation.

Playwright MCP still wins on depth. Network interception, multi-tab handling, PDF generation, better waiting logic—if you need those, you need Playwright. For complex browser automation, it’s the more capable tool.

Deploy what your agent builds

Pulumi provisions the infrastructure your coding agent builds and tests, so a single pulumi up takes each verified change to the cloud.

Get started

When to use each

Agent-browser fits long autonomous sessions where context budget matters, basic navigation and verification tasks, and setups where you want to skip MCP configuration. CLI-based skills also avoid the schema overhead—tool definitions for multi-tool MCPs eat tokens even when idle.

Playwright MCP fits when you need the advanced stuff: network interception, PDF generation, multi-tab workflows, or sophisticated synchronization. It’s also the right choice if you have existing Playwright test suites. If the full MCP schema feels heavy, Playwright skills offer a lighter wrapper.

Start with agent-browser for AI validation loops. Move to Playwright when you outgrow it.

How Ralph Wiggum Built a Serverless SaaS with Pulumi

Engin Diri — Wed, 14 Jan 2026 00:00:00 +0000

I was about to do something that felt either genius or completely reckless: hand over my AWS credentials to an AI and step away from my computer. The technique is called “Ralph Wiggum,” named after the Simpsons character who eats glue and says “I’m in danger” while everything burns around him. And honestly, that felt about right for what I was attempting.

The problem with AI coding assistants

If you have spent any time with AI coding assistants, you know the frustration. You are in the middle of a task. It is going great. Claude is writing beautiful infrastructure code, understanding your architecture, making progress… and then it says “I have completed your request.”

You stare at the screen. “No. No, you have not. You maybe did a third of what I asked.”

So you reprompt it. It does a little more. Then it stops again. You are babysitting. The whole point of using an AI assistant was to save time, but if you have to check on it every five minutes, you are not saving anything. You are just a very expensive supervisor.

This is the problem Geoffrey Huntley decided to solve.

What if we just do not let it stop?

Geoffrey Huntley, a developer based in rural Australia (who, according to internet lore, lives on a property with goats), had a deceptively simple idea: what if every time Claude Code finishes and tries to exit, we just feed it the prompt again?

He named the technique “Ralph Wiggum” because the character embodies a kind of childlike persistence. Ralph repeatedly fails, makes silly mistakes, yet stubbornly continues in an endless loop until he eventually succeeds. Sound familiar? That is exactly how AI coding agents work. They make mistakes. They try again. They iterate.

The philosophy behind Ralph is beautifully stated: “Better to fail predictably than succeed unpredictably.” Every failure is just data for the next iteration.

How the Ralph loop works

At its core, the Ralph Wiggum technique is almost embarrassingly simple. It is a bash loop:

while true; do
 cat PROMPT.md | claude --print --dangerously-skip-permissions
done

That is it. That is the whole thing. You write your instructions in a PROMPT.md file, and the loop keeps feeding it to Claude Code over and over.

But here is why it works: each time Claude starts up, it looks at the project. It sees the files that exist, the code that is already written, the git history. It is not starting from zero. It picks up where it left off, sees what still needs to be done, and keeps going.

The official Claude Code plugin formalizes this with some important additions:

Completion promises: You tell Claude to output a specific string like <promise>COMPLETE</promise> when it has genuinely finished
Max iterations: A safety limit so your API bill does not reach infinity
Stop hooks: The mechanism that intercepts exit attempts and re-injects the prompt

To install it:

/plugin install ralph-wiggum@ghuntley

For end-to-end testing of deployed infrastructure, you will also want the Playwright MCP server. This lets Claude interact with your deployed application in a real browser:

claude mcp add playwright npx @playwright/mcp@latest

Why Pulumi is perfect for Ralph

Infrastructure as code has something most application code does not: objective success criteria. Your Lambda either deploys or it does not. Your DynamoDB table either exists or it does not. Tests either pass or they fail.

This makes Pulumi an ideal candidate for autonomous AI development:

Clear completion signals: pulumi preview and pulumi up provide unambiguous feedback
Iterative feedback: Failed deployments tell Claude exactly what went wrong
Testable outputs: You can write integration tests that verify your infrastructure actually works
Registry access: With the Pulumi MCP server, Claude has real-time access to documentation and examples

The experiment

I decided to build a serverless URL shortener on AWS. Not because the world needs another URL shortener, but because it is the perfect test case: multiple AWS services, clear requirements, and objective success criteria. But I did not want just a basic demo. I wanted a full SaaS-like experience with a polished frontend.

Here is the PROMPT.md file I wrote:

# Build a Production-Ready URL Shortener SaaS on AWS

Using Pulumi TypeScript, create a complete URL shortener SaaS with a polished frontend experience.

## Infrastructure Requirements
- DynamoDB table for storing URL mappings (shortCode -> originalUrl, clickCount, createdAt)
- Lambda functions for: creating short URLs, redirecting to original URLs, getting stats
- API Gateway REST API exposing the Lambda functions
- S3 bucket for the React frontend with static website hosting
- CloudFront distribution with HTTPS for the frontend and API
- Use the Pulumi ESC environment pulumi-idp/auth for the AWS credentials

## Frontend Requirements (Use /frontend-design skill)
Create a polished, production-ready SaaS website with:

### Landing Page
- Hero section with catchy headline and URL input form
- Feature highlights (fast redirects, analytics, custom aliases)
- Pricing section (Free tier, Pro tier, Enterprise tier)
- Testimonials section with 3 fake but realistic customer reviews
- Footer with links to Docs, Privacy, Terms

### Dashboard Page
- URL shortening form with optional custom alias
- List of created short URLs with click counts
- Copy-to-clipboard functionality
- Delete URL option

### Documentation Page
- Getting started guide
- API reference (POST /shorten, GET /{code}, GET /stats/{code})
- Rate limits and usage policies
- FAQ section

### Design Requirements
- Modern, clean aesthetic (no generic AI look)
- Responsive design (mobile, tablet, desktop)
- Dark mode support
- Smooth animations and transitions
- Consistent color scheme and typography

## API Requirements
- POST /shorten: accepts { url: string, alias?: string }, returns { shortCode, shortUrl }
- GET /{shortCode}: redirects to original URL (301)
- GET /stats/{shortCode}: returns { originalUrl, clickCount, createdAt }

## Success Criteria
- All unit tests pass
- All integration tests pass
- pulumi preview shows no errors
- pulumi up deploys successfully

## End-to-End Verification (Required)
After deployment, use Playwright MCP to verify the live site:
1. Open the CloudFront URL and verify the landing page loads with all sections
1. Navigate to the docs page and verify content renders
1. Create a short URL using the dashboard
1. Verify the short URL redirects correctly (301 status)
1. Check the stats show the click was recorded
1. Test responsive design by resizing the viewport
1. Take screenshots of landing page, dashboard, and docs as proof

Only output <promise>COMPLETE</promise> after ALL of the above including E2E tests pass.

The key addition here is the end-to-end verification section. Without it, Claude reports success after pulumi up finishes, but you have no guarantee the deployed infrastructure actually works. By requiring Playwright to test the live URLs, you catch issues like the CloudFront 403 error I mentioned earlier.

Before running the loop, I started Claude Code with permission bypass mode to prevent it from stopping to ask for approval on every file write or command:

claude --permission-mode bypassPermissions

There are other ways to handle permissions, but this is the simplest for autonomous execution. Then I ran the Ralph loop:

/ralph-wiggum:ralph-loop PROMPT.md --max-iterations 25 --completion-promise "COMPLETE"

Then I stepped away and let it run unsupervised. For complex projects, you could even let it run overnight while you sleep.

Here is Claude Code in action during the Ralph loop, fixing CloudFront configuration issues and writing unit tests:

The results

The complete source code for this project is available on GitHub: url-shortener-saas

When Claude finished, it had:

Successfully built:

A complete Pulumi program with proper resource organization
DynamoDB table with a GSI for querying by creation date
Three Lambda functions with proper error handling
API Gateway with CORS configuration
S3 bucket with static website hosting
CloudFront distribution with proper cache behaviors

The frontend was genuinely impressive:

A polished landing page with hero section, feature cards, and pricing tiers
Working testimonials section with three fictional but believable customer reviews
A functional dashboard where you could create and manage short URLs
A documentation page with API reference and getting started guide
Dark mode toggle that actually worked
Responsive design that looked good on mobile

Home screen

Dashboard

Documentation

Here is a snippet of what it generated:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

const stack = pulumi.getStack();
const projectName = "url-shortener";

// DynamoDB Table
const urlTable = new aws.dynamodb.Table("url-table", {
 name: `${projectName}-urls-${stack}`,
 billingMode: "PAY_PER_REQUEST",
 hashKey: "shortCode",
 attributes: [{ name: "shortCode", type: "S" }],
 tags: {
 Environment: stack,
 Project: projectName,
 ManagedBy: "Pulumi",
 },
});

// Lambda function for URL shortening
const shortenFunction = new aws.lambda.Function("shorten-function", {
 name: `${projectName}-shorten-${stack}`,
 runtime: aws.lambda.Runtime.NodeJS20dX,
 handler: "shorten.handler",
 role: lambdaRole.arn,
 timeout: 30,
 memorySize: 256,
 code: new pulumi.asset.AssetArchive({
 ".": new pulumi.asset.FileArchive("./lambda"),
 }),
 environment: {
 variables: {
 TABLE_NAME: urlTable.name,
 },
 },
});

// CloudFront Distribution with S3 and API Gateway origins
const distribution = new aws.cloudfront.Distribution("frontend-distribution", {
 enabled: true,
 defaultRootObject: "index.html",
 origins: [
 {
 domainName: frontendBucketWebsite.websiteEndpoint,
 originId: "S3Origin",
 customOriginConfig: {
 httpPort: 80,
 httpsPort: 443,
 originProtocolPolicy: "http-only",
 originSslProtocols: ["TLSv1.2"],
 },
 },
 {
 domainName: pulumi.interpolate`${api.id}.execute-api.${aws.config.region}.amazonaws.com`,
 originId: "APIGatewayOrigin",
 customOriginConfig: {
 httpPort: 80,
 httpsPort: 443,
 originProtocolPolicy: "https-only",
 originSslProtocols: ["TLSv1.2"],
 },
 },
 ],
 defaultCacheBehavior: {
 targetOriginId: "S3Origin",
 viewerProtocolPolicy: "redirect-to-https",
 allowedMethods: ["GET", "HEAD", "OPTIONS"],
 cachedMethods: ["GET", "HEAD"],
 forwardedValues: {
 queryString: false,
 cookies: { forward: "none" },
 },
 },
 orderedCacheBehaviors: [{
 pathPattern: "/api/*",
 targetOriginId: "APIGatewayOrigin",
 viewerProtocolPolicy: "https-only",
 allowedMethods: ["DELETE", "GET", "HEAD", "OPTIONS", "PATCH", "POST", "PUT"],
 cachedMethods: ["GET", "HEAD"],
 forwardedValues: {
 queryString: true,
 headers: ["Authorization", "Content-Type"],
 cookies: { forward: "none" },
 },
 minTtl: 0,
 defaultTtl: 0,
 maxTtl: 0,
 }],
 viewerCertificate: {
 cloudfrontDefaultCertificate: true,
 },
});

Where it struggled:

The first few iterations had the Lambda code inline instead of in separate files
It initially forgot to set up IAM permissions for the Lambda to access DynamoDB
The CloudFront configuration took three attempts to get the cache behaviors right

The funny parts:

At iteration 12, it decided to completely refactor the project structure. Then at iteration 13, it refactored it back
One commit message simply read: “fix the fix that fixed the previous fix”
It wrote a test, ran the test, fixed the code to pass the test, then realized the test was wrong

Pair your coding agent with Pulumi

You can point Claude Code, Cursor, Codex, or any coding agent at Pulumi through the Pulumi MCP server and Agent Skills, and let infrastructure as code give it objective success criteria. Start a project to try it.

Get started

Best practices for Ralph with Pulumi

After running several experiments, here is what I have learned:

Write extremely specific prompts. Vague instructions lead to vague implementations. Be explicit about every requirement, every edge case, every success criterion.

Use pulumi preview as your feedback loop. Add this to your success criteria:

Success Criteria:
- pulumi preview completes with no errors
- pulumi preview shows expected resource count (approximately X resources)

Set realistic iteration limits. For infrastructure projects, I have found 20-30 iterations is usually sufficient. More than that and you are probably missing something in your prompt.

Include test requirements. Claude is surprisingly good at test-driven development when you tell it to write tests first:

## Development Approach
1. Write unit tests for each Lambda function first
1. Implement Lambda functions to pass the tests
1. Write integration tests that verify the deployed infrastructure
1. Only output COMPLETE when all tests pass

Mention cost awareness. Add a note like “Use PAY_PER_REQUEST billing for DynamoDB” or “Use the smallest Lambda memory allocation that works.” Claude will optimize for what you tell it to optimize for.

The honest assessment

Is this the future of infrastructure development? Probably not entirely. But it is a genuinely useful technique for specific scenarios:

Great for:

Greenfield projects where you have clear requirements
Large migrations or refactors
Building proof-of-concept infrastructure while you focus on other work
Tasks where iteration is more valuable than perfection on the first try

Not great for:

Production infrastructure changes (please do not let an AI modify your production AWS account unsupervised)
Complex architecture decisions that require human judgment
Anything involving sensitive data or security configurations
Small tweaks or quick fixes (overkill)

The API costs were reasonable for what I got. For a complete infrastructure project built autonomously, a few dollars in Claude API calls is a bargain compared to my hourly rate.

Try it yourself

If you want to experiment with Ralph Wiggum and Pulumi:

Install Claude Code and the Ralph Wiggum plugin
Set up your Pulumi account and AWS credentials
Write a detailed PROMPT.md with clear success criteria
Start with a small project and low iteration limits
Review the git history when it finishes

The combination of autonomous AI loops and infrastructure as code feels like a glimpse of where development is heading. Not replacing developers, but changing how we use our time. Instead of babysitting an AI through each step, you hand it a well-defined problem and come back to working infrastructure.

Just make sure you have billing alerts set up. Ralph Wiggum does not know when to stop spending your money.

Or try Pulumi Neo: Ralph built-in

If setting up the Ralph Wiggum plugin feels like too much ceremony, Pulumi has something similar built right into Pulumi Neo.

Neo Tasks work on the same principle: you describe what you want, and Neo plans and executes the infrastructure changes. The key is auto mode. When you set a task to auto mode, Neo runs without requesting approvals. It plans, executes, validates with pulumi preview, and iterates until the task is complete.

The experience is remarkably similar to Ralph:

Persistent execution: Tasks continue running even if you close your browser. Come back later and Neo shows you what it accomplished while you were away.
Iterative refinement: Failed deployments inform the next attempt, just like Ralph re-reading the project state.
Clear completion: Neo knows when the infrastructure matches your requirements.

The main difference is that Neo runs in Pulumi Cloud rather than on your local machine, and it is purpose-built for infrastructure tasks. You get the autonomous loop experience without needing to configure plugins or bash loops.

For teams already using Pulumi Cloud, Neo with auto mode might be the fastest path to autonomous infrastructure development.

Resources

Host your Python app for $1.12 a month

Adam Gordon Bell — Wed, 29 Jan 2025 00:00:00 +0000

TL;DR (2026 pricing): Packaging a Flask app as a container and running it on AWS Lambda behind an HTTP API Gateway costs ~$1.12/month worst-case, or $0 when idle. The breakdown: $0.04 for API Gateway requests, $0.07 for Lambda compute (fully absorbed by the always-free tier), and $1.08 for 12 GB of egress at $0.09/GB (covered by AWS’s 100 GB/month free egress allowance unless other services in the account have already used it). Pulumi handles the entire infrastructure. Verified against AWS pricing as of April 2026.

How cheap can you host a Python app in 2026? For a low-traffic Flask API (say, 40,000 requests per month at 512 MB of memory), the answer is roughly $1.12/month on AWS worst-case, dropping to $0 when idle. The trick is to stop thinking of AWS Lambda as “one function per endpoint” and instead package your entire web framework as a container, deploy it to Lambda, and put it behind an HTTP API Gateway. Your code stays standard Flask. Your bill stays in the loose-change zone.

This post walks through the whole setup with Pulumi, then compares the resulting cost against Google Cloud Run, Fly.io, Railway, and Vercel using current 2026 prices.

Most developers maintain at least one low-traffic service that still needs to be reliably available: an internal reporting API, a webhook receiver, a side project with occasional use. Traditional hosting means paying for 24/7 server time, even when the service sits idle. AWS Lambda’s container support flips that: pay only when your API is being called, and any web framework that handles HTTP requests (Flask, FastAPI, Express) works without modification.

How does the pricing break down? (2026)

Plugging 40,000 requests per month at 512 MB memory and ~200 ms average execution into 2026 AWS pricing:

Lambda compute: 40,000 × 0.5 GB × 0.2 s = 4,000 GB-seconds × $0.0000166667 = $0.067, fully covered by the always-free tier (400,000 GB-seconds free per month).
Lambda requests: 40,000 × $0.20 / 1,000,000 = $0.008, also covered by the free tier (1M requests free per month).
HTTP API Gateway: 40,000 × $1.00 / 1,000,000 = $0.04.
Data transfer out: 12 GB × $0.09/GB = $1.08 if billed, but AWS’s always-free tier (expanded to 100 GB/month outbound in late 2021) covers this row outright unless other services in the same account have already exhausted that allowance.
Total: ~$0.04/month (egress in the free tier, only API Gateway bills) to ~$1.12/month (12 GB egress billed in full: $1.08 + $0.04 API Gateway), with Lambda compute and requests always fully covered by the free tier.

The underlying math hasn’t changed since this post was first published in early 2025; Lambda and API Gateway pricing has been remarkably stable, and AWS has only expanded free-tier allowances since.

Cheap Python hosting compared (2026)

Provider	Approx. cost / month	2026 notes
AWS Lambda + HTTP API	~$1.12	~$1.08 for 12 GB egress at $0.09/GB + $0.04 for 40k API Gateway requests at $1/M; Lambda compute ($0.067) is fully covered by the always-free tier (1M req/400k GB-s).
Google Cloud Run	~$1.44	2M requests, 360,000 GiB-seconds of memory, and 180,000 vCPU-seconds are free every month — compute is $0. Egress dominates: Premium tier is ~~$0.12/GB, so 12 GB ≈ $1.44. Standard tier (~~$0.085/GB) lands near $1.02.
Fly.io	~$2.18	Cheapest persistent `shared-cpu-1x@256MB` machine is ~$1.94/mo; 12 GB outbound at $0.02/GB (NA/EU) adds ~$0.24 (free tier was retired in late 2024).
Railway	~$6.00	Hobby plan starts at $5/mo and includes $5 of usage; a 512 MB always-on container plus 12 GB egress lands around $5.50–6.50 total.
Vercel (Pro)	$20+	Pro plan has a $20/mo floor; 40k function invocations are well under the included quotas, so you mostly pay for the seat.

Approximate 2026 hosting costs for a 512 MB Python API at 40,000 requests/month with 12 GB egress. Egress dominates at this scale; even 1 million requests/month stays under $30 on Lambda.

What we’ll cover:

Why use AWS Lambda for low-traffic Python apps?

“But Lambda means one function per REST endpoint, right?” While that’s how serverless was initially positioned, the Lambda monolith approach here is a more straightforward pattern. Take any REST API service, package it as a container, put it in Lambda, and give it an entire HTTP route behind an IP, domain name, or API Gateway prefix. Done.

This approach gives you several benefits:

Minimal Serverless Knowledge Required: You don’t need to be an expert in Lambda architecture or serverless patterns. If you can build a REST API and containerize it, you’re ready to go.
Standard Development Experience: Write your API exactly as you would for any other hosting platform: no special frameworks, no Lambda-specific patterns to learn.
Simple Local Development: Run your container locally, and it will behave exactly like it will in production. No complex serverless emulation is needed.
Zero Cost at Zero Usage: Get all the cost benefits of serverless without restructuring your application.

The trade-off? Cold starts when Lambda spins up a new container instance. But this is often a worthwhile exchange for operational simplicity and zero-cost idle periods for low-traffic services.

Let me show you how to set this up. With some Pulumi Python code for setting up the Infra and some containerization code, you get an easy hosting solution.

Why containers?

Containers let us package everything—code, system dependencies, runtime—into one unit. Any HTTP service that fits in a container will work: Flask, Express, Rails, plus whatever system tools you need. Local testing? Just start the container. No Lambda-specific configuration is required.

How do I build a Flask app for Lambda?

Let’s start with the fun part - writing some Python code, but I will keep it really simple. We’ll build a simple Flask application that demonstrates why running a full web framework in Lambda makes sense.

Here’s our starting point in scripts/simple_server.py

from typing import Dict
from flask import Flask, jsonify, request
from mangum import Mangum

# Create Flask app
app = Flask(__name__)

@app.route("/hello")
def hello() -> Dict[str, str]:
 return jsonify({"message": "Hello from Flask!"})

@app.route("/echo", methods=['POST'])
def echo() -> Dict[str, str]:
 data = request.get_json()
 return jsonify({"reversed": data['message'][::-1]})

# Create Lambda handler
lambda_handler = Mangum(app)

if __name__ == "__main__":
 app.run(host="0.0.0.0", port=3000)

Notice a few key things about this code:

It’s Just Flask: No Lambda-specific code, no special frameworks. If you know Flask, you know how to modify this.
Multiple Endpoints: We have several routes demonstrating different HTTP methods and URL patterns.

How do I containerize a Flask app for Lambda?

To containerize our Flask app for Lambda, we’ll start with a Dockerfile that meets AWS’s requirements. The key is using the right base image and adapting our application to Lambda’s runtime interface.

First, let’s create our Dockerfile:

FROM public.ecr.aws/lambda/python:3.12

# Set up working directory
WORKDIR /app
COPY scripts/simple_server.py scripts/
COPY scripts/simple_entrypoint.sh scripts/

# Install dependencies
RUN pip install --no-cache-dir \
 flask>=3.0.0 \
 mangum>=0.17.0

# Make entrypoint script executable
RUN chmod +x scripts/simple_entrypoint.sh

# Set environment variables
ENV FLASK_APP=scripts/simple_server.py
ENV PYTHONPATH=/app
ENV RUN_MODE=lambda
EXPOSE 3000

# Use our entrypoint script
ENTRYPOINT ["/app/scripts/simple_entrypoint.sh"]

Notice that:

We’re using AWS’s official Lambda Python base image. You can use your own base image and your Python package manager of choice, but starting with this base and pip install is the easiest.
The ${LAMBDA_TASK_ROOT} is a special directory where Lambda expects to find your code
We’re introducing a new file: simple_entrypoint.sh

Do I need to learn Lambda-specific patterns?

The beauty of everything covered so far is that it’s just a container. You can work with it locally with Docker run, put it in Kubernetes, or run it in one of the 1000 places you can run containers. Lambda has some odd requirements about function calling that get in the way of this, but simple_entrypoint.sh saves the day:

#!/bin/bash

if [ "$RUN_MODE" = "local" ]; then
 echo "Starting in local mode..."
 exec python scripts/simple_server.py
else
 echo "Starting in Lambda mode..."
 exec python -m awslambdaric scripts.simple_server.lambda_handler
fi

All this means is that locally, you don’t need to know about lambdas at all. If you supply RUN_MODE=local it runs the standard way you are used to:

#!/bin/bash

# Build the container
docker build -t simple-flask -f infra/Dockerfile .
docker run -d -p 3000:3000 -e RUN_MODE=local simple-flask

# Test the endpoints:
# 1. Test GET /hello
curl http://localhost:3000/hello

# 2. Test POST /echo with a message
curl -X POST "http://localhost:3000/echo" \
 -H "Content-Type: application/json" \
 -d '{"message": "hello world"}'

This local testing setup is crucial because it helps catch issues before deployment.

The beauty of this setup is that once it works locally, it’ll work the same way in AWS.

And since this is just a standard Python application in a container, you can use all your favorite Python tools and patterns – your preferred package manager (pip, poetry, uv), pytest for testing, standard logging, type hints, linting, and pre-commit hooks.

( I’m skipping those details here to focus on deployment, but the container approach means there are no serverless-specific limitations on your Python workflow. )

How do I deploy a Lambda container with Pulumi?

Now let’s get our containerized Flask app into AWS. While you could click around in the AWS console, we’re going to do this properly with infrastructure as code. You are welcome to use Cloudformation or manual setup, but it’s nice to be able to keep everything in Python, and this is the Pulumi blog. Using Python to define our AWS resources means we get type-checking and proper IDE support.

Here’s our infrastructure code:

Don’t let the infrastructure code below intimidate you. It’s just typed Python code, hooking up the Lambda, that you can version control and modify easily.

from typing import Any
import pulumi
import pulumi_aws as aws
import pulumi_awsx as awsx
import json

# First, create an API Gateway
api_gateway: aws.apigatewayv2.Api = aws.apigatewayv2.Api("flask-api",
 name="Flask Lambda API",
 protocol_type="HTTP",
 route_selection_expression="$request.method $request.path"
)

# Create IAM role for Lambda
lambda_role: aws.iam.Role = aws.iam.Role("flask-lambda-role",
 assume_role_policy=json.dumps({
 "Version": "2012-10-17",
 "Statement": [{
 "Action": "sts:AssumeRole",
 "Principal": {
 "Service": "lambda.amazonaws.com"
 },
 "Effect": "Allow",
 "Sid": ""
 }]
 })
)

# Attach basic execution policy - Lambda needs this to write logs
lambda_role_policy: aws.iam.RolePolicyAttachment = aws.iam.RolePolicyAttachment("lambda-role-policy",
 role=lambda_role.name,
 policy_arn="arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
)

# Create ECR repository for our container
repository: aws.ecr.Repository = aws.ecr.Repository("flask-app-repo",
 name="flask-app-repo",
 force_delete=True, # Makes cleanup easier for testing
 image_scanning_configuration=aws.ecr.RepositoryImageScanningConfigurationArgs(
 scan_on_push=True,
 )
)

# Build and push the Docker image to ECR
image: awsx.ecr.Image = awsx.ecr.Image("flask-app-image",
 repository_url=repository.repository_url,
 path=".", # Path to your Dockerfile
 platform="linux/amd64" # Important for M1/M2 Mac users
)

# Create the Lambda function
lambda_function: aws.lambda_.Function = aws.lambda_.Function("flask-app",
 name="flask-app",
 package_type="Image",
 image_uri=image.image_uri,
 role=lambda_role.arn,
 timeout=30,
 memory_size=512,
 environment={
 "variables": {
 "FLASK_ENV": "production",
 "PYTHONUNBUFFERED": "1"
 }
 }
)

# Create API Gateway stage
stage: aws.apigatewayv2.Stage = aws.apigatewayv2.Stage("api-stage",
 api_id=api_gateway.id,
 name="$default",
 auto_deploy=True
)

# Connect API Gateway to Lambda
integration: aws.apigatewayv2.Integration = aws.apigatewayv2.Integration("lambda-integration",
 api_id=api_gateway.id,
 integration_type="AWS_PROXY",
 integration_uri=lambda_function.arn,
 integration_method="POST",
 payload_format_version="2.0"
)

# Create catch-all route
route: aws.apigatewayv2.Route = aws.apigatewayv2.Route("catch-all-route",
 api_id=api_gateway.id,
 route_key="ANY /{proxy+}",
 target=integration.id.apply(lambda id: f"integrations/{id}")
)

# Allow API Gateway to invoke Lambda
lambda_permission: aws.lambda_.Permission = aws.lambda_.Permission("api-lambda-permission",
 action="lambda:InvokeFunction",
 function=lambda_function.name,
 principal="apigateway.amazonaws.com",
 source_arn=api_gateway.execution_arn.apply(lambda arn: f"{arn}/*/*")
)

# Export the API Gateway URL
pulumi.export("url", api_gateway.api_endpoint)

Let’s break down what’s happening here:

API Gateway: We create an HTTP API (v2) that will receive all web requests and forward them to our Lambda.
IAM Role: Lambda needs permissions to execute and write logs. We create a role with the minimum necessary permissions.
Container Registry: We create an ECR repository and use Pulumi’s awsx package to build and push our Docker image automatically.
Lambda Function: We create the Lambda function using our container image. Notice how we set memory and timeout - these are important for a web application.
Integration: We connect API Gateway to Lambda using the AWS_PROXY integration type. ANY /{proxy+} means we route everything through with the full path.

Need a custom domain, that’s easy to use as well. Setup some Route53 records, CNAME a domain name appropriately and you’ll have a container backed by a custom domain name.

Why is this approach worth it?

The cool thing about using Pulumi and lambdas is that they make complex infrastructure changes safe and repeatable.

Want to increase the Lambda memory? Change one number and run pulumi up.
You get complete AWS Lambda and API Gateway management for your low-traffic REST APIs
And Infrastructure testing capabilities mean you can validate your serverless configuration before deployment

You can find the full code in the service status monitor repo from my Python devops article. A larger version with rolling updates is coming soon.

Alternative approaches

There are other ways to run containers that scale to zero. Google Cloud Run offers similar functionality on GCP, and AWS App Runner is another AWS service that can do this. Both have similar pricing models—very cheap for low-volume services. And SST is a similar solution for getting TypeScript / JavaScript solutions into an AWS Lambda.

The beauty of the container approach used here is that switching between these services is straightforward. Since we’re just running a standard container with a REST endpoint, there’s nothing AWS-specific in our application code. If you need to move to a different platform later, you can take your container with you.

Conclusion

Running Flask applications in Lambda containers represents a sweet spot between development simplicity and operational efficiency. You get:

True zero-cost periods when your application isn’t being used
Familiar development patterns - it’s just Flask
Infrastructure as code with Pulumi
Automatic scaling without managing servers

The trade-offs - primarily cold starts and some AWS-specific configuration - are worth it for many applications, especially:

APIs with sporadic traffic
Cost-sensitive projects

The key is understanding that this isn’t just about saving money - it’s about reducing operational complexity. No more worrying about scaling policies, server maintenance, or capacity planning. Your application scales automatically, and you only pay for what you use.

Is this approach right for every application? No, but this combination of familiar local development with serverless operations is under utilized.

I’d love to hear about your experiences with monolithic serverless applications. What hosting solutions have you tried? What challenges have you encountered?

Get started with Pulumi on AWS: the official quickstart for the tooling used in this post.
AWS Lambda function reference: every property on the resource we used above.
Code, deploy, and manage a serverless REST API on AWS with Pulumi: the function-per-route alternative to the monolith pattern.
Controlling AWS costs with Lambda and Pulumi: patterns for keeping your bill predictable as traffic grows.
The cursed container iceberg: every other place you can run the same container if you outgrow Lambda.
Python for DevOps: the companion post that ships the full source code for this example.

Changelog

2026-04-30: Refreshed for 2026. Re-verified Lambda/API Gateway/egress math, replaced the comparison table with current Cloud Run, Fly.io, Railway, and Vercel pricing, restructured H2s as user questions, added an answer-first intro and TL;DR, added a worked cost example and a HowTo schema for the deployment steps.
2025-03-13: Minor copy edits.
2025-01-29: Original publication.

Pinecone Provider Now Available for Pulumi

Engin Diri — Tue, 16 Jan 2024 00:00:00 +0000

Hello, Pulumi Pinecone Provider! 👋

The Pinecone integration with Pulumi offers a native way to manage Pinecone indexes, including the newly-announced serverless indexes. Utilize any of Pulumi’s supported languages to effortlessly create, update, and remove your Pinecone indexes. This integration facilitates the application of Infrastructure as Code principles, helping you to work even more efficiently. Furthermore, this gives you the benefit of tapping into Pulumi’s wide range of providers, offering you a diverse and powerful set of tools to enhance your development work.

What is Pinecone? 🧐

Pinecone is a fully managed vector database with an easy-to-use API that allows you to build and deploy high-performance AI applications. Applications involving large language models, generative AI, and semantic search require a vector database to store and retrieve vector embeddings. Vector embeddings enable AI applications to gain a deeper understanding of the data and maintain a long-term memory to draw upon.

The backbone of Pinecone is the vector index. A vector index allows you to store and retrieve vector embeddings. What makes Pinecone indexes so powerful is their low latency for billions of vectors.

Pinecone is now extending its lead as the technology and market leader by completely re-inventing the vector database. This groundbreaking serverless vector database lets companies add unlimited knowledge to their GenAI apps at 10-100x lower cost to enable developers to deliver remarkably better, commercially viable, differentiated AI applications.

A key advantage of this true-serverless experience is its cost-effectiveness, achieved through a “pay for what you use” pricing strategy.

Getting Started 🤖

Using the Pinecone Provider for Pulumi is as easy as it can be. You can install the Pinecone Provider following the documentation for the Pinecone Provider in the Pulumi Registry or by using dedicated starter templates.

The starter templates are specially designed for you to bootstrap a Pulumi program with a Pinecone index. Once you have your bootstrap program, you can use this as a starting point to create your own Pinecone Serverless indexes.

Deploying a Pinecone Serverless Index

pulumi new pinecone-<language>

Will create a new Pulumi project with a Pinecone serverless index. Let’s take a look at the Typescript example:

pulumi new pinecone-typescript

This will create a new Pulumi project and all the necessary files to deploy a Pinecone index. The index.ts file contains the following code:

import * as pulumi from "@pulumi/pulumi";
import * as pinecone from "@pinecone-database/pulumi";

const myPineconeIndex = new pinecone.PineconeIndex("my-pinecone-index", {
 name: "my-pinecone-index",
 metric: pinecone.IndexMetric.Cosine,
 spec: {
 serverless: {
 cloud: pinecone.ServerlessSpecCloud.Aws,
 region: "us-west-2",
 },
 },
});
export const output = {
 value: myPineconeIndex.host,
};

In this code, we define a new Pinecone Serverless index with the name my-pinecone-index and we use cosine as the distance metric and define the index to be serverless and to be deployed in the us-west-2 region of AWS.

To deploy this example, you need to set your Pinecone API key as an environment variable:

export PINECONE_API_KEY="xxxxx-xxxx-xxxx-xxxx-xxxxx"

Then you can run the following Pulumi command to deploy the example:

pulumi up

You will get a preview of the changes that will be applied to your Pinecone account. If you are happy with the changes, you can confirm them by typing yes. This will create the Pinecone index and output the host name of the index to use in your application.

pulumi up
Previewing update (dev)

 Type Name Plan
 + pulumi:pulumi:Stack pinecone-ts-dev create
 + └─ pinecone:index:PineconeIndex my-pinecone-index create

Outputs:
 output: {
 value: output<string>
 }

Resources:
 + 2 to create

Do you want to perform this update? yes
Updating (dev)

 Type Name Status
 + pulumi:pulumi:Stack pinecone-ts-dev created (6s)
 + └─ pinecone:index:PineconeIndex my-pinecone-index created (3s)

Outputs:
 output: {
 value: "my-pinecone-index-t2khngi.svc.apw5-4e34-81fa.pinecone.io"
 }

Resources:
 + 2 created

Duration: 8s

Have a look into your Pinecone console, and you should see the newly created serverless index there, ready to use.

Deleting a Pinecone Index

Deleting a Pinecone index is as simple as creating one. You can delete a Pinecone Serverless index by simply running following command:

pulumi destroy

This will delete the Pinecone index for you.

Be aware that this will also delete all the data stored in the index.

Programmatically Creating a Pulumi Pinecone Provider

By default, Pulumi will use the default provider configuration for the Pinecone provider. However, there may be situations where you need to use a different configuration. In that case, programmatically creating an explicit Pinecone provider with its own configuration is how to meet that requirement. You might also need to create one or more explicit Pinecone providers if you needed to deploy indexes in multiple Pinecone accounts, for example.

Here is an example on how to programmatically create a Pinecone provider and then pass the provider when creating an index.

const myPineconeProvider = new pinecone.Provider("my-pinecone-provider", {
 APIKey: "my-pinecone-api-key",
});

const myPineconeIndex = new pinecone.PineconeIndex("my-pinecone-index", {
 name: "my-pinecone-index",
 metric: pinecone.IndexMetric.Cosine,
 spec: {
 serverless: {
 cloud: pinecone.ServerlessSpecCloud.Aws,
 region: "us-west-2",
 },
 },
}, {
 provider: myPineconeProvider,
});

Conclusion 💡

As shown in this blog post, the Pinecone Provider for Pulumi allows you to seamlessly integrate and manage your Pinecone indexes—including the new serverless indexes—using any Pulumi-supported language. This enables platform engineers and developers to maintain easily manageable and reproducible infrastructure as code for their AI applications that use Pinecone indexes.

As always, we welcome your feedback and contributions in the Pulumi Community Slack, GitHub repository, and Pulumi Community Discussions.

New to Pulumi? Signing up is easy and free. Get started today!

Happy AI building!

Serverless Arch Templates

Kat Cosgrove — Mon, 07 Nov 2022 00:00:00 +0000

Whether you’re building a new application or moving an existing application over from another provider, the basic framework of your infrastructure probably isn’t something you want to worry about if you don’t have to. The cloud is complicated enough as it is. With Architecture Templates, Pulumi takes on some of the work involved in deploying your application to the cloud provider of your choice. Let’s take a tour of the new Serverless Templates for AWS, GCP, and Azure!

Serverless on any Cloud, in any Language

If you need a serverless application, whether that’s AWS Lambda, Google Cloud Functions, or Azure Functions, one of these templates can give you the starting point you need in any language Pulumi supports. Each template deploys a serverless function for you, with all necessary permissions to make it publicly available. While each of them does deploy a small sample function, these are designed to be extensible; we give you all the necessary tools to begin building your own application, without having to fuss around standing up the infrastructure yourself.

Generating the necessary starter code happens with a single CLI command, regardless of which cloud and language you choose:

AWS

$ mkdir my-serverless-app && cd my-serverless-app
$ pulumi new serverless-aws-typescript

$ mkdir my-serverless-app && cd my-serverless-app
$ pulumi new serverless-aws-python

$ mkdir my-serverless-app && cd my-serverless-app
$ pulumi new serverless-aws-go

$ mkdir my-serverless-app && cd my-serverless-app
$ pulumi new serverless-aws-csharp

$ mkdir my-serverless-app && cd my-serverless-app
$ pulumi new serverless-aws-yaml

GCP

$ mkdir my-serverless-app && cd my-serverless-app
$ pulumi new serverless-gcp-typescript

$ mkdir my-serverless-app && cd my-serverless-app
$ pulumi new serverless-gcp-python

$ mkdir my-serverless-app && cd my-serverless-app
$ pulumi new serverless-gcp-go

$ mkdir my-serverless-app && cd my-serverless-app
$ pulumi new serverless-gcp-csharp

$ mkdir my-serverless-app && cd my-serverless-app
$ pulumi new serverless-gcp-yaml

Azure

$ mkdir my-serverless-app && cd my-serverless-app
$ pulumi new serverless-azure-typescript

$ mkdir my-serverless-app && cd my-serverless-app
$ pulumi new serverless-azure-python

$ mkdir my-serverless-app && cd my-serverless-app
$ pulumi new serverless-azure-go

$ mkdir my-serverless-app && cd my-serverless-app
$ pulumi new serverless-azure-csharp

$ mkdir my-serverless-app && cd my-serverless-app
$ pulumi new serverless-azure-yaml

Pulumi to the Cloud

Each of these commands will generate some boilerplate for you, and includes defaults you can accept or modify (such as the name of the project, or the application directory). Run pulumi up from this directory, ensuring that you’ve connected the Pulumi CLI to your preferred cloud provider, and your starting point is online, ready for you to sub in your own application code. For the purposes of this demonstration, I’ve accepted the defaults. I’ve also left the provided application and function code in place. Let’s examine what Pulumi creates for us here, using Python for AWS, Go for GCP, and Typescript for Azure.

import json
import pulumi
import pulumi_aws as aws
import pulumi_aws_apigateway as apigateway

# An execution role to use for the Lambda function
role = aws.iam.Role("role",
 assume_role_policy=json.dumps({
 "Version": "2012-10-17",
 "Statement": [{
 "Action": "sts:AssumeRole",
 "Effect": "Allow",
 "Principal": {
 "Service": "lambda.amazonaws.com",
 },
 }],
 }),
 managed_policy_arns=[aws.iam.ManagedPolicy.AWS_LAMBDA_BASIC_EXECUTION_ROLE])

# A Lambda function to invoke
fn = aws.lambda_.Function("fn",
 runtime="python3.9",
 handler="handler.handler",
 role=role.arn,
 code=pulumi.FileArchive("./function"))

# A REST API to route requests to HTML content and the Lambda function
api = apigateway.RestAPI("api",
 routes=[
 apigateway.RouteArgs(path="/", local_path="www"),
 apigateway.RouteArgs(path="/date", method=apigateway.Method.GET, event_handler=fn)
 ])

# The URL at which the REST API will be served.
pulumi.export("url", api.url)

Here Pulumi has provided an IAM execution role to use for the Lambda function, the Lambda function itself, and a REST API gateway to handle request routing. The Lambda function is pointed at an included ./function directory, which contains a Python function that returns the latest datetime. This can be replaced with your own code, or you can change the directory the Pulumi code is pointed at to your own. Similarly, the API gateway is pointed at a www directory containing a simple website to display the current datetime, ready to be replaced with your own content.

package main

import (
 "fmt"

 "github.com/pulumi/pulumi-gcp/sdk/v6/go/gcp/cloudfunctions"
 "github.com/pulumi/pulumi-gcp/sdk/v6/go/gcp/storage"
 synced "github.com/pulumi/pulumi-synced-folder/sdk/go/synced-folder"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi/config"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {

 // Import the program's configuration settings.
 cfg := config.New(ctx, "")
 sitePath := "./www"
 if param := cfg.Get("sitePath"); param != "" {
 sitePath = param
 }
 appPath := "./app"
 if param := cfg.Get("appPath"); param != "" {
 appPath = param
 }
 indexDocument := "index.html"
 if param := cfg.Get("indexDocument"); param != "" {
 indexDocument = param
 }
 errorDocument := "error.html"
 if param := cfg.Get("errorDocument"); param != "" {
 errorDocument = param
 }

 // Create a storage bucket and configure it as a website.
 siteBucket, err := storage.NewBucket(ctx, "site-bucket", &storage.BucketArgs{
 Location: pulumi.String("US"),
 Website: &storage.BucketWebsiteArgs{
 MainPageSuffix: pulumi.String(indexDocument),
 NotFoundPage: pulumi.String(errorDocument),
 },
 })
 if err != nil {
 return err
 }

 // Create an IAM binding to allow public read access to the bucket.
 _, err = storage.NewBucketIAMBinding(ctx, "site-bucket-iam-binding", &storage.BucketIAMBindingArgs{
 Bucket: siteBucket.Name,
 Role: pulumi.String("roles/storage.objectViewer"),
 Members: pulumi.StringArray{
 pulumi.String("allUsers"),
 },
 })
 if err != nil {
 return err
 }

 // Use a synced folder to manage the files of the website.
 _, err = synced.NewGoogleCloudFolder(ctx, "synced-folder", &synced.GoogleCloudFolderArgs{
 Path: pulumi.String(sitePath),
 BucketName: siteBucket.Name,
 })
 if err != nil {
 return err
 }

 // Create another storage bucket for the serverless app.
 appBucket, err := storage.NewBucket(ctx, "app-bucket", &storage.BucketArgs{
 Location: pulumi.String("US"),
 })
 if err != nil {
 return err
 }

 // Upload the serverless app to the storage bucket.
 appArchive, err := storage.NewBucketObject(ctx, "app-archive", &storage.BucketObjectArgs{
 Bucket: appBucket.Name,
 Source: pulumi.NewFileArchive(appPath),
 })
 if err != nil {
 return err
 }

 // Create a Cloud Function that returns some data.
 dataFunction, err := cloudfunctions.NewFunction(ctx, "data-function", &cloudfunctions.FunctionArgs{
 SourceArchiveBucket: appBucket.Name,
 SourceArchiveObject: appArchive.Name,
 Runtime: pulumi.String("go116"),
 EntryPoint: pulumi.String("Data"),
 TriggerHttp: pulumi.Bool(true),
 })
 if err != nil {
 return err
 }

 // Create an IAM member to invoke the function.
 _, err = cloudfunctions.NewFunctionIamMember(ctx, "data-function-invoker", &cloudfunctions.FunctionIamMemberArgs{
 Project: dataFunction.Project,
 Region: dataFunction.Region,
 CloudFunction: dataFunction.Name,
 Role: pulumi.String("roles/cloudfunctions.invoker"),
 Member: pulumi.String("allUsers"),
 })
 if err != nil {
 return err
 }

 // Create a JSON configuration file for the website.
 _, err = storage.NewBucketObject(ctx, "site-config", &storage.BucketObjectArgs{
 Name: pulumi.String("config.json"),
 Bucket: siteBucket.Name,
 ContentType: pulumi.String("application/json"),
 Source: dataFunction.HttpsTriggerUrl.ApplyT(func(url string) pulumi.AssetOrArchiveOutput {
 config := fmt.Sprintf(`{ "api": "%s" }`, url)
 return pulumi.NewStringAsset(config).ToAssetOrArchiveOutput()
 }).(pulumi.AssetOrArchiveOutput),
 })
 if err != nil {
 return err
 }

 // Export the URLs of the website and serverless endpoint.
 ctx.Export("siteURL", pulumi.Sprintf("https://storage.googleapis.com/%s/index.html", siteBucket.Name))
 ctx.Export("apiURL", dataFunction.HttpsTriggerUrl)

 return nil
 })
}

Here Pulumi is defining your application’s initial configuration values, including the directories it should be checking for your serverless application code and the index and error pages for the website it stands up. It then creates two storage buckets, each with appropriate IAM roles for access, one containing your serverless function and another configured to act like a website. We also have the Cloud Function declaration itself, and all configurations necessary for it, followed by exporting the URL for you.

import * as pulumi from "@pulumi/pulumi";
import * as azure from "@pulumi/azure-native";
import * as synced from "@pulumi/synced-folder";

// Import the program's configuration settings.
const config = new pulumi.Config();
const sitePath = config.get("sitePath") || "./www";
const appPath = config.get("appPath") || "./app";
const indexDocument = config.get("indexDocument") || "index.html";
const errorDocument = config.get("errorDocument") || "error.html";

// Create a resource group for the website.
const resourceGroup = new azure.resources.ResourceGroup("resource-group", {});

// Create a blob storage account.
const account = new azure.storage.StorageAccount("account", {
 resourceGroupName: resourceGroup.name,
 kind: azure.storage.Kind.StorageV2,
 sku: {
 name: azure.storage.SkuName.Standard_LRS,
 },
});

// Create a storage container for the pages of the website.
const website = new azure.storage.StorageAccountStaticWebsite("website", {
 accountName: account.name,
 resourceGroupName: resourceGroup.name,
 indexDocument: indexDocument,
 error404Document: errorDocument,
});

// Use a synced folder to manage the files of the website.
const syncedFolder = new synced.AzureBlobFolder("synced-folder", {
 path: sitePath,
 resourceGroupName: resourceGroup.name,
 storageAccountName: account.name,
 containerName: website.containerName,
});

// Create a storage container for the serverless app.
const appContainer = new azure.storage.BlobContainer("app-container", {
 accountName: account.name,
 resourceGroupName: resourceGroup.name,
 publicAccess: azure.storage.PublicAccess.None,
});

// Upload the serverless app to the storage container.
const appBlob = new azure.storage.Blob("app-blob", {
 accountName: account.name,
 resourceGroupName: resourceGroup.name,
 containerName: appContainer.name,
 source: new pulumi.asset.FileArchive(appPath),
});

// Create a shared access signature to give the Function App access to the code.
const signature = azure.storage.listStorageAccountServiceSASOutput({
 resourceGroupName: resourceGroup.name,
 accountName: account.name,
 protocols: azure.storage.HttpProtocol.Https,
 sharedAccessStartTime: "2022-01-01",
 sharedAccessExpiryTime: "2030-01-01",
 resource: azure.storage.SignedResource.C,
 permissions: azure.storage.Permissions.R,
 contentType: "application/json",
 cacheControl: "max-age=5",
 contentDisposition: "inline",
 contentEncoding: "deflate",
 canonicalizedResource: pulumi.interpolate`/blob/${account.name}/${appContainer.name}`,
});

// Create an App Service plan for the Function App.
const plan = new azure.web.AppServicePlan("plan", {
 resourceGroupName: resourceGroup.name,
 sku: {
 name: "Y1",
 tier: "Dynamic",
 },
});

// Create the Function App.
const functionApp = new azure.web.WebApp("function-app", {
 resourceGroupName: resourceGroup.name,
 serverFarmId: plan.id,
 kind: "FunctionApp",
 siteConfig: {
 appSettings: [
 {
 name: "FUNCTIONS_WORKER_RUNTIME",
 value: "node",
 },
 {
 name: "WEBSITE_NODE_DEFAULT_VERSION",
 value: "~14",
 },
 {
 name: "FUNCTIONS_EXTENSION_VERSION",
 value: "~3",
 },
 {
 name: "WEBSITE_RUN_FROM_PACKAGE",
 value: pulumi.all([account.name, appContainer.name, appBlob.name, signature])
 .apply(([accountName, containerName, blobName, signature]) => `https://${accountName}.blob.core.windows.net/${containerName}/${blobName}?${signature.serviceSasToken}`),
 },
 ],
 cors: {
 allowedOrigins: [
 "*"
 ],
 },
 },
});

// Create a JSON configuration file for the website.
const configFile = new azure.storage.Blob("config.json", {
 source: functionApp.defaultHostName
 .apply(host => new pulumi.asset.StringAsset(JSON.stringify({ api: `https://${host}/api` }))),
 contentType: "application/json",
 accountName: account.name,
 resourceGroupName: resourceGroup.name,
 containerName: website.containerName,
});

// Export the URLs of the website and serverless endpoint.
export const siteURL = account.primaryEndpoints.apply(primaryEndpoints => primaryEndpoints.web);
export const apiURL = pulumi.interpolate`https://${functionApp.defaultHostName}/api`;

Here, Pulumi first defines the configuration options for your project, including the directories to look at for your application code and your website, as well as what the index and error documents should be. It then goes on to set up your blob storage account and containers for your website and serverless function, before uploading the serverless function itself with all permissions and configurations necessary to operate.

Extending Templates

In addition to the Pulumi code above, each template includes a simple website that utilizes a provided serverless function to display the current time and date, constantly updating. With only two CLI commands, you have deployed the core infrastructure necessary to get your serverless application online in the cloud, without writing a single line of code yourself.

From here, you’re ready to drop in your own application code. Pulumi has already handled the boring work for you, complying with best practices from the beginning so you only have to worry about building your own application, not how to get the infrastructure online!

Pulumi Architecture Templates exist for a variety of applications and common use-cases, including containerized applications and kubernetes, with more being added over time. Keep an eye out for new additions and ways to make engineering for the cloud less painful!

Accessing Secrets Safely in Lambda Functions

Piers Karsenbarg — Wed, 12 Oct 2022 17:19:53 +0100

The subject of how to make use of secrets in Lambda Functions comes up a fair bit, and although there seems to be a lot of discussion on where you should store them, the one thing that comes up is that you should never store the plain text values of secrets in the Lambda Function’s environment variables. One such discussion I was having with a customer made me think about how it should be possible to take the secrets that you’ve got on your stack config file and then use them to configure your Lambda Function, with the plain text values going into the Function’s environment variables and the encrypted secret values going into AWS’ Secrets Manager.

Getting started

I’m going to assume you’ve already got Pulumi and AWS set up, you’re logged into your backend, and you’ve already created a project and stack. I’m also using the Pulumi Service as my secrets provider.

Although Lambda Functions are free to deploy, and you get a generous allowance as part of AWS’ free tier, storing and accessing secrets is not. You can view the cost on the AWS Secrets Manager Pricing page.

Structured configuration

We’re going to start by adding settings, both secret and plaintext, to our stack configuration, in a structured way. We’re going to add these settings as a dictionary so we can use the names as the variable names in our Lambda Function.

First we add two settings in plaintext that are going to be used as environment variables in our lambda:

pulumi config set --path 'lambdawithsecrets.envvars["envvar1"]' envvar1value
pulumi config set --path 'lambdawithsecrets.envvars["envvar2"]' envvar2value

and then we add the secret settings:

pulumi config set --path 'lambdawithsecrets.secrets["secret1"]' secretvalue1 --secret
pulumi config set --path 'lambdawithsecrets.secrets["secret2"]' secretvalue2 --secret

If we take a look at the stack config (in my stack named dev it’s the Pulumi.dev.yaml file):

config:
 lambda-secrets:lambdawithsecrets:
 envvars:
 envvar1: envvar1value
 envvar2: envvar2value
 secrets:
 secret1:
 secure: AAABALZIAzDaE45MbdO08w/QMHK6CE4hssidMK70pyYt6hIFHpKeq0mBNic=
 secret2:
 secure: AAABAI+y9HpdgQ1u1rSPKGZeBkAIIbZfki84zmRrcWVltIMI5SGWZbYOyfQ=

there are several things I want to point out:

The entire config item is named lambdawithsecrets (this is how we’ll retrieve the configuration later)
There are two sections, envvars and secrets. These are how I’ll refer to the dictionaries later on in the code, but you can name these in any way you want.
We’ll use the dictionary keys (envvar1, envvar2, secret1, secret2) to refer to the variables in the Lambda Function so we know what they’re named and can easily access them. Again, you can name these to be whatever values you want.

Writing our Pulumi program

Next we’re going to start updating our index.ts file. Everything from now on can be copied and pasted in order into the file, and I’ll post the entire contents at the end.

As we’re only going to use the Pulumi and the AWS NPM packages you can replace everything in the file with the following:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

A warning about accessing the configuration

Before we continue, it’s worth pointing out the following (from the Pulumi documentation on secrets ):

On pulumi up, secret values are decrypted and made available in plaintext at runtime. These may be read through any of the standard pulumi.Config getters. While it is possible to read a secret using ordinary non-secret getters, this is almost certainly not what you want. Use the secret variants of the configuration APIs instead since this ensures that all transitive uses of that secret are themselves also marked as secrets.

Now that’s out the way, let’s continue.

Accessing the configuration

Since we’re using structured configuration to store the values, we can access them in a strongly-typed way. We’re going to create an interface to hold the structure:

interface LambdaConfig {
 envvars: string[];
 secrets: pulumi.Output<string>[];
}

You’ll notice that the property names match the dictionary names in the stack config and that the secrets property is of type pulumi.Output<string>[] instead of string[]. This is because they’re Pulumi secrets, and we want to handle them in the correct way.

Next we’ll access the config and store the values in a variable:

const config = new pulumi.Config();
const lambdaconfig = config.requireObject<LambdaConfig>("lambdawithsecrets");

Storing the secret values

As I said in the beginning, we’re going to store the secrets from the stack config in AWS Secrets Manager, and then the ARN of each of the secrets will be stored in the Lambda Function’s environment variables.

We’re going to create a couple of arrays to store (non-sensitive) information about the Secret resources that we’re going to create to use later on. One to hold the Secret resources and one to hold the ARNs of the secrets. Due to the fact that it’s the SecretVersion where we store the secret value, it’s pretty safe to store the Secret.

Next we’ll loop through the config’s secrets dictionary, grab the values and for each one create a Secret and a SecretValue resource, and then populate the two arrays with the relevant values.

const secretsArnArray: pulumi.Output<string>[] = new Array();
const secretArray: aws.secretsmanager.Secret[] = new Array();
for (const key in lambdaconfig.secrets) {
 if (lambdaconfig.secrets.hasOwnProperty(key)) {
 const secret = new aws.secretsmanager.Secret(`${key}`);
 const secretVersion = new aws.secretsmanager.SecretVersion(`secretversion-${key}`, {
 secretId: secret.id,
 secretString: lambdaconfig.secrets[key],
 });
 secretArray.push(secret);
 secretsArnArray[key.toLocaleUpperCase()] = secret.id;
 }
}

We’re not actually doing anything with the SecretVersion resources, so that’s why we’re not assigning them to any variables.

Identity and Access Management

There are a few resources we’re going to need to create and assign to the Lambda Function to give it access to the Secret resources:

A role that we can assign to the Lambda Function
A Policy that is going to give that role read access to the values of the secrets we’ve created
A couple of RolePolicyAttachment resources which is how we attach policies to the above role

Firstly, the role:

const role = new aws.iam.Role("roleLambdaWithSecrets", {
 assumeRolePolicy: aws.iam.assumeRolePolicyForPrincipal(aws.iam.Principals.LambdaPrincipal),
});

We’re using the assumeRolePolicyForPrincipal function because doesn’t that look better than this:

const role = new aws.iam.Role("roleLambdaWithSecrets", {
 assumeRolePolicy: JSON.stringify({
 Version: "2012-10-17",
 Statement: [{
 Action: "sts:AssumeRole",
 Effect: "Allow",
 Sid: "",
 Principal: {
 Service: "lambda.amazonaws.com",
 },
 }],
 }),
});

(either of the above Role resources can be used, but I’m using the top one because I think it’s neater).

Next we create a policy document and a policy. We’re going to use the secretArray variable that we created earlier to allow us to only allow the Lambda Function access to the secrets that we’ve created:

const secretManagerPolicyDoc = aws.iam.getPolicyDocumentOutput({
 statements: [
 {
 effect: "Allow",
 actions: [
 "secretsmanager:GetSecretValue",
 ],
 resources: secretArray.map(x => pulumi.interpolate`${x.arn}`),
 },
 ],
});

const secretManagerPolicy = new aws.iam.Policy("secretsPolicy", {
 policy: secretManagerPolicyDoc.apply(doc => doc.json),
});

Finally we’re going to attach the above policy and the AWS Managed AWSLambdaBasicExecutionRole policy to the IAM role we created:

const rpaBasic = new aws.iam.RolePolicyAttachment("rpa-basic", {
 role: role,
 policyArn: aws.iam.ManagedPolicy.AWSLambdaBasicExecutionRole,
});

const rpaSecrets = new aws.iam.RolePolicyAttachment("rpa-secrets", {
 role: role,
 policyArn: secretManagerPolicy.arn,
});

The aws.iam.ManagedPolicy.AWSLambdaBasicExecutionRole is one of our enums that points at the relevant ARN for this policy.

Finally, the Function

There are two parts to this: the Lambda Function resource and the code we’re going to deploy as part of the function.

The Resource

The function resource itself is pretty simple. The interesting bit is how we take the array of the ARNs we created earlier, combine it with the array of environment variable values we have in the config and assign them to the environment input:

const lambdaEnvVars: pulumi.Input<{
 [key: string]: pulumi.Input<string>;
}> = {};

for (const key in secretsArnArray) {
 if (secretsArnArray.hasOwnProperty(key)) {
 lambdaEnvVars[key] = secretsArnArray[key];
 }
}

for (const key in lambdaconfig.envvars) {
 if (lambdaconfig.envvars.hasOwnProperty(key)) {
 lambdaEnvVars[key.toLocaleUpperCase()] = lambdaconfig.envvars[key];
 }
}

The reason that we’re using the .toLocaleUpperCase() function is because as a rule, the usual way of defining environment variable names is using upper case letters.

Once we’ve done this we can put together the function resource:

const lambda = new aws.lambda.Function("lambdaWithSecrets", {
 code: new pulumi.asset.AssetArchive({
 ".": new pulumi.asset.FileArchive("./app"),
 }),
 role: role.arn,
 handler: "index.handler",
 runtime: aws.lambda.Runtime.NodeJS16dX,
 environment: {
 variables: lambdaEnvVars,
 },
 timeout: 15,
});

export const lambdaName = lambda.name;

There’s nothing complicated about the function. We’re using the role from earlier, setting the environment variables to be the object we created just above and pointed the code input at a local folder called app, which we’ll look at next. The lambda name being exported at the end is so that we can use the Pulumi CLI to get this value and invoke the function using the AWS CLI.

The Code

First we need somewhere to put the code:

mkdir app && cd app

Then we create a new NPM application and install the AWS client-secrets-manager package that we’ll use to access the secrets in the code:

npm init -y && npm install @aws-sdk/client-secrets-manager

We’re also going to update the type setting in the package.json file to be module so that we can have a top level async function. Here’s what our file looks like:

{
 "name": "app",
 "version": "1.0.0",
 "description": "",
 "main": "index.js",
 "scripts": {
 "test": "echo \"Error: no test specified\" && exit 1"
 },
 "keywords": [],
 "author": "",
 "license": "ISC",
 "dependencies": {
 "@aws-sdk/client-secrets-manager": "^3.183.0"
 },
 "type": "module"
}

The index.js file for the lambda isn’t exciting. We’re going to make the function more efficient by making the calls to the secrets manager API outside of the main handler code, so it doesn’t get run for all invocations (just the ones during a cold start):

import { SecretsManagerClient, GetSecretValueCommand } from "@aws-sdk/client-secrets-manager"; // ES Modules import
const client = new SecretsManagerClient();

const getSecretForLambda = async (secretId) => {
 const input = { "SecretId": secretId };
 const command = new GetSecretValueCommand(input);
 console.log('Retrieving secret during top-level await');
 return await client.send(command);
}

const secret1 = getSecretForLambda(process.env.SECRET1);
const secret2 = getSecretForLambda(process.env.SECRET2);

export async function handler() {
 console.log('Using secret in handler');
 return {
 secret1: (await secret1).SecretString,
 secret2: (await secret2).SecretString,
 };
};

The above JavaScript code will return the secret values from the function and is purely for demo purposes. In the Real World^TM you definitely shouldn’t do this.

Deployment time

Now that we’ve got everything together we can deploy this:

pulumi up

and you should see something like the following:

Previewing update (dev)

View Live: https://app.pulumi.com/acmecorp/lambda-secrets/dev/previews/209f07fb-190d-4596-af9f-d2d72e9c5cc9

 Type Name Plan
 + pulumi:pulumi:Stack lambda-secrets-dev create
 + ├─ aws:secretsmanager:Secret secret1 create
 + ├─ aws:secretsmanager:Secret secret2 create
 + ├─ aws:iam:Role roleLambdaWithSecrets create
 + ├─ aws:secretsmanager:SecretVersion secretversion-secret2 create
 + ├─ aws:secretsmanager:SecretVersion secretversion-secret1 create
 + ├─ aws:iam:Policy secretsPolicy create
 + ├─ aws:iam:RolePolicyAttachment rpa-basic create
 + ├─ aws:iam:RolePolicyAttachment rpa-secrets create
 + └─ aws:lambda:Function lambdaWithSecrets create

Outputs:
 lambdaName: "lambdaWithSecrets-b8ac5e9"

Resources:
 + 10 to create

Do you want to perform this update? yes
Updating (dev)

View Live: https://app.pulumi.com/acmecorp/lambda-secrets/dev/updates/1

 Type Name Status
 + pulumi:pulumi:Stack lambda-secrets-dev created
 + ├─ aws:secretsmanager:Secret secret1 created
 + ├─ aws:secretsmanager:Secret secret2 created
 + ├─ aws:iam:Role roleLambdaWithSecrets created
 + ├─ aws:secretsmanager:SecretVersion secretversion-secret2 created
 + ├─ aws:secretsmanager:SecretVersion secretversion-secret1 created
 + ├─ aws:iam:Policy secretsPolicy created
 + ├─ aws:iam:RolePolicyAttachment rpa-basic created
 + ├─ aws:lambda:Function lambdaWithSecrets created
 + └─ aws:iam:RolePolicyAttachment rpa-secrets created

Outputs:
 lambdaName: "lambdaWithSecrets-ca62cf8"

Resources:
 + 10 created

Duration: 24s

Let’s see what this has done. If we go to look at the Lambda Function in the console, click on the Configuration tab and select Environment variables you’ll see the two plaintext environment variables and their values along with the two secret environment variables containing the ARNs to the relative secret in the Secrets Manager. Since we know the environment variable key (the uppercase version of what’s in the stack config), we can reliably use that same name in the function code (the two lines where we call the getSecretForLambda function).

For me, it looks something like this:

Finally, let’s run the function via the AWS CLI and see what we get:

aws lambda invoke --function-name $(pulumi stack output lambdaName) /dev/stdout

You should see something like the following (depending on what values you added to the stack config at the beginning):

{"secret1":"secret1value","secret2":"secret2value"}

You can also use the test function in the AWS Console and see the results there.

Tidying up

As stated earlier, it costs to store and access secrets so we need to make sure that we remove the infrastructure once we’ve completed this demo. As ever, you can do this by running pulumi destroy:

Previewing destroy (dev)

View Live: https://app.pulumi.com/pierskarsenbarg/lambda-secrets/dev/previews/f111f92e-1dbb-4615-9871-30de417bf8f1

 Type Name Plan
 - pulumi:pulumi:Stack lambda-secrets-dev delete
 - ├─ aws:iam:RolePolicyAttachment rpa-secrets delete
 - ├─ aws:iam:RolePolicyAttachment rpa-basic delete
 - ├─ aws:secretsmanager:SecretVersion secretversion-secret1 delete
 - ├─ aws:iam:Policy secretsPolicy delete
 - ├─ aws:lambda:Function lambdaWithSecrets delete
 - ├─ aws:secretsmanager:SecretVersion secretversion-secret2 delete
 - ├─ aws:secretsmanager:Secret secret2 delete
 - ├─ aws:iam:Role roleLambdaWithSecrets delete
 - └─ aws:secretsmanager:Secret secret1 delete

Outputs:
 - lambdaName: "lambdaWithSecrets-ca62cf8"

Resources:
 - 10 to delete

Do you want to perform this destroy? [Use arrows to move, enter to select, type
Do you want to perform this destroy? [Use arrows to move, enter to select, type
Do you want to perform this destroy? yes
Destroying (dev)

View Live: https://app.pulumi.com/pierskarsenbarg/lambda-secrets/dev/updates/64

 Type Name Status
 - pulumi:pulumi:Stack lambda-secrets-dev deleted
 - ├─ aws:iam:RolePolicyAttachment rpa-secrets deleted
 - ├─ aws:iam:Policy secretsPolicy deleted
 - ├─ aws:secretsmanager:SecretVersion secretversion-secret1 deleted
 - ├─ aws:secretsmanager:SecretVersion secretversion-secret2 deleted
 - ├─ aws:lambda:Function lambdaWithSecrets deleted
 - ├─ aws:iam:RolePolicyAttachment rpa-basic deleted
 - ├─ aws:secretsmanager:Secret secret2 deleted
 - ├─ aws:secretsmanager:Secret secret1 deleted
 - └─ aws:iam:Role roleLambdaWithSecrets deleted

Outputs:
 - lambdaName: "lambdaWithSecrets-ca62cf8"

Resources:
 - 10 deleted

Duration: 9s

The resources in the stack have been deleted, but the history and configuration associated with the stack are still maintained.
If you want to remove the stack completely, run `pulumi stack rm dev`.

And to complete the tidying up, let’s delete the stack and all the relevant history as well:

pulumi stack rm dev

Complete code

If you just want to copy and paste the entire index.ts file without worrying which section goes where, I got you covered:

// Copyright 2016-2022, Pulumi Corporation. All rights reserved.

import * as aws from "@pulumi/aws";
import * as pulumi from "@pulumi/pulumi";

interface LambdaConfig {
 envvars: string[];
 secrets: pulumi.Output<string>[];
}

const config = new pulumi.Config();
const lambdaconfig = config.requireObject<LambdaConfig>("lambdawithsecrets");

const secretsArnArray: pulumi.Output<string>[] = new Array();
const secretArray: aws.secretsmanager.Secret[] = new Array();
for (const key in lambdaconfig.secrets) {
 if (lambdaconfig.secrets.hasOwnProperty(key)) {
 const secret = new aws.secretsmanager.Secret(`${key}`);
 const secretVersion = new aws.secretsmanager.SecretVersion(`secretversion-${key}`, {
 secretId: secret.id,
 secretString: lambdaconfig.secrets[key],
 });
 secretArray.push(secret);
 secretsArnArray[key.toLocaleUpperCase()] = secret.id;
 }
}

const role = new aws.iam.Role("roleLambdaWithSecrets", {
 assumeRolePolicy: aws.iam.assumeRolePolicyForPrincipal(aws.iam.Principals.LambdaPrincipal),
});

const rpaBasic = new aws.iam.RolePolicyAttachment("rpa-basic", {
 role: role,
 policyArn: aws.iam.ManagedPolicy.AWSLambdaBasicExecutionRole,
});



const secretManagerPolicyDoc = aws.iam.getPolicyDocumentOutput({
 statements: [
 {
 effect: "Allow",
 actions: [
 "secretsmanager:GetSecretValue",
 ],
 resources: secretArray.map(x => pulumi.interpolate`${x.arn}`),
 },
 ],
});

const secretManagerPolicy = new aws.iam.Policy("secretsPolicy", {
 policy: secretManagerPolicyDoc.apply(doc => doc.json),
});

const rpaSecrets = new aws.iam.RolePolicyAttachment("rpa-secrets", {
 role: role,
 policyArn: secretManagerPolicy.arn,
});

const lambdaEnvVars: pulumi.Input<{
 [key: string]: pulumi.Input<string>;
}> = {};

for (const key in secretsArnArray) {
 if (secretsArnArray.hasOwnProperty(key)) {
 lambdaEnvVars[key] = secretsArnArray[key];
 }
}

for (const key in lambdaconfig.envvars) {
 if (lambdaconfig.envvars.hasOwnProperty(key)) {
 lambdaEnvVars[key.toLocaleUpperCase()] = lambdaconfig.envvars[key];
 }
}

const lambda = new aws.lambda.Function("lambdaWithSecrets", {
 code: new pulumi.asset.AssetArchive({
 ".": new pulumi.asset.FileArchive("./app"),
 }),
 role: role.arn,
 handler: "index.handler",
 runtime: aws.lambda.Runtime.NodeJS16dX,
 environment: {
 variables: lambdaEnvVars,
 },
 timeout: 15,
});

export const lambdaName = lambda.name;

Summary

This post describes one of the ways that you can securely access secrets in your Lambda Functions by adding secrets to your stack config, uploading them to AWS Secrets Manager and then accessing them in your functions, all without exposing the plaintext values. There are other ways, but I think this is one of the neatest.

Of course you can use other tools to manage your secrets, such as Azure Key Vault, Google Cloud Secret Manager or Hashicorp Vault, getting the path to the key back from the resource and using it in the same way as we used the ARN from the Secret.

If you’d like more information on how you can use Pulumi to securely and easily deploy your cloud infrastructure using modern programming languages, get in touch and ask us how.

Secure AWS Lambda with IAM ABAC Policies

Paul Stack — Tue, 19 Jul 2022 00:00:00 +0000

Event-driven, serverless functions have become a defining feature of many modern cloud architectures. With recent capabilities such as AWS Lambda URLs and AWS Lambda Containers, AWS has made it clear that Lambda Functions are a platform that teams can use to deliver increasingly sophisticated services without worrying about managing underlying compute resources.

Today, AWS announced another advancement for their Lambda Functions platform: Attribute-Based Access Control (ABAC). At its core, ABAC support brings more granular permissions that are automatically applied based on IAM role tags, Lambda tags, or both. This update builds on well-established Role-Based Access Control (RBAC) principles while making it possible to implement granular controls without permissions updates for every new user and resource.

What are Attributes?

Attributes are a key or key/value pair - in AWS, these attributes are called tags. Tags can be applied to multiple roles – for example identifying members of a team.

Across many teams, your organization may share a lot of common roles, and, in order to enforce the principle of least access, you may wish to restrict access across teams such that Developer Team members cannot access Ops Team resources and vice versa.

Using ABAC for AWS Lambda Functions

For this example, we’ll define a new role ‘abac-test’ as well as a team attribute that must have the value of either ‘developers’ or ‘ops’ to enable creating a Lambda function.

To get started, let’s first install the Pulumi AWS provider:

$ npm install @pulumi/aws

$ dotnet add package Pulumi.Aws

$ pip3 install pulumi_aws

$ go get github.com/pulumi/pulumi-aws/sdk/v5

Now, create a new IAM Role that a user can assume:

const role = new aws.iam.Role("abac-test", {
 assumeRolePolicy: JSON.stringify({
 Version: "2012-10-17",
 Statement: [
 {
 Sid: "",
 Effect: "Allow",
 Principal: {
 AWS: "arn:aws:iam::MYACCOUNTID:root",
 },
 Action: "sts:AssumeRole",
 },
 ],
 }),
});

var role = new Aws.Iam.Role("abac-test", new Aws.Iam.RoleArgs
{
 AssumeRolePolicy = JsonSerializer.Serialize(new Dictionary<string, object?>
 {
 { "Version", "2012-10-17" },
 { "Statement", new[]
 {
 new Dictionary<string, object?>
 {
 { "Action", "sts:AssumeRole" },
 { "Effect", "Allow" },
 { "Principal", new Dictionary<string, object?>
 {
 { "AWS", "arn:aws:iam::MYACCOUNTID:root" },
 } },
 },
 }
 },
 }),
});

role = aws.iam.Role("abac-test",
 assume_role_policy=json.dumps({
 "Version": "2012-10-17",
 "Statement": [{
 "Action": "sts:AssumeRole",
 "Effect": "Allow",
 "Principal": {
 "AWS": "arn:aws:iam::MYACCOUNTID:root",
 },
 }],
 }),
 tags={
 "Team": "developers",
 })

role, err := iam.NewRole(ctx, "abac-test", &iam.RoleArgs{
 AssumeRolePolicy: pulumi.String(`{
 "Version": "2012-10-17",
 "Statement": [{
 "Sid": "",
 "Effect": "Allow",
 "Principal": {
 "AWS": "arn:aws:iam::MYACCOUNTID:root"
 },
 "Action": "sts:AssumeRole"
 }]
 }`),
})
if err != nil {
 return err
}

 abacTest:
 type: aws:iam:Role
 properties:
 assumeRolePolicy:
 Fn::ToJSON:
 Version: 2012-10-17
 Statement:
 - Action: sts:AssumeRole
 Effect: Allow
 Principal:
 AWS: arn:aws:iam::MYACCOUNTID:root

Let’s now create a new IAM Policy that allows a user to create Lambda functions:

const createPolicy = new aws.iam.Policy(
 "createLambda",
 {
 policy: JSON.stringify({
 Version: "2012-10-17",
 Statement: [
 {
 Effect: "Allow",
 Action: ["lambda:CreateFunction", "lambda:TagResource"],
 Resource: "arn:aws:lambda:*:*:function:*",
 Condition: {
 // the requesting resource must have a tag with
 // team = developers/ops
 StringEquals: {
 "aws:RequestTag/Team": ["developers", "ops"], // must match
 },
 // your request must contain these tags
 // eg. project, but you might not have this defined yet
 "ForAllValues:StringEquals": {
 "aws:TagKeys": "Team", // must exist
 },
 },
 },
 {
 // Pulumi needs to check what functions exist and what versions exist to create updates and new versions
 Effect: "Allow",
 Action: [
 "lambda:GetFunction",
 "lambda:ListVersionsByFunction",
 "lambda:GetFunctionCodeSigningConfig",
 ],
 Resource: "*",
 },
 {
 Effect: "Allow",
 // These are assume role policies
 Action: ["iam:ListRoles", "iam:PassRole"],
 Resource: "*",
 },
 ],
 }),
 },
);

var createPolicy = new Aws.Iam.Policy("createPolicy", new Aws.Iam.PolicyArgs
{
 PolicyDocument = JsonSerializer.Serialize(new Dictionary<string, object?>
 {
 { "Version", "2012-10-17" },
 { "Statement", new[]
 {
 new Dictionary<string, object?>
 {
 { "Effect", "Allow" },
 { "Action", new[]
 {
 "lambda:CreateFunction",
 "lambda:TagResource",
 }
 },
 { "Resource", "arn:aws:lambda:*:*:function:*" },
 { "Condition", new Dictionary<string, object?>
 {
 // the requesting resource must have a tag with
 // team = developers/ops
 { "StringEquals", new Dictionary<string, object?>
 {
 { "aws:RequestTag/Team", new[]
 {
 "developers",
 "ops",
 }
 },
 } },
 // your request must contain these tags
 // eg. project, but you might not have this defined yet
 { "ForAllValues:StringEquals", new Dictionary<string, object?>
 {
 { "aws:TagKeys", new[]
 {
 "Team",
 }
 },
 } },
 } },
 },
 new Dictionary<string, object?>
 {
 // Pulumi needs to check what functions exist and what versions exist to create updates and new versions
 { "Effect", "Allow" },
 { "Action", new[]
 {
 "lambda:GetFunction",
 "lambda:ListVersionsByFunction",
 "lambda:GetFunctionCodeSigningConfig",
 }
 },
 { "Resource", "*" },
 },
 new Dictionary<string, object?>
 {
 // These are assume role policies
 { "Effect", "Allow" },
 { "Action", new[]
 {
 "iam:ListRoles",
 "iam:PassRole",
 }
 },
 { "Resource", "*" },
 },
 }
 },
 }),
});

create_policy = aws.iam.Policy("createPolicy", policy=json.dumps({
 "Version": "2012-10-17",
 "Statement": [
 {
 "Effect": "Allow",
 "Action": [
 "lambda:CreateFunction",
 "lambda:TagResource",
 ],
 "Resource": "arn:aws:lambda:*:*:function:*",
 "Condition": {
 #the requesting resource must have a tag with
 #team = developers/ops
 "StringEquals": {
 "aws:RequestTag/Team": [
 "developers",
 "ops",
 ],
 },
 # your request must contain these tags
 # eg. project, but you might not have this defined yet
 "ForAllValues:StringEquals": {
 "aws:TagKeys": ["Team"],
 },
 },
 },
 {
 # Pulumi needs to check what functions exist and what versions exist to create updates and new versions
 "Effect": "Allow",
 "Action": [
 "lambda:GetFunction",
 "lambda:ListVersionsByFunction",
 "lambda:GetFunctionCodeSigningConfig",
 ],
 "Resource": "*",
 },
 {
 # These are assume role policies
 "Effect": "Allow",
 "Action": [
 "iam:ListRoles",
 "iam:PassRole",
 ],
 "Resource": "*",
 },
 ],
}))

createPolicy, err := iam.NewRolePolicy(ctx, "createPolicy", &iam.RoleArgs{
 Policy: pulumi.String(`{
 "Version": "2012-10-17",
 "Statement": [{
 "Sid": "",
 "Effect": "Allow",
 "Condition": map[string]interface{}{
 "StringEquals": map[string]interface{}{
 "aws:RequestTag/Team": []string{
 "developers",
 "ops",
 },
 },
 "ForAllValues:StringEquals": map[string]interface{}{
 "aws:TagKeys": []string{
 "Team",
 },
 },
 },
 "Resource": "arn:aws:lambda:*:*:function:*",
 "Action": []string{
 "lambda:CreateFunction",
 "lambda:TagResource",
 },
 },
 {
 "Effect": "Allow",
 "Resource": "*",
 "Action": []string{
 "lambda:GetFunction",
 "lambda:ListVersionsByFunction",
 "lambda:GetFunctionCodeSigningConfig",
 },
 },
 {
 {
 "Effect": "Allow",
 "Resource": "*",
 "Action": []string{
 "lambda:CreateFunction",
 "lambda:TagResource",
 },
 }
 }]
 }`),
})
if err != nil {
 return err
}

 createPolicy:
 type: aws:iam:Policy
 properties:
 policy:
 Fn::ToJSON:
 Version: 2012-10-17
 Statement:
 - Effect: Allow
 Action:
 - lambda:CreateFunction
 - lambda:TagResource
 Resource: 'arn:aws:lambda:*:*:function:*'
 Condition:
 StringEquals:
 aws:RequestTag/Team:
 - developers
 - ops
 ForAllValues:StringEquals:
 aws:TagKeys:
 - Team
 - Effect: Allow
 Action:
 - lambda:GetFunction
 - lambda:ListVersionsByFunction
 - lambda:GetFunctionCodeSigningConfig
 Resource: '*'
 - Effect: Allow
 Action:
 - iam:ListRoles
 - iam:PassRole
 Resource: '*'

Notice, as part of the policy condition, we ensure that the requesting resource must have a tag with team: developers or team: ops.

Let’s ensure that this policy is attached to the Role we initially created:

const rpa = new aws.iam.RolePolicyAttachment("rpa", {
 policyArn: createPolicy.arn,
 role: role,
})

var rpa = new Aws.Iam.RolePolicyAttachment("rpa", new Aws.Iam.RolePolicyAttachmentArgs
{
 PolicyArn = createPolicy.Arn,
 Role = role.Name,
});

rpa = aws.iam.RolePolicyAttachment("rpa",
 policy_arn=create_policy.arn,
 role=role.name)

rpa, err := iam.NewRolePolicyAttachment(ctx, "rpa", &iam.RolePolicyAttachmentArgs{
 PolicyArn: createPolicy.Arn,
 Role: role.Name,
})
if err != nil {
 return err
}

 rpa:
 type: aws:iam:RolePolicyAttachment
 properties:
 policyArn: ${createPolicy.arn}
 role: ${abacTest.name}

We can deploy these resources using the command pulumi update:

pulumi up
Previewing update (abac-testing)

View Live: https://app.pulumi.com/stack72/aws-abac-test/abac-testing/previews/c6fb4580-2706-47ee-8a56-b8942631254e

 Type Name Plan
 + pulumi:pulumi:Stack aws-abac-test-abac-testing create
 + ├─ aws:iam:Policy createLambda create
 + ├─ aws:iam:Role abac-test create
 + └─ aws:iam:RolePolicyAttachment createPolicy create

Resources:
 + 4 to create

Do you want to perform this update? yes
Updating (abac-testing)

View Live: https://app.pulumi.com/stack72/aws-abac-test/abac-testing/updates/1

 Type Name Status
 + pulumi:pulumi:Stack aws-abac-test-abac-testing created
 + ├─ aws:iam:Role abac-test created
 + ├─ aws:iam:Policy createLambda created
 + └─ aws:iam:RolePolicyAttachment createPolicy created

Resources:
 + 4 created

Duration: 7s

For the purposes of testing, we are now going to create a specific AWS Provider resource that we can pass the IAM Role details to assume.

const abacUserProvider = new aws.Provider(
 "assumeRole",
 {
 // assume the previously created role with the ABAC configured policy into our provider
 assumeRole: {
 roleArn: role.arn,
 sessionName: "abacTesting",
 },
 region: "us-east-1",
 },
 {
 // we want to depend on the role policy attachment being in place before we create the provider
 dependsOn: [rpa],
 }
);

var abacUserProvider = new Aws.Provider("abacUserProvider", new Aws.ProviderArgs
{
 AssumeRole = new Aws.Config.Inputs.AssumeRoleArgs
 {
 RoleArn = role.Arn,
 SessionName = "abacTesting",
 },
 Region = "us-east-1",
}, new CustomResourceOptions
{
 DependsOn =
 {
 rpa,
 },
});

abac_user_provider = pulumi.providers.Aws("abacUserProvider",
 assume_role=aws.config.AssumeRoleArgs(
 role_arn=role.arn,
 session_name="abacTesting",
 ),
 region="us-east-1",
 opts=pulumi.ResourceOptions(depends_on=[rpa]))

abacUserProvider, err := providers.Newaws(ctx, "abacUserProvider", &providers.awsArgs{
 AssumeRole: config.AssumeRole{
 RoleArn: role.Arn,
 SessionName: "abacTesting",
 },
 Region: "us-east-1",
}, pulumi.DependsOn([]pulumi.Resource{
 rpa,
}))
if err != nil {
 return err
}

 abacUserProvider:
 type: pulumi:providers:aws
 properties:
 assumeRole:
 roleArn: ${role.arn}
 sessionName: "abacTesting"
 region: "us-east-1"
 options:
 dependsOn:
 - ${rpa}

Now we can create an AWS Lambda with Pulumi as follows:

const lambdaRole = new aws.iam.Role("iamRole", {
 assumeRolePolicy: aws.iam.assumeRolePolicyForPrincipal({
 Service: "lambda.amazonaws.com",
 }),
});

const deleteLambdaPolicy = new aws.iam.Policy(
 "deleteLambdaPolicy",
 {
 policy: JSON.stringify({
 Version: "2012-10-17",
 Statement: [
 {
 Effect: "Allow",
 Action: ["lambda:DeleteFunction"],
 Resource: "*",
 },
 ],
 }),
 },
 { parent: lambdaRole }
);

new aws.iam.RolePolicyAttachment(
 `deletePolicy`,
 {
 policyArn: deletePolicy.arn,
 role: role,
 },
 { parent: deletePolicy })

const lambda = new aws.lambda.Function(
 "lambdaFunction",
 {
 code: new pulumi.asset.AssetArchive({
 ".": new pulumi.asset.FileArchive("./app"),
 }),
 runtime: "nodejs12.x",
 role: lambdaRole.arn,
 handler: "index.handler",
 },
 { provider: abacUserProvider }
);

var lambdaRole = new Aws.Iam.Role("lambdaRole", new Aws.Iam.RoleArgs
{
 AssumeRolePolicy = JsonSerializer.Serialize(new Dictionary<string, object?>
 {
 { "Version", "2012-10-17" },
 { "Statement", new[]
 {
 new Dictionary<string, object?>
 {
 { "Action", "sts:AssumeRole" },
 { "Effect", "Allow" },
 { "Principal", new Dictionary<string, object?>
 {
 { "Service", "lambda.amazonaws.com" },
 } },
 },
 }
 },
 }),
});
var deletePolicy = new Aws.Iam.Policy("deletePolicy", new Aws.Iam.PolicyArgs
{
 PolicyDocument = JsonSerializer.Serialize(new Dictionary<string, object?>
 {
 { "Version", "2012-10-17" },
 { "Statement", new[]
 {
 new Dictionary<string, object?>
 {
 { "Action", "lambda:DeleteFunction" },
 { "Effect", "Allow" },
 { "Resource", "*" },
 },
 }
 },
 }),
});
var deletePolicyAttachment = new Aws.Iam.RolePolicyAttachment("deletePolicyAttachment", new Aws.Iam.RolePolicyAttachmentArgs
{
 Role = role.Name,
 PolicyArn = deletePolicy.Arn,
});
var lambdaFunction = new Aws.Lambda.Function("lambdaFunction", new Aws.Lambda.FunctionArgs
{
 Code = new FileArchive("./app"),
 Role = lambdaRole.Arn,
 Handler = "index.handler",
 Runtime = "nodejs14.x",
}, new CustomResourceOptions
{
 Provider = abacUserProvider,
});

lambda_role = aws.iam.Role("lambdaRole", assume_role_policy=json.dumps({
 "Version": "2012-10-17",
 "Statement": [{
 "Action": "sts:AssumeRole",
 "Effect": "Allow",
 "Principal": {
 "Service": "lambda.amazonaws.com",
 },
 }],
}))
delete_policy = aws.iam.Policy("deletePolicy", policy=json.dumps({
 "Version": "2012-10-17",
 "Statement": [{
 "Action": "lambda:DeleteFunction",
 "Effect": "Allow",
 "Resource": "*",
 }],
}))
delete_policy_attachment = aws.iam.RolePolicyAttachment("deletePolicyAttachment",
 role=role.name,
 policy_arn=delete_policy.arn)
lambda_function = aws.lambda_.Function("lambdaFunction",
 code=pulumi.FileArchive("./app"),
 role=lambda_role.arn,
 handler="index.handler",
 runtime="nodejs14.x",
 opts=pulumi.ResourceOptions(provider=abac_user_provider))

lambdaRole, err := iam.NewRole(ctx, "lambdaRole", &iam.RoleArgs{
 AssumeRolePolicy: pulumi.String(`{
 "Version": "2012-10-17",
 "Statement": [{
 "Effect": "Allow",
 "Principal": {
 "Service": "lambda.amazonaws.com"
 },
 "Action": "sts:AssumeRole"
 }]
 }`),
})
if err != nil {
 return err
}

deletePolicy, err := iam.NewRolePolicy(ctx, "deletePoicy", &iam.RoleArgs{
 Policy: pulumi.String(`{
 "Version": "2012-10-17",
 "Statement": [{
 "Effect": "Allow",
 "Resource": "*",
 "Action": []string{
 "lambda:DeleteFunction",
 },
 }]
 }`),
})
if err != nil {
 return err
}

_, err = iam.NewRolePolicyAttachment(ctx, "deletePolicyAttachment", &iam.RolePolicyAttachmentArgs{
 Role: role.Name,
 PolicyArn: deletePolicy.Arn,
})
if err != nil {
 return err
}

_, err = lambda.NewFunction(ctx, "lambdaFunction", &lambda.FunctionArgs{
 Code: pulumi.NewFileArchive("./app"),
 Role: lambdaRole.Arn,
 Handler: pulumi.String("index.handler"),
 Runtime: pulumi.String("nodejs14.x"),
}, pulumi.Provider(abacUserProvider))
if err != nil {
 return err
}

 lambdaRole:
 type: aws:iam:Role
 properties:
 assumeRolePolicy:
 Fn::ToJSON:
 Version: 2012-10-17
 Statement:
 - Action: sts:AssumeRole
 Effect: Allow
 Principal:
 Service: lambda.amazonaws.com
 deletePolicy:
 type: aws:iam:Policy
 properties:
 policy:
 Fn::ToJSON:
 Version: 2012-10-17
 Statement:
 - Action: lambda:DeleteFunction
 Effect: Allow
 Resource: '*'
 deletePolicyAttachment:
 type: aws:iam:RolePolicyAttachment
 properties:
 role: ${abacRole.name}
 policyArn: ${deletePolicy.arn}
 lambdaFunction:
 type: aws:lambda:Function
 properties:
 code:
 Fn::FileArchive: ./app
 role: ${lambdaRole.arn}
 handler: index.handler
 runtime: nodejs14.x

If we try and deploy the lambda, we will get the following using the role we are going to assume then we will get an error as the lambda function has no Team tag.

pulumi up
Previewing update (abac-testing)

View Live: https://app.pulumi.com/stack72/aws-abac-test/abac-testing/previews/7fdc10b2-5e44-497e-a067-f2e8b257496c

 Type Name Plan
 pulumi:pulumi:Stack aws-abac-test-abac-testing
 + ├─ aws:iam:Role iamRole create
 ├─ aws:iam:Role abac-test
 + │ └─ aws:iam:Policy deletePolicy create
 + │ └─ aws:iam:RolePolicyAttachment deletePolicy create
 + ├─ pulumi:providers:aws assumeRole create
 + └─ aws:lambda:Function lambdaFunction create

Resources:
 + 5 to create
 4 unchanged

Do you want to perform this update? yes
Updating (abac-testing)

View Live: https://app.pulumi.com/stack72/aws-abac-test/abac-testing/updates/2

 Type Name Status Info
 pulumi:pulumi:Stack aws-abac-test-abac-testing **failed** 1 error
 + ├─ aws:iam:Role iamRole created
 ├─ aws:iam:Role abac-test
 + │ └─ aws:iam:Policy deletePolicy created
 + │ └─ aws:iam:RolePolicyAttachment deletePolicy created
 + ├─ pulumi:providers:aws assumeRole created
 + └─ aws:lambda:Function lambdaFunction **creating failed** 1 error

Diagnostics:
 pulumi:pulumi:Stack (aws-abac-test-abac-testing):
 error: update failed

 aws:lambda:Function (lambdaFunction):
 error: 1 error occurred:
 * error creating Lambda Function (1): AccessDeniedException:
 status code: 403, request id: 4a8676f2-2b68-4d42-ad4b-f0e10ca31afd

Resources:
 + 4 created
 4 unchanged

Duration: 6s

If we add the correct Team tag to the Lambda function then the deploy will proceed as expected:

pulumi up
Previewing update (abac-testing)

View Live: https://app.pulumi.com/stack72/aws-abac-test/abac-testing/previews/0afa267f-5f9f-43f5-9e00-d3ee19dbf2b8

 Type Name Plan
 pulumi:pulumi:Stack aws-abac-test-abac-testing
 + └─ aws:lambda:Function lambdaFunction create

Resources:
 + 1 to create
 8 unchanged

Do you want to perform this update? yes
Updating (abac-testing)

View Live: https://app.pulumi.com/stack72/aws-abac-test/abac-testing/updates/3

 Type Name Status
 pulumi:pulumi:Stack aws-abac-test-abac-testing
 + └─ aws:lambda:Function lambdaFunction created

Resources:
 + 1 created
 8 unchanged

Duration: 9s

Try It Out!

Lambda ABAC capabilities will help you to simplify governance by helping you to standardize roles and maintain security boundaries between teams. A common use case is using tags to restrict/enable the invoke action for a function in addition to the create action demonstrated above. Give Lambda ABAC functionality a try and let us know what you think in Community Slack.

Deploying a URL Shortener with Cloudflare Workers

David Flanagan — Wed, 29 Jun 2022 23:00:09 +0100

Cloudflare Workers provides a serverless execution environment that allows you to create entirely new applications or augment existing ones without configuring or maintaining infrastructure. They support NodeJS and WebAssembly (WASM), as well as any language that can compile to WASM.

Delivered from over 250 locations worldwide, Cloudflare could be the best way to bring down that latency that’s plaguing your customers. Claiming 0ms for cold starts, automatic scaling, 100k free requests per day, and edge storage built-in: Cloudflare offers a pretty compelling edge compute platform for serverless workloads.

Let’s see how we can deploy a low-latency serverless URL shortener to Cloudflare Workers with Pulumi.

Setting Up

You’ll need a Pulumi project setup with the Cloudflare package added to your Pulumi project and an API token configured.

pulumi new typescript
npm install @pulumi/cloudflare

pulumi new python
pip install pulumi-cloudflare

pulumi new go
go get github.com/pulumi/pulumi-cloudflare/sdk/v4/go/...

pulumi new yaml

Now that we’ve added the package, we need to configure it. You’ll need your Cloudflare accountId and an API Token. The permissions you need for the API token are:

All Accounts
- Workers Tail:Read
- Workers Scripts:Edit
- Account Settings:Read
All Zones
- Zone Settings:Edit
- Zone:Edit
- Workers Routes:Edit
- SSL and Certificates:Edit
- DNS:Edit
All Users
- User Details:Read

You can reduce the scope of “All Zones”, but this means you cannot create a Zone with Pulumi and instead would need to use getZone, described below.

pulumi config set --secret cloudflare:apiToken
pulumi config set cloudflare:accountId

Our URL Shortener

We’re going to use plain old JavaScript for our worker code, as this means we don’t need to transpile anything down. Our URL Shortener will store domains, short tokens, and long URLs in a JavaScript object. We’ll also provide a DEFAULT_URL in-case we can’t match the path requested.

We need to keep our worker source code separate from our Pulumi code, so we’ll be keeping this contained within a file called worker.js.

// worker.js
const DEFAULT_URL = "https://youtube.com/pulumitv"

const redirects = {
 "pulumi.tv": {
 "modern-infrastructure": {
 to: "https://www.youtube.com/playlist?list=PLyy8Vx2ZoWloyj3V5gXzPraiKStO2GGZw"
 }
 },
};

addEventListener("fetch", async (event) => {
 event.respondWith(handleRequest(event.request));
});

async function handleRequest(request) {
 const requestUrl = new URL(request.url);
 const domain = requestUrl.host;
 const path = requestUrl.pathname.substring(1);

 console.log(`Handling request on domain ${domain} for ${path}`);

 if (!(domain in redirects)) {
 console.log(`Domain '${domain}' not found`);
 return Response.redirect(DEFAULT_URL);
 }

 if (path === "") {
 return Response.redirect(DEFAULT_URL);
 }

 const paths = redirects[domain];

 if (path in paths) {
 console.log(`Redirecting too ${paths[path].to}`);
 return Response.redirect(paths[path].to);
 }

 console.log(`Path '${path}' not found`);
 return Response.redirect(DEFAULT_URL);
}

Deploying Our Worker

Now on to our Pulumi code. We need to read our JavaScript code and send it to Cloudflare Workers via the WorkerScript resource.

// index.ts
import * as cloudflare from "@pulumi/cloudflare";
import * as fs from "fs";

const worker = new cloudflare.WorkerScript(
 "url-shortener",
 {
 name: "url-shortener",
 content: fs.readFileSync("worker.js").toString(),
 },
);

# __main__.py
import pulumi
import pulumi_cloudflare as cloudflare

worker_script_file = open("worker.js", "r")
worker_script = worker_script_file.read()
worker_script_file.close()

worker = cloudflare.WorkerScript("url-shortener",
 name="url-shortener",
 content=worker_script)

// main.go
package main

import (
 "fmt"
 "io/ioutil"

 "github.com/pulumi/pulumi-cloudflare/sdk/v4/go/cloudflare"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 workerScriptFile, err := ioutil.ReadFile("worker.js")
 if err != nil {
 log.Fatal(err)
 }
 workerScript := string(workerScriptFile)

 worker, err := cloudflare.NewWorkerScript(ctx, "url-shortener", &cloudflare.WorkerScriptArgs{
 Content: pulumi.String(workerScript),
 Name: pulumi.String("links"),
 }, pulumi.Protect(true))
 if err != nil {
 return err
 }
 return nil
 })
}

# Pulumi.yaml
name: url-shortener
runtime: yaml
description: url-shortener in Pulumi YAML
variables:
 workerScript:
 Fn::ReadFile: ./worker.js
resources:
 worker:
 type: cloudflare:index:WorkerScript
 properties:
 name: url-shortener
 content: ${workerScript}

Where’s the Types?!

If you wanted to use TypeScript for the worker code for some guarantees around the short/long URL objects, you can definitely do so. This does require an additional step to transpile the TypeScript to JavaScript though. Remember to update the Pulumi code to read the ts extension.

// worker.ts
interface Redirect {
 to: string;
}

interface Redirects {
 [domain: string]: {
 [path: string]: Redirect;
 };
}

const redirects: Redirects = {
 "pulumi.tv": {
 "/": {
 to: "https://youtube.com/pulumitv",
 },
 "modern-infrastructure": {
 to: "https://www.youtube.com/playlist?list=PLyy8Vx2ZoWloyj3V5gXzPraiKStO2GGZw"
 }
 },
};

You’ll also need a tsconfig.json:

{
 "compilerOptions": {
 "experimentalDecorators": true,
 "forceConsistentCasingInFileNames": true,
 "lib": ["ES2020"],
 "module": "ES2020",
 "moduleResolution": "node",
 "noFallthroughCasesInSwitch": true,
 "noImplicitReturns": true,
 "outDir": "dist",
 "pretty": true,
 "sourceMap": true,
 "strict": true,
 "target": "ES2020",
 "outDir": "dist",
 "types": ["@cloudflare/workers-types"]
 },
 "include": ["index.ts"]
}

Then we can transpile the code with a local.Command resource.

We need to add our command package and then add a local command to our code.

npm install @pulumi/command

// index.ts
import * as command from "@pulumi/command";

const compileWorker = new command.local.Command("compile-worker", {
 create:
 "yarn run tsc >/dev/null && cat ./dist/index.js",
 dir: "../worker",
});

You could then use the stdout output, compileWorker.stdout, from our local command resource for the content input on the WorkerScript resource, instead of fs.readFileSync.

pip install pulumi_command

# __main__.py
import pulumi_command as command

compile_worker = command.local.Command('compile-worker', create='yarn run tsc >/dev/null && cat ./dist/index.js')

You could then use the stdout output, compile_worker.stdout, from our local command resource for the content input on the WorkerScript resource, instead of open and read on the file.

go get github.com/pulumi/pulumi-command/sdk/go/...

// main.go
import (
 "github.com/pulumi/pulumi-command/sdk/go/command/local"
)

compileWorker, err := local.NewCommand(ctx, "compile-worker", &local.CommandArgs{
 Create: pulumi.String("yarn run tsc >/dev/null && cat ./dist/index.js"),
})

if err != nil {
 return err
}

You could then use the stdout output, compileWorker.Stdout, from our local command resource for the content input on the WorkerScript resource, instead of ioutil.ReadFile.

# Pulumi.yaml
resources:
 compileWorker:
 type: command:local:Command
 properties:
 create: yarn run tsc >/dev/null && cat ./dist/index.js

You could then use the stdout output, ${compileWorker.stdout}, from our local command resource for the content input on the WorkerScript resource, instead of Fn:ReadFile.

Hooking Up a Domain

Cloudflare Workers allow us to connect a domain name, instead of using the generated workers.dev subdomain. To do so with Pulumi, we need to create or fetch a Cloudflare Zone.

Create a Zone(s)

Once we create the cloudflare.Zone resource, we also need to create a cloudflare.ZoneSettingsOverride resource to ensure that we enable:

alwaysUseHttps: "on"
automaticHttpsRewrites: "on"
ssl: "strict"
universalSsl: "on".

Without these settings, our traffic will be available over HTTP and … come on, it’s 2022 already.

// index.ts
const zone = new cloudflare.Zone("pulumi.tv", {
 zone: "pulumi.tv",
 plan: "free",
});

const zoneSettings = new cloudflare.ZoneSettingsOverride("pulumi.tv", {
 zoneId: z.id,
 settings: {
 alwaysUseHttps: "on",
 automaticHttpsRewrites: "on",
 ssl: "strict",
 minTlsVersion: "1.2",
 universalSsl: "on",
 },
});

# __main__.py
zone = cloudflare.Zone("pulumi.tv", zone="pulumi.tv", plan="free")
zone_settings = cloudflare.ZoneSettingsOverride(
 "pulumi.tv",
 zone_id=zone.id,
 settings=cloudflare.ZoneSettingsOverrideSettingsArgs(
 always_use_https="on",
 automatic_https_rewrites="on",
 ssl="strict",
 min_tls_version="1.2",
 universal_ssl="on",
 ),
)

// main.go
zone, err := cloudflare.NewZone(ctx, "pulumi.tv", &cloudflare.ZoneArgs{
 Zone: pulumi.String("pulumi.tv"),
 Plan: pulumi.String("free"),
})
if err != nil {
 return err
}

_, err = cloudflare.NewZoneSettingsOverride(ctx, "pulumi.tv", &cloudflare.ZoneSettingsOverrideArgs{
 ZoneId: zone.ID(),
 Settings: &cloudflare.ZoneSettingsOverrideSettingsArgs{
 AlwaysUseHttps: pulumi.String("on"),
 AutomaticHttpsRewrites: pulumi.String("on"),
 UniversalSsl: pulumi.String("on"),
 Ssl: pulumi.String("strict"),
 MinTlsVersion: pulumi.String("1.2"),
 },
})
if err != nil {
 return err
}

# Pulumi.yaml
resources:
 zone:
 type: cloudflare:index:Zone
 properties:
 zone: pulumi.tv
 plan: free
 zoneSettings:
 type: cloudflare:index:ZoneSettingsOverride
 properties:
 zoneId: ${zone.id}
 settings:
 alwaysUseHttps: "on"
 automaticHttpsRewrites: "on"
 universalSsl: "on"
 ssl: "strict"
 minTlsVersion: "1.2"

Fetch a Zone

// index.ts
const zone = await cloudflare.getZone({
 name: "pulumi.tv",
});

# __main__.py
zone = cloudflare.get_zone(name="pulumi.tv")

// main.go
zone, err := cloudflare.LookupZone(ctx, &cloudflare.LookupZoneArgs{
 Name: pulumi.String("pulumi.tv"),
})

# Pulumi.yaml
variables:
 zoneId:
 Fn::Invoke:
 Function: cloudflare:index:getZone
 Arguments:
 name: pulumi.tv
 Return: id
 zoneName:
 Fn::Invoke:
 Function: cloudflare:index:getZone
 Arguments:
 name: pulumi.tv
 Return: name

Creating the Routes

In-order to actually send traffic to our workers from these zones, we need to setup the DNS records and create the routes. While Cloudflare does provide a web UI to “connect” a custom domain, in beta, this isn’t yet available over the Cloudflare API.

Instead, we need to use a proxied DNS record on a subdomain, or domain apex, that points to workers.dev.

// index.ts
const record = new cloudflare.Record(
 "pulumi.tv",
 {
 zoneId: zone.id,
 name: "@",
 type: "CNAME",
 value: "workers.dev",
 proxied: true,
 },
 {
 deleteBeforeReplace: true,
 }
);

const route = new cloudflare.WorkerRoute(
 "pulumi.tv",
 {
 zoneId: zone.id,
 pattern: pulumi.interpolate`${zone.zone}/*`,
 scriptName: worker.name,
 }
);

# __main__.py
from pulumi import Output

record = cloudflare.Record("pulumi.tv",
 name="@", zone_id=zone.id, type="CNAME", value="workers.dev")

route = cloudflare.WorkerRoute("pulumi.tv", zone_id=zone.id, pattern=Output.concat(zone.zone, "/*"), script_name=worker.name)

// main.go
_, err = cloudflare.NewRecord(ctx, "pulumi.tv", &cloudflare.RecordArgs{
 Name: pulumi.String("@"),
 ZoneId: zone.ID(),
 Type: pulumi.String("CNAME"),
 Value: pulumi.String("workers.dev"),
})
if err != nil {
 return err
}

_, err = cloudflare.NewWorkerRoute(ctx, "pulumi.tv", &cloudflare.WorkerRouteArgs{
 ZoneId: zone.ID(),
 Pattern: pulumi.Sprintf("%s/*", zone.Zone),
 ScriptName: worker.Name,
})
if err != nil {
 return err
}

# Pulumi.yaml
resources:
 record:
 type: cloudflare:index:Record
 properties:
 name: "@"
 zoneId: ${zone.id}
 type: CNAME
 value: workers.dev
 route:
 type: cloudflare:index:WorkerRoute
 properties:
 # If you fetched a Zone, this would be ${zoneId}
 zoneId: ${zone.id}
 # If you fetched a Zone, this would be ${zoneName}
 pattern: ${zone.zone}/*
 scriptName: ${worker.name}

Fin

Lastly, we can run pulumi up and run a curl 😀

❯ pulumi up

 Type Name Status
 + pulumi:pulumi:Stack short-links-production created
 + ├─ cloudflare:index:Zone pulumi.tv created
 + ├─ cloudflare:index:WorkerScript links created
 + ├─ cloudflare:index:ZoneSettingsOverride pulumi.tv created
 + ├─ cloudflare:index:Record pulumi.tv created
 + └─ cloudflare:index:WorkerRoute pulumi.tv created

Resources:
 + 7 created

Duration: 27s

❯ curl -I https://pulumi.tv
HTTP/2 302
date: Wed, 29 Jun 2022 21:57:20 GMT
location: https://youtube.com/pulumitv

❯ curl -I https://pulumi.tv/modern-infrastructure
HTTP/2 302
date: Wed, 29 Jun 2022 21:57:20 GMT
location: https://www.youtube.com/playlist?list=PLyy8Vx2ZoWloyj3V5gXzPraiKStO2GGZw

That’s it! Deploying your own URL Shortener with some JavaScript and Cloudflare Workers with Pulumi.

Complete Code

// index.ts
import * as pulumi from "@pulumi/pulumi";
import * as cloudflare from "@pulumi/cloudflare";
import * as fs from "fs";


const zone = new cloudflare.Zone("pulumi.tv", {
 "pulumi.tv",
 plan: "free",
});

const zoneSettings = new cloudflare.ZoneSettingsOverride("pulumi.tv", {
 zoneId: zone.id,
 settings: {
 alwaysUseHttps: "on",
 automaticHttpsRewrites: "on",
 ssl: "strict",
 minTlsVersion: "1.2",
 universalSsl: "on",
 },
});

const worker = new cloudflare.WorkerScript(
 "pulumi.tv",
 {
 name: "pulumi.tv",
 content: fs.readFileSync("worker.js").toString(),
 }
);

const record = new cloudflare.Record(
 "pulumi.tv",
 {
 zoneId: zone.id,
 name: "@",
 type: "CNAME",
 value: "workers.dev",
 proxied: true,
 },
 {
 deleteBeforeReplace: true,
 }
);

const workerRoute = new cloudflare.WorkerRoute(
 "pulumi.tv",
 {
 zoneId: zone.id,
 pattern: pulumi.interpolate`${zone.zone}/*`,
 scriptName: worker.name,
 }
);

# __main__.py
import pulumi
from pulumi import Output
import pulumi_cloudflare as cloudflare

worker_script_file = open("worker.js", "r")
worker_script = worker_script_file.read()
worker_script_file.close()

worker = cloudflare.WorkerScript("url-shortener", name="url-shortener", content=worker_script)

zone = cloudflare.Zone("pulumi.tv", zone="pulumi.tv", plan="free")
zone_settings = cloudflare.ZoneSettingsOverride(
 "pulumi.tv",
 zone_id=zone.id,
 settings=cloudflare.ZoneSettingsOverrideSettingsArgs(
 always_use_https="on",
 automatic_https_rewrites="on",
 ssl="strict",
 min_tls_version="1.2",
 universal_ssl="on",
 ),
)

record = cloudflare.Record("pulumi.tv",
 name="@", zone_id=zone.id, type="CNAME", value="workers.dev")

route = cloudflare.WorkerRoute("pulumi.tv", zone_id=zone.id, pattern=Output.concat(zone.zone, "/*"), script_name=worker.name)

// main.go
package main

import (
 "io/ioutil"
 "log"

 "github.com/pulumi/pulumi-cloudflare/sdk/v4/go/cloudflare"
 "github.com/pulumi/pulumi/sdk/v3/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 workerScriptFile, err := ioutil.ReadFile("worker.js")
 if err != nil {
 log.Fatal(err)
 }
 workerScript := string(workerScriptFile)

 worker, err := cloudflare.NewWorkerScript(ctx, "url-shortener", &cloudflare.WorkerScriptArgs{
 Content: pulumi.String(workerScript),
 Name: pulumi.String("links"),
 }, pulumi.Protect(true))
 if err != nil {
 return err
 }

 zone, err := cloudflare.NewZone(ctx, "pulumi.tv", &cloudflare.ZoneArgs{
 Zone: pulumi.String("pulumi.tv"),
 Plan: pulumi.String("free"),
 })
 if err != nil {
 return err
 }

 _, err = cloudflare.NewZoneSettingsOverride(ctx, "pulumi.tv", &cloudflare.ZoneSettingsOverrideArgs{
 ZoneId: zone.ID(),
 Settings: &cloudflare.ZoneSettingsOverrideSettingsArgs{
 AlwaysUseHttps: pulumi.String("on"),
 AutomaticHttpsRewrites: pulumi.String("on"),
 UniversalSsl: pulumi.String("on"),
 Ssl: pulumi.String("strict"),
 MinTlsVersion: pulumi.String("1.2"),
 },
 })
 if err != nil {
 return err
 }

 _, err = cloudflare.NewRecord(ctx, "pulumi.tv", &cloudflare.RecordArgs{
 Name: pulumi.String("@"),
 ZoneId: zone.ID(),
 Type: pulumi.String("CNAME"),
 Value: pulumi.String("workers.dev"),
 })
 if err != nil {
 return err
 }

 _, err = cloudflare.NewWorkerRoute(ctx, "pulumi.tv", &cloudflare.WorkerRouteArgs{
 ZoneId: zone.ID(),
 Pattern: pulumi.Sprintf("%s/*", zone.Zone),
 ScriptName: worker.Name,
 })
 if err != nil {
 return err
 }

 return nil
 })
}

# Pulumi.yaml
name: url-shortener
runtime: yaml
description: url-shortener in Pulumi YAML
variables:
 workerScript:
 Fn::ReadFile: ./worker.js
resources:
 worker:
 type: cloudflare:index:WorkerScript
 properties:
 name: url-shortener
 content: ${workerScript}
 zone:
 type: cloudflare:index:Zone
 properties:
 zone: rawkoded.link
 plan: free
 zoneSettings:
 type: cloudflare:index:ZoneSettingsOverride
 properties:
 zoneId: ${zone.id}
 settings:
 alwaysUseHttps: "on"
 automaticHttpsRewrites: "on"
 universalSsl: "on"
 ssl: "strict"
 minTlsVersion: "1.2"
 record:
 type: cloudflare:index:Record
 properties:
 name: "@"
 zoneId: ${zone.id}
 type: CNAME
 value: workers.dev
 route:
 type: cloudflare:index:WorkerRoute
 properties:
 zoneId: ${zone.id}
 pattern: ${zone.zone}/*
 scriptName: ${worker.name}

Test-Driven Infrastructure Development with Pulumi and Jest

Christian Nunciato — Mon, 13 Jun 2022 00:00:00 +0000

When I was a kid growing up in Southern California, there was a phone number you could call to find out what time it was. It was a local number, 853-1212 (easy to remember as the arrangement of the numbers on the keypad made a capital T), and I used it all the time, to set my watch, adjust the alarm clock, fix the display on the VCR. I don’t recall the last time I used it, probably sometime in the mid ’90s, but I do remember clearly the sound of the voice at the other end of the line.

Services like these were called speaking clocks (although I never called them that; to me, it was just “calling Time”), and they date back to the early 1930s or so. In the U.S., most were powered by a machine called an Audichron that used mechanical drums to play back the time of day, often with a little advertisement or the current temperature to go along with it.

By the mid-2000s, though, most of these locally run services had been shut down — by then, we had clocks baked into our phones — and today, only a handful of Audichrons survive. Thanks to the National Institute of Standards and Technology, however, you can still call a phone number to get the time, and while the voice might not be the same, and long distance rates will apply, it’s definitely there, and you can use it. So on the off chance you happen to find yourself with no idea what time it is and only an analog phone line in reach, fear not — old-school telephone tech has your back. For now. Assuming you remember the number.

Recalling all this stuff did make me wonder, though, what a more modern version of a speaking clock might look like. So in this post, we’re going to build one ourselves. We won’t use an actual phone number, but we will use Pulumi and AWS — and because we want to do it right, we’ll take a test-driven approach to developing the infrastructure with Jest, the JavaScript testing framework. We’ll use TypeScript and Node.js for everything, focus on unit tests, and when we’re done, we’ll have a single, serverless, browser-friendly HTTPS endpoint that returns an MP3 audio stream that speaks the current time.

Let’s get started.

Sketching it out

The first thing we’ll need is a runtime environment — someplace to run some server-side JavaScript that can render and deliver an audio file. Until recently, the easiest way to get an HTTP endpoint up and running on AWS has generally been with AWS Lambda and API Gateway, using Lambda to run the requisite code and API Gateway to expose the Lambda to the internet. Pulumi Crosswalk actually makes this really easy, too — but with the release of AWS Lambda Function URLs this April, we now have another option, one that doesn’t need API Gateway at all.

A Lambda Function URL is just what it sounds like: a URL that exposes a Lambda function. Specifically, it’s an AWS cloud resource that consists of a few properties that tell AWS whether to allow anonymous access to the function or to protect it with AWS IAM, and optionally, you can also provide a few cross-origin resource-sharing (CORS) rules to provide (or restrict) access by websites running on other domains.

For an app like this one, a Function URL is a good fit, as it’s simple, quick to deploy (and update), and it gives us all the control we need to get the job done. Architecturally, then, our little app will look something like this:

That about covers the infrastructure — but what about the sound?

As it happens, Google Translate offers a publicly accessible text-to-speech API that can, as long as you don’t overuse it, render a string to MP3 audio. We can use that API to generate the time-of-day message, tack the sound of a beep onto the end of it (for maximum nostalgic effect), and voilà: instant, infinitely scalable, cloud-native speaking clock.

Plan in place, we can kick things off by creating a new Pulumi project.

Create a new TypeScript project

Start by creating a new AWS TypeScript project in the usual way:

$ mkdir audichron-2022 && cd audichron-2022
$ pulumi new aws-typescript

Step through the prompts to create a new stack (you’ll only need one stack for this project), and when the new-project wizard completes, clear out the contents of index.ts entirely, as we’ll be building this program entirely from the ground up.

Install and configure Jest

Since we’re working with TypeScript, we can use ts-jest, which conveniently brings Jest along for the ride:

$ npm install ts-jest @types/jest --save-dev

Jest requires a bit of configuration, so add a new file at the root of the project, called jest.config.ts, with the following code:

import type { Config } from "@jest/types";

const config: Config.InitialOptions = {
 preset: "ts-jest",
};

export default config;

With Jest now installed and configured, and its helpers (like describe and expect) declared globally, you can create a new spec file to hold your tests. Add another file at the root of the project called index.spec.ts, and give it the following temporary content:

import "jest";

describe("My speaking clock", () => {
 it ("works", () => {
 expect(true).toBeTruthy;
 });
});

Lastly, in package.json, add a scripts block with a helper to make running the tests a little easier:

{
 "name": "audichron-2022",
 "devDependencies": {
 "@types/node": "^14",
 "ts-jest": "^28.0.3"
 },
 ...
 "scripts": {
 "test": "jest"
 }
}

Now run the tests, just to make sure you’ve got everything wired up correctly. If all goes well, you should see that the temporary spec we just added happily passes:

$ npm test

PASS ./index.spec.ts
 My speaking clock
 ✓ works

With that, it’s time to get to start writing some real tests.

Start with some failing tests

As devoted practitioners of test-driven development, we’re going to start by writing some unit tests — specifically, some failing unit tests that we can fix by writing the Pulumi code to make them pass.

Recall that our design requires just two cloud resources: a Lambda function and a Lambda function URL. For the function itself, we’ll use the high-level aws.lambda.CallbackFunction resource, one of my favorites for managing for Lambdas because it requires only one property: an inline JavaScript function to handle the event that triggers the Lambda.

For the URL resource — the eventual triggerer of that event — you’ll use an aws.lambda.FunctionURL configured to make the Lambda publicly accessible (i.e., available to anyone on the internet) and embeddable on any domain, making it easy, for example, to embed the URL as the src attribute of an HTML5 audio element.

So to get things going, we’ll need to:

Configure Pulumi to mock AWS resources. We’re writing unit tests, after all, and we want them to be fast, so we’ll need to prevent those tests from provisioning any real cloud infrastructure.
Import your Pulumi resource declarations — the function and the function URL — into index.spec.ts so you can reference them in tests.

In that light, replace the contents of index.spec.ts with the following code, which, after some setup to switch Pulumi into unit-testing mode, contains just one test: a check to make sure that the function URL is configured with the right authorizationType. I’ve included some comments to help explain what each line is doing:

import * as pulumi from "@pulumi/pulumi";
import "jest";

describe("My speaking clock", () => {

 // Define the infra variable as a type whose shape matches that of the
 // to-be-defined resources module.
 // https://www.typescriptlang.org/docs/handbook/2/typeof-types.html
 let infra: typeof import("./resources");

 beforeAll(() => {

 // Put Pulumi in unit-test mode, mocking all calls to cloud-provider APIs.
 pulumi.runtime.setMocks({

 // Mock requests to provision cloud resources and return a canned response.
 newResource: (args: pulumi.runtime.MockResourceArgs): {id: string, state: any} => {

 // Here, we're returning a same-shaped object for all resource types.
 // We could, however, use the arguments passed into this function to
 // customize the mocked-out properties of a particular resource based
 // on its type. See the unit-testing docs for details:
 // https://www.pulumi.com/docs/iac/concepts/testing/unit
 return {
 id: `${args.name}-id`,
 state: args.inputs,
 };
 },

 // Mock function calls and return whatever input properties were provided.
 call: (args: pulumi.runtime.MockCallArgs) => {
 return args.inputs;
 },
 });
 });

 beforeEach(async () => {

 // Dynamically import the resources module.
 // https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/import#dynamic_imports
 infra = await import("./resources");
 });

 describe("function URL", () => {

 it("is publicly accessible", (done) => {
 infra.timeURL.authorizationType.apply(authType => {
 try {
 expect(authType).toBe("NONE");
 done();
 } catch (err) {
 done(err);
 }
 });
 });
 });
});

We can’t quite run these tests just yet, though, as they rely on a module called resources that we haven’t yet defined. Let’s do that now by creating a new file, resources.ts, containing some stubs for the cloud resources we’ve elected to use:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

export const timeFunction = new aws.lambda.CallbackFunction("time-function", {
 callback: () => {
 // Leave the callback implementation empty for now.
 },
});

export const timeURL = new aws.lambda.FunctionUrl("time-url", {
 functionName: timeFunction.name,
 authorizationType: "", // Leave this empty, too. (We want our first test to fail.)
});

Now run the tests, and you should see that the public-accessibility test fails — which shouldn’t be surprising, as we haven’t yet supplied an acceptable value:

$ npm test

FAIL ./index.spec.ts
 My speaking clock
 function URL
 ✕ is publicly accessible

 ● My speaking clock › function URL › is publicly accessible

 expect(received).toBe(expected) // Object.is equality

 Expected: "NONE"
 Received: ""

Now let’s turn that sad-looking ❌ into a ✅.

Make the tests pass

Update the FunctionUrl resource to change its authorizationType input to NONE (meaning for anonymous access, no authorization required):

export const timeURL = new aws.lambda.FunctionUrl("time-url", {
 functionName: timeFunction.name,
- authorizationType: "",
+ authorizationType: "NONE",
});

Once you do that, you should be able to run the test suite again, and this time, see it pass:

$ npm test

PASS ./index.spec.ts
 My speaking clock
 function URL
 ✓ is publicly accessible

Excellent: you’re now well on your way to writing more tests with Jest — and we’ll do that in a moment. But first, let’s digress momentarily to explore a few other ways of writing Jest specs with Pulumi.

Tidy up the specs

If you’ve written JavaScript tests before, particularly with tools like Jest, Jasmine, and Mocha, there’s a good chance you’ve gotten used to writing those tests in a particular way. And if you’re like me, you might’ve raised a bit of an eyebrow when you saw how that first test was written. Here it is again, this time with comments that capture the questions I might’ve had myself if I were seeing this code for the first time:

it("is publicly accessible", (done) => {
 infra.timeURL.authorizationType.apply(auth => { // Why do I have to use .apply() here?
 try { // What happens if I don't try/catch?
 expect(auth).toBe("NONE"); // Is it possible to use expect() by itself?
 done(); // Do I have to call done()? What if I don't?
 } catch (err) { // How can I make these tests more readable?
 done(err); // Is all of this code necessary?
 }
 });
});

This is a perfectly valid test, of course; we know because we just ran it. Still, you might wonder, especially since we’re only checking one property, whether this test might be written a bit more concisely — something more like this, maybe:

it("is publicly accessible", () => {
 expect(infra.timeURL.authorizationType).toBe("NONE"); // Alas, nope.
});

The main reason is the asynchronous nature of the objects we’re testing. An aws.lambda.FunctionUrl, after all, isn’t just a plain ol’ JavaScript object — it’s a Pulumi CustomResource, a representation of an eventual resource running in the cloud, with properties that may not be known until sometime after the resource has been deployed. While it’s true we provided authorizationType as a regular JavaScript string, by the time we attempt to read it in the test, it’s been transformed into a pulumi.Output<string> — an asynchronous value, like a promise, that has to be unwrapped with a call to apply() before it can be read.

However, just adding apply and calling expect from within its callback doesn’t quite work, either:

it("is publicly accessible", () => {
 infra.timeURL.authorizationType.apply(authType => {
 expect(authType).toBe("NONE"); // Also nope.
 });
});

Although that might not be obvious at first, since on the surface, the test appears to pass:

$ npm test

PASS ./index.spec.ts
 My speaking clock
 function URL
 ✓ is publicly accessible

It isn’t until you force the test to fail that it becomes clearer something’s gone sideways. Try changing the expected authorizationType to something other than NONE and you’ll see what I mean:

$ npm test

PASS ./index.spec.ts
 My time app
 function URL
 ✓ is publicly accessible

Test Suites: 1 passed, 1 total
Tests: 1 passed, 1 total

[UnhandledPromiseRejection: This error originated either by throwing inside of an async function
without a catch block, or by rejecting a promise which was not handled with .catch(). The promise
rejected with the reason "Error: expect(received).toBe(expected) // Object.is equality

Expected: "AWS_IAM"
Received: "NONE"".] {
 code: 'ERR_UNHANDLED_REJECTION'
}

All is not lost, though. We do have other options, and that UnhandledPromiseRejection offers a bit of a clue.

Turn outputs into promises

Jest actually has built-in support for writing asynchronous tests with promises, and there are a couple of ways you can write specs that use them:

Have the function passed to it (or its alias, test) return a promise containing your expectation
Pass an async function to it and await the promise in the usual way

There’s only one problem: pulumi.Outputs may indeed be promise-like, but they aren’t actual promises.

They can, however, be wrapped in promises pretty easily. I like the way Pulumi community member @aviflax handled this with a little utility function that wraps the Output and preserves the type of its underlying value. Here’s that function, followed by a couple of examples that use it:

// Convert a pulumi.Output to a promise of the same type.
function promiseOf<T>(output: pulumi.Output<T>): Promise<T> {
 return new Promise(resolve => output.apply(resolve));
}

// Our test, rewritten to have it() return a Promise<string>.
it("is publicly accessible", () => {
 return promiseOf(infra.timeURL.authorizationType).then(authType => {
 expect(authType).toBe("NONE");
 });
});

// The same test, rewritten to use async/await.
it("is publicly accessible", async () => { // <-- Don't miss that async!
 const authType = await promiseOf(infra.timeURL.authorizationType);
 expect(authType).toBe("NONE");
});

Either approach works equally well, but I’m slightly more partial to the latter for its conciseness. So for now, update index.spec.ts to add the promiseOf helper and adjust the test accordingly, then run the suite one more time to make sure all’s well:

// ...

// Convert a pulumi.Output to a promise of the same type.
function promiseOf<T>(output: pulumi.Output<T>): Promise<T> {
 return new Promise(resolve => output.apply(resolve));
}

describe("My speaking clock", () => {
 // ...

 describe("function URL", () => {

 it("is publicly accessible", async () => {
 const authType = await promiseOf(infra.timeURL.authorizationType);
 expect(authType).toBe("NONE");
 });
 });
});

Assuming you’re back in ✅ territory, let’s move on so we can finish things off.

Finish the app

We’re in good shape: we’ve got scaffolding for our Jest-powered unit tests and we’re generally happy with how they look, so it’s time to round out the rest of the app and its tests so we can deploy. We’ll go through this last part briskly, as we’ve covered most of the important stuff at this point, and pause only briefly to point out what’s important as we go.

Here’s the plan:

We’ll add one more test to make sure the function URL is CORS-friendly
We’ll add another test to make sure the URL resource is bound to the right Lambda function
We’ll add the Lambda code to generate and return the audio file
We’ll deploy, and bask in the warm glow of nostalgia

Let’s get to it.

Add the remaining tests

To index.spec.ts, add two more tests, one to ensure the function is accessible to browsers running on any domain, the other to verify the function URL resource is exposing the right function (if we didn’t, it’d be easy to change that reference by mistake and accidentally expose the wrong function):

describe("My speaking clock", () => {
 // ...

 describe("function URL", () => {
 // ...

 it("is CORS-friendly", async () => {
 const authType = await promiseOf(infra.timeURL.cors);
 expect(authType).toEqual({
 allowOrigins: ["*"],
 allowMethods: ["GET"],
 });
 });

 it("is bound to the right Lambda function", async () => {
 const timeFuncName = await promiseOf(infra.timeFunction.name);
 const timeURLFunc = await promiseOf(infra.timeURL.functionName);
 expect(timeFuncName).toEqual(timeURLFunc);
 });
 });
});

Run the tests, and you should see one failure:

$ npm test

FAIL ./index.spec.ts (14.256 s)
 My speaking clock
 function URL
 ✓ is publicly accessible (3022 ms)
 ✕ is CORS-friendly (3 ms)
 ✓ is bound to the right Lambda function (1 ms)

 Expected: {"allowMethods": ["GET"], "allowOrigins": ["*"]}
 Received: undefined

Tests: 1 failed, 2 passed, 3 total

A quick addition to the FunctionUrl in resources.ts should get you to green:

// ...
export const timeURL = new aws.lambda.FunctionUrl("time-url", {
 functionName: timeFunction.name,
 authorizationType: "NONE",
+ cors: {
+ allowOrigins: ["*"],
+ allowMethods: ["GET"],
+ },
});

$ npm test

PASS ./index.spec.ts
 My speaking clock
 function URL
 ✓ is publicly accessible
 ✓ is CORS-friendly
 ✓ is bound to the right Lambda function

Tests: 3 passed, 3 total

That’s it for the tests! Now for the final additions.

Produce the audio message

The last thing we’ll need to do is render some audio. For this, we’ll need to communicate with Google Translate, our text-to-speech provider of choice, and that means we’ll need an HTTP client. I generally like node-fetch, so let’s go with that:

npm install --save node-fetch@2

Now, finish the program by importing node-fetch into resources.ts, adding a function to produce a speech-friendly string indicating the current time and updating the CallbackFunction’s callback property to send the string to Google Translate to produce the desired result. Here’s the complete file, with comments, for reference:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

const fetch = require("node-fetch");

// Convert the current time into a speech-friendly string.
export function getSpeechText() {
 const now = new Date();
 now.setSeconds(now.getSeconds() + 6);

 const local = new Date(now.toLocaleString("en-US", { timeZone: "America/Los_Angeles" }));
 const localHour = local.getHours();
 const localMinute = local.getMinutes();
 const h = localHour > 12 ? localHour - 12 : localHour;
 const m = localMinute < 10 ? `oh ${localMinute}` : localMinute; // So 2:03 -> "two-oh-three"
 const s = local.getSeconds();

 return `At the tone, the time will be ${h} ${m}. And ${s} seconds.`;
}

export const timeFunction = new aws.lambda.CallbackFunction("time-function", {

 // Update the Lambda callback body to convert the current time into an MP3 file.
 callback: async () => {
 const text = getSpeechText();
 const speechURL = `https://translate.google.com/translate_tts?ie=UTF-8&q=${encodeURIComponent(text)}&textLen=${text.length}&tl=en&client=tw-ob`;
 const beepURL = "https://www.pulumi.com/uploads/beep.mp3";
 const ttsResponse = await fetch(speechURL);
 const beepResponse = await fetch(beepURL);
 const speech = await ttsResponse.arrayBuffer();
 const beep = await beepResponse.arrayBuffer();

 // Tack a beep onto the end of the audio returned from Google Translate, then
 // render the whole thing as a base-64 encoded string.
 const body = Buffer.concat([Buffer.from(speech), Buffer.from(beep)]).toString("base64");

 // Return an appropriately shaped HTTP response.
 return {
 body,
 headers: { "Content-Type": "audio/mpeg" },
 isBase64Encoded: true,
 statusCode: 200,
 };
 },
});

export const timeURL = new aws.lambda.FunctionUrl("time-url", {
 functionName: timeFunction.name,
 authorizationType: "NONE",
 cors: {
 allowOrigins: ["*"],
 allowMethods: ["GET"],
 },
});

// Export the public URL of our shiny new service.
export const audioURL = timeURL.functionUrl;

Finally, open index.ts (which should still be empty) and add a couple of lines to import the resources module and export the function URL as a Pulumi stack output:

import "./resources";

export { audioURL } from "./resources";

And that’s it. We’re ready to deploy.

Deploy the service

Doublecheck your AWS credentials are configured, then deploy the new service with a single pulumi up:

$ pulumi up

...
Updating (dev)

View Live: https://app.pulumi.com/cnunciato/audichron-2022/dev/updates/1

 Type Name Status
 + pulumi:pulumi:Stack audichron-2022-dev created
 + ├─ aws:iam:Role time-function created
 + ├─ aws:lambda:Function time-function created
 + ├─ aws:iam:RolePolicyAttachment time-function-7cd09230 created
 + ├─ aws:iam:RolePolicyAttachment time-function-a1de8170 created
 + ├─ aws:iam:RolePolicyAttachment time-function-1b4caae3 created
 + ├─ aws:iam:RolePolicyAttachment time-function-019020e7 created
 + ├─ aws:iam:RolePolicyAttachment time-function-4aaabb8e created
 + ├─ aws:iam:RolePolicyAttachment time-function-74d12784 created
 + ├─ aws:iam:RolePolicyAttachment time-function-e1a3786d created
 + ├─ aws:iam:RolePolicyAttachment time-function-6c156834 created
 + ├─ aws:iam:RolePolicyAttachment time-function-b5aeb6b6 created
 + └─ aws:lambda:FunctionUrl time-url created

Outputs:
 audioURL: "https://52rjmmybwfwcud3nxxhosznpum0cycbw.lambda-url.us-west-2.on.aws/"

Resources:
 + 13 created

Duration: 20s

When the update completes, open the URL in your browser of choice, and you should hear the message you’ve been waiting for — one that might even make Pat Fleet proud:

$ open $(pulumi stack output audioURL)

When you’re ready, you can tear everything down with a pulumi destroy.

Wrapping up

Beyond just having a nifty (and admittedly rather silly) new way to tell time, you should have a much better sense at this point of how you can use Pulumi with Jest to write better, safer infrastructure code.

From here, there’s a bunch more you might think about next: writing more tests to cover the code we just added, exploring some additional flavors of testing in the docs, or having a look at a few examples. You’ll find the full source for this walkthrough up on GitHub as well.

Happy testing!

API Gateway to EventBridge with Pulumi

Christian Nunciato — Wed, 12 Jan 2022 13:00:00 -0700

If you’re familiar with Amazon API Gateway, you know it’s all about making it easier to provision and manage a web API. Maybe you’ve used it, as I have, with Crosswalk, our AWS extension library, to stand up a REST API and handle requests with AWS Lambda functions:

import * as awsx from "@pulumi/awsx";

// Create a new API Gateway instance.
const api = new awsx.apigateway.API("my-api", {
 routes: [
 {
 // Define an HTTP endpoint.
 method: "GET",
 path: "/things",

 // Handle requests with an AWS Lambda function.
 eventHandler: async (apiGatewayEvent) => {
 return {
 statusCode: 200,
 body: JSON.stringify(["thingOne", "thingTwo"]),
 };
 },
 },
 ],
});

// Export the API's public URL. 🎉
export const apiUrl = api.url;

I love this abstraction, and I use it all the time; it’s an incredibly convenient way to solve a really common problem. But if you examine the code, you’ll notice that it’s making a fairly strong assumption about how you’ll be handling HTTP requests—namely, that you’ll do so with a single Lambda function, and that that function will always return a JavaScript object of a particular shape.

Indeed, this arrangement is the API contract of a Lambda proxy integration—API Gateway integrations come in many shapes and sizes; Lambda integrations just happen to be one of the more popular—and much of the time, an approach like this one works out just fine. But depending on the needs of the application, it might not always be the best fit.

Imagine you were building a print-on-demand service, for example, and you wanted to expose an API to let your customers upload documents and have them converted into orders. On AWS, you might reach for API Gateway, as above, to define an HTTP method and route (POST /uploads, say), wire it up to an AWS Lambda, and have the Lambda parse the upload, write the order to a database, and return a response. Visually, such a design might look something like this:

It’d definitely work, and again, it’s quite common. But at some point, you might find this tight coupling between API Gateway and Lambda too limiting. Say you wanted to be notified whenever a new order was received, with a Slack message, maybe, in one of your team’s shared workspace channels. Under the current design, you’d probably add a few lines of code to the Lambda function to import an HTTP library and make a call to the Slack API to post the message:

That’d work, too—but it’d be less than ideal for a number of reasons. For one, that Lambda would now have two jobs: taking orders and sending Slack notifications. That might be fine for today (it’s only a few lines of code, after all), but over time, those two jobs could easily become three, and then four, and soon enough, that poor Lambda could wind up becoming a lot more difficult to maintain. And given the importance of its main job—capturing orders—it’s not something you’d want to risk failing at runtime because of a random Slack outage or other transient internet mishap. Moreover, with every bit of extra work you tack on to that function, you take a tiny step closer to hitting API Gateway’s 30-second limit.

What you really need, then, is to be able to take multiple independent, possibly long-running actions based on a single API Gateway request. And one way to do that is with Amazon EventBridge.

Hello, EventBridge

Amazon EventBridge (formerly CloudWatch Events) is as a serverless event bus whose job is to receive structured event data—from your own applications, from other AWS Services—and use that data to notify other applications or services using event-handling rules that you specify. With EventBridge, you can build loosely coupled, event-driven systems that take advantage of AWS’s rich support for serverless architectures and that scale gracefully as the needs of those systems change over time.

For this particular application, EventBridge would let you address the multiple-handler problem in a more scalable and easily maintainable way. Rather than have API Gateway invoke Lambda directly, leaving one Lambda responsible for handling multiple tasks, you could have API Gateway publish to EventBridge instead, and let EventBridge invoke as many Lambdas (or other AWS services) as you like, all serverlessly and in parallel—and of course, all easily managed with Pulumi.

Let’s see how. Here’s a revised architecture diagram showing how you might approach building an application like this one with EventBridge positioned between API Gateway and Lambda:

Now let’s have a look at what it’d be like to build it with Pulumi. We won’t build everything in this diagram—things like writing to the database or messaging Slack are left for you to explore—but we will build enough to give you clear picture and a working example of how to connect all of these parts into a working application. Specifically:

an API Gateway instance to act as a container for your public API, along with a stage and a route to handle inbound HTTP requests;
an EventBridge integration (comprised of an event bus and event rule) to handle notifications from API Gateway; and finally,
one or more Lambda functions to be invoked in response to event-rule matches.

Let’s get started.

Create a new project and stack

As always, it’s good practice to begin with a new project and stack:

$ pulumi new aws-typescript

$ pulumi new aws-python

Make sure you configure your AWS credentials as well, and when prompted, choose whatever stack name and AWS region work best for you.

In this post, we’ll be using API Gateway V2 as it’s newer and has a few features (like auto-deploy) that make it easier to work with in this context. The same ideas apply to both V1 and V2, though, so while V2 still lacks full feature parity with V1, the underlying resources we’ll be using—methods, routes, integrations—are the same. You’ll find links to examples of both at the end of this post.

Create the API gateway and stage

Start by replacing the contents of index.js index.ts __main__.py main.go Program.cs Program.fs Program.vb App.java Pulumi.yaml main.tf with the following code to declare a new API Gateway API:

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

// Create an HTTP API.
const api = new aws.apigatewayv2.Api("api", {
 protocolType: "HTTP",
});

import json
import pulumi
import pulumi_aws as aws

# Create an HTTP API.
api = aws.apigatewayv2.Api("example",
 protocol_type="HTTP"
)

Next, add a stage (feel free to name it whatever you like; I usually use the current stack name for convenience), and set it to deploy automatically whenever a change is made to the API:

// ...

// Create a stage and set it to deploy automatically.
const stage = new aws.apigatewayv2.Stage("stage", {
 apiId: api.id,
 name: pulumi.getStack(),
 autoDeploy: true,
});

# Create a stage and set it to deploy automatically.
stage = aws.apigatewayv2.Stage("stage",
 api_id=api.id,
 name=pulumi.get_stack(),
 auto_deploy=True
);

The next thing to do is register a route on the gateway to give your users a publicly accessible endpoint to upload to. But in order to do that, you’ll need to tell API Gateway what to do when an upload happens. Since the plan of record is to notify EventBridge (using API Gateway’s built-in support for it), you’ll need to declare a few EventBridge things first.

Add an event bus and event rule

Every AWS account gets an event bus by default (one that’s aptly named default), but given how easy it is to create one, we might as well do that for this application. You’ll also need to define an event rule—a resource that watches a specific event bus for events that conform to a particular pattern or shape, and then routes those events to one or more targets (e.g., Lambda functions). Add the following lines to your program for both:

// ...

// Create an event bus.
const bus = new aws.cloudwatch.EventBus("bus");

// Create an event rule to watch for events.
const rule = new aws.cloudwatch.EventRule("rule", {
 eventBusName: bus.name,

 // Specify the event pattern to watch for.
 eventPattern: JSON.stringify({
 source: ["my-event-source"],
 }),
});

# ...

# Create an event bus.
bus = aws.cloudwatch.EventBus("bus")

# Create an event rule to watch for events.
rule = aws.cloudwatch.EventRule("rule",
 event_bus_name=bus.name,
 event_pattern=json.dumps({"source": ["my-event-source"]})
)

The most noteworthy property of the EventRule resource is probably the eventPattern. EventBridge events all conform to a certain schema, and in this case, we’re expressing that this particular event rule should take action on any event that originates from my-event-source. (The source property is just a free-form string that by convention identifies the application or service responsible for the event.)

With the event bus and event rule in place, you’re ready to define the integration itself—the resource responsible for connecting the gateway route (which we’ll get back to shortly) to your newly created event bus. As I mentioned earlier, there are several types of API Gateway integration to choose from, each one suited to a particular purpose. For this example, the AWS_PROXY type is good fit, as it’s simple and requires very little code; it doesn’t give you quite as much control over the API Gateway response as you might like—as a proxy, it just returns to the caller whatever is returned by the backend, in this case EventBridge—but it’s more than enough for the task at hand.

Add the following lines for the integration and route. The comments should explain what each block is doing:

// ...

// Define a policy granting API Gateway permission to publish to EventBridge.
const apiGatewayRole = new aws.iam.Role("api-gateway-role", {
 assumeRolePolicy: {
 Version: "2012-10-17",
 Statement: [
 {
 Action: "sts:AssumeRole",
 Effect: "Allow",
 Principal: {
 Service: "apigateway.amazonaws.com",
 },
 },
 ],
 },
 managedPolicyArns: ["arn:aws:iam::aws:policy/AmazonEventBridgeFullAccess"],
});

// Create an API Gateway integration to forward requests to EventBridge.
const integration = new aws.apigatewayv2.Integration("integration", {
 apiId: api.id,

 // The integration type and subtype.
 integrationType: "AWS_PROXY",
 integrationSubtype: "EventBridge-PutEvents",
 credentialsArn: apiGatewayRole.arn,

 // The body of the request to be sent to EventBridge. Note the
 // event source matches pattern defined on the EventRule, and the
 // Detail expression, which just forwards the body of the original
 // API Gateway request (i.e., the uploaded document).
 requestParameters: {
 EventBusName: bus.name,
 Source: "my-event-source",
 DetailType: "my-detail-type",
 Detail: "$request.body",
 },
});

// Finally, define the route.
const route = new aws.apigatewayv2.Route("route", {
 apiId: api.id,
 routeKey: "POST /uploads",
 target: pulumi.interpolate`integrations/${integration.id}`,
});

# ...

# Define a policy granting API Gateway permission to publish to EventBridge.
api_gateway_role = aws.iam.Role("api-gateway-role",
 assume_role_policy=json.dumps({
 "Version": "2012-10-17",
 "Statement": [
 {
 "Action": "sts:AssumeRole",
 "Effect": "Allow",
 "Principal": {
 "Service": "apigateway.amazonaws.com",
 },
 },
 ],
 }),
 managed_policy_arns=[
 "arn:aws:iam::aws:policy/AmazonEventBridgeFullAccess",
 ],
);

# Create an API Gateway integration to forward requests to EventBridge.
integration = aws.apigatewayv2.Integration("integration",
 api_id=api.id,

 # The integration type and subtype.
 integration_type="AWS_PROXY",
 integration_subtype="EventBridge-PutEvents",
 credentials_arn=api_gateway_role.arn,

 # The body of the request to be sent to EventBridge. Note the
 # event source matches pattern defined on the EventRule, and the
 # Detail expression, which just forwards the body of the original
 # API Gateway request (i.e., the uploaded document).
 request_parameters={
 "EventBusName": bus.name.apply(lambda name: name),
 "Source": "my-event-source",
 "DetailType": "my-detail-type",
 "Detail": "$request.body",
 },
);

# Finally, define the route.
route = aws.apigatewayv2.Route("route",
 api_id=api.id,
 route_key="POST /uploads",
 target=integration.id.apply(lambda id: f"integrations/{id}"),
);

With that, you’re ready to configure the Lambda.

Add a Lambda function handler

The hard part’s done: you’ve declared an API and route, mapped that route to an integration, configured the integration to publish events to an event bus, and defined an event rule to respond to those events. All that’s left now is to tell the rule how to respond.

So to finish things off, you need:

a Lambda function to handle uploads,
an EventBridge target to bind your event rule to that function, and
an IAM policy granting EventBridge permission to invoke the function.

Add the following lines to your program to complete it:

First, make a new folder and file alongside your Pulumi program, at ./api/handlers.py, and add a Python function to act as your function handler:

// ...

// Create a Lambda function handler with permission to log to CloudWatch.
const lambda = new aws.lambda.CallbackFunction("lambda", {
 policies: [aws.iam.ManagedPolicies.CloudWatchLogsFullAccess],
 callback: async (event: any) => {
 // For now, just log the event, including the uploaded document.
 // That'll be enough to verify everything's working.
 console.log({ source: event.source, detail: event.detail });
 },
});

// Create an EventBridge target associating the event rule with the function.
const lambdaTarget = new aws.cloudwatch.EventTarget("lambda-target", {
 arn: lambda.arn,
 rule: rule.name,
 eventBusName: bus.name,
});

// Give EventBridge permission to invoke the function.
const lambdaPermission = new aws.lambda.Permission("lambda-permission", {
 action: "lambda:InvokeFunction",
 principal: "events.amazonaws.com",
 function: lambda.arn,
 sourceArn: rule.arn,
});

// Export the API Gateway URL to give us something to POST to.
export const url = pulumi.interpolate`${api.apiEndpoint}/${stage.name}`;

# In ./api/handlers.py:
import json

def capture_order(event, context):
 # For now, just log the event, including the uploaded document.
 # That'll be enough to verify everything's working.
 print(json.dumps({ "source": event["source"], "detail": event["detail"] }))

Now, back in index.js index.ts main.py main.go Program.cs Program.fs Program.vb App.java Pulumi.yaml main.tf , add the following lines to your program to complete it:

# ...

# Define a role and policy allowing Lambda functions to log to CloudWatch.
lambda_role = aws.iam.Role("lambda-role",
 assume_role_policy=json.dumps({
 "Version": "2012-10-17",
 "Statement": [
 {
 "Action": "sts:AssumeRole",
 "Principal": {
 "Service": "lambda.amazonaws.com"
 },
 "Effect": "Allow"
 }
 ]
 })
)
lambda_role_policy = aws.iam.RolePolicy("lambda-role-policy",
 role=lambda_role.id,
 policy=json.dumps({
 "Version": "2012-10-17",
 "Statement": [{
 "Effect": "Allow",
 "Action": [
 "logs:CreateLogGroup",
 "logs:CreateLogStream",
 "logs:PutLogEvents"
 ],
 "Resource": "arn:aws:logs:*:*:*"
 }]
 })
)

# Create a Lambda function handler.
lambda_function = aws.lambda_.Function("lambda",
 role=lambda_role.arn,
 runtime="python3.7",
 handler="handlers.capture_order",
 code=pulumi.AssetArchive({
 ".": pulumi.FileArchive('./api')
 })
)

# Create an EventBridge target associating the event rule with the function.
lambda_target = aws.cloudwatch.EventTarget("lambda-target",
 arn=lambda_function.arn,
 rule=rule.name,
 event_bus_name=bus.name,
);

# Give EventBridge permission to invoke the function.
lambda_permission = aws.lambda_.Permission("lambda-permission",
 action="lambda:InvokeFunction",
 principal="events.amazonaws.com",
 function=lambda_function.arn,
 source_arn=rule.arn,
);

# Export the API Gateway URL to give us something to POST to.
pulumi.export("url", pulumi.Output.concat(api.api_endpoint, "/", stage.name))

All Together, Now

Now that the program’s complete, you can run Pulumi to bring it to life:

$ pulumi up
...

Updating (dev)
...

 Type Name Status
 + pulumi:pulumi:Stack eventbridge-v2-dev created
 + ├─ aws:apigatewayv2:Api api created
 + ├─ aws:apigatewayv2:Stage stage created
 + ├─ aws:cloudwatch:EventBus bus created
 ...

Outputs:
 apiURL: "https://geqfietbcl.execute-api.us-west-2.amazonaws.com/dev"

Resources:
 + 15 created

Duration: 31s

When the update finishes, you’ll have a fully functioning API Gateway-EventBridge integration that you can verify with curl:

$ curl --data '{"some-key": "some-value"}' --header "Content-Type: application/json" \
 "$(pulumi stack output url)/uploads"

{"Entries":[{"EventId":"cdc44763-6976-286c-9378-7cce674dff81"}],"FailedEntryCount":0}

Note the response, which comes directly from EventBridge (courtesy of the AWS_PROXY integration), confirming the event was received and written to the event bus.

The final step is to confirm that the request made it all the way to Lambda, which you can easily do by tailing its output with pulumi logs:

$ pulumi logs --follow

Collecting logs for stack dev since 2022-01-06T16:18:48.000-08:00.
...

{
 source: 'my-event-source',
 detail: { 'some-key': 'some-value' }
}

When you’re happy, be sure to tidy up with a pulumi destroy.

What’s next?

There’s a lot more you can do with integrations like this that we didn’t cover: add more Lambda function handlers, have EventBridge target other AWS services (Step Functions might be a good one to try next), validate HTTP request bodies (with API Gateway models, to keep bad data from ever reaching EventBridge), and more. Hopefully this gives you sense of what’s possible, though. And as promised, you’ll find examples that use both versions of API Gateway in our examples repository on GitHub:

API Gateway V2 to EventBridge in TypeScript
API Gateway V2 to EventBridge in Python
API Gateway V1 to EventBridge in TypeScript, with request validation and custom HTTP response mapping

Top 5 Things an Azure Developer Needs to Know: Serverless

Sophia Parafina — Fri, 06 Aug 2021 00:00:00 +0000

The previous article was a deep dive into virtual machines. First, we used the Azure Portal to create and deploy a virtual machine; then, we repeated the process using infrastructure as code. We further demonstrated how to automate provisioning as part of cloud engineering’s build and deploy processes.

This article will explore the other end of the cloud infrastructure with serverless, which is an on-demand, fully-managed cloud architecture.

What is serverless, and when to use it?

Serverless is an event-driven architecture that executes code when triggered. Azure implements serverless with Azure Functions and Events. A function typically performs a single task, such as saving a log file. A function can perform multiple tasks, but in general, functions adhere to the principle of “you only had one job.” Functions can be written in any Azure supported language: C #, F#, JavaScript, PowerShell, Python, and Typescript.

An Event calls a function to execute. It can be a webhook, an HTTP request, or a scheduled trigger. Azure provides many function templates with a range of triggers; furthermore, they can be customized to application requirements.

Azure Functions are ideal for specific use cases that include:

when demand varies unpredictably, serverless can scale automatically
when a service is used infrequently, serverless can reduce costs by only charging per use
serverless can eliminate maintenance costs
when you have client-heavy applications, where most of the business logic is in the client
if you have high latency tasks like data processing

Serverless is not ideal if the application runs constantly, needs a quick response (otherwise known as the cold start problem, or is a long-running process that can exceed the timeout period for your execution plan.

With those caveats, let’s jump into creating a serverless function using the Azure Portal to understand how to write a function and how the pieces work together.

Our first serverless function

For this example, we’ll use Visual Studio Code and the Azure Functions extension.

Step 1: With the Azure Functions extension installed, click on the Activity Bar and click on Create New Project.

Select a folder for your project.
Select a language, use Typescript.
Select HTTP trigger.
Name your function, e.g., myHttpTrigger.
Set the Authorization level to Anonymous

When finished, you’ll have a template.

import { AzureFunction, Context, HttpRequest } from "@azure/functions"

const httpTrigger: AzureFunction = async function (context: Context, req: HttpRequest): Promise<void> {
 context.log('HTTP trigger function processed a request.');
 const name = (req.query.name || (req.body && req.body.name));
 const responseMessage = name
 ? "Hello, " + name + ". This HTTP triggered function executed successfully."
 : "This HTTP triggered function executed successfully. Pass a name in the query string or in the request body for a personalized response.";

 context.res = {
 // status: 200, /* Defaults to 200 */
 body: responseMessage
 };

};

export default httpTrigger;

Step 2: (Optional) If you have Azure Functions Core tools installed, you can run the function locally. Press F5 or click on Run > Start debugging.

The service will run locally and provide you with a URL:

Functions:

 myHttpTrigger: [GET,POST] http://localhost:7071/api/myHttpTrigger

Navigate to the URL in a browser or use curl.

$ curl http://localhost:7071/api/myHttpTrigger
This HTTP triggered function executed successfully. Pass a name in the query string or in the request body for a personalized response.

Press Control-C in the terminal to stop the debugger.

Step 3: Publish the function to Azure.

Make sure you are signed into Azure and click on Deploy function app…

Select Create a new Function App in Azure and enter a name, “exampleHttpTrigger”
Select a runtime stack, Node.js 14 LTS
Select a location for new resources, East US

In the terminal pane, click on the Output tab, and you’ll see the URL for the function. You can call the function with curl with the name parameter.

$ curl https://myazurehttptrigger.azurewebsites.net/api/myhttptrigger?name=Pulumipus
Hello, Pulumipus. This HTTP triggered function executed successfully.

Don’t forget to delete the function in the Azure Portal!

Deploying Azure Functions with code

Creating and deploying an Azure Function is a bit more involved than using the Azure Function extension with VS Code. What goes on behind the scenes after you write the function is a process that includes:

Creating a ResourceGroup

// Create a separate resource group for this example.
const resourceGroup = new resources.ResourceGroup("functions-rg");

Creating the storage account and code container that holds the function

// Storage account is required by Function App.
// Also, we will upload the function code to the same storage account.
const storageAccount = new storage.StorageAccount("sa", {
 resourceGroupName: resourceGroup.name,
 sku: {
 name: storage.SkuName.Standard_LRS,
 },
 kind: storage.Kind.StorageV2,
});

// Function code archives will be stored in this container.
const codeContainer = new storage.BlobContainer("zips", {
 resourceGroupName: resourceGroup.name,
 accountName: storageAccount.name,
});

Uploading the function code as a zip file to the storage account. Note that the source parameter points to the function’s directory.

// Upload Azure Function's code as a zip archive to the storage account.
const codeBlob = new storage.Blob("zip", {
 resourceGroupName: resourceGroup.name,
 accountName: storageAccount.name,
 containerName: codeContainer.name,
 source: new pulumi.asset.FileArchive("./javascript"),
});

Setting the Consumption Plan to determine how to bill when the function executes

// Define a Consumption Plan for the Function App.
// You can change the SKU to Premium or App Service Plan if needed.
const plan = new web.AppServicePlan("plan", {
 resourceGroupName: resourceGroup.name,
 sku: {
 name: "Y1",
 tier: "Dynamic",
 },
});

Set the connection string to the storage account and the zip archive’s Shared Access Signature (SAS) to grant rights to use the storage account. We will need these to create the function in Azure.

// Build the connection string and zip archive's SAS URL. They will go to Function App's settings.
const storageConnectionString = getConnectionString(resourceGroup.name, storageAccount.name);
const codeBlobUrl = signedBlobReadUrl(codeBlob, codeContainer, storageAccount, resourceGroup);

Now that we have all the parameters needed to deploy the function: the resource group, the billing plan, the connection string to Azure storage, the Functions version (3), the runtime (Node.js), the runtime Node version (14.0), and the SAS url for the function code, we can build and deploy our Azure Function.

const app = new web.WebApp("fa", {
 resourceGroupName: resourceGroup.name,
 serverFarmId: plan.id,
 kind: "functionapp",
 siteConfig: {
 appSettings: [
 { name: "AzureWebJobsStorage", value: storageConnectionString },
 { name: "FUNCTIONS_EXTENSION_VERSION", value: "~3" },
 { name: "FUNCTIONS_WORKER_RUNTIME", value: "node" },
 { name: "WEBSITE_NODE_DEFAULT_VERSION", value: "~14" },
 { name: "WEBSITE_RUN_FROM_PACKAGE", value: codeBlobUrl },
 ],
 http20Enabled: true,
 nodeVersion: "~14",
 },
});

export const endpoint = pulumi.interpolate`https://${app.defaultHostName}/api/HelloNode?name=Pulumi`;

At the very end of the script, we export the URL to the service for you to trigger.

Why is it so complicated?

Isn’t infrastructure as code supposed to simplify infrastructure deployments? This seems more effort than using the Visual Studio Code extension or even the Azure CLI on the surface. However, it’s important to note that this is boiler-plate code that you can reuse to deploy multiple functions with different runtimes and Consumption Plans. The key to using infrastructure code is that it is repeatable, customizable, and a CI/CD pipeline can deploy changes automatically.

Try out the complete example in either Typescript or C# to get familiar with Azure Functions. You can also learn how Functions are written in all supported languages by checking out the Azure Functions in All Supported Languages example.

In the following article, we’ll do a deep dive into Azure static websites, which is a popular option for Jamstacks.

Top 5 Things an Azure Developer Needs to Know: Introduction

Sophia Parafina — Mon, 02 Aug 2021 00:00:00 +0000

The Azure cloud platform includes over 200 products and cloud services. Wherever you are in your Microsoft cloud engineering journey, you should be familiar with these top 5 cloud tasks that are essential building blocks commonly used to deploy applications and infrastructure to the Azure cloud.

In this series of articles, we’ll go in-depth on virtual machines, Azure Functions, static websites, building an Azure Kubernetes Service cluster and deploying applications on AKS, and DevOps with Azure App Service.

Azure - a comprehensive cloud platform

With Azure, you can develop, test, deploy and manage your applications in an integrated cloud computing environment. With the Azure Portal, you can manage the cloud infrastructure for your applications. Azure gives you the choice of a portal for managing services or managing them programmatically using service APIs and templates.

With over 200 services, knowing which one to use can be confusing. Deciding where to host your application is the first step. This series of articles covers the most commonly used Azure services such as virtual machines, serverless functions, static websites, Kubernetes, and Azure App Service. In addition, we’ll cover how to deploy applications using Azure DevOps.

In this article, we’ll match use cases to cloud services to help you navigate Azure.

Azure Virtual Machines

A common cloud migration path is to use virtual machines if you have existing infrastructure or on-premise applications. It is especially true if you want complete control over the infrastructure and application environment.

Azure provides Infrastructure as a Service (IaaS) for deploying your applications on either Linux or Windows VMs. IaaS includes all necessary support, such as Azure Virtual Network and storage. As with any IaaS solution, you are responsible for the installation of software, configuration, and maintenance of your application and any maintenance the VM might require, such as operating system patches.

If your application doesn’t fit into a Platform as a Service (PaaS) model, consider using virtual machines to provide the fine-grained control required for your deployment.

Serverless

At the opposite end of cloud infrastructure is serverless computing. Serverless eliminates the need to manage infrastructure and lets you focus on writing and deploying code. Azure automatically provisions, scales, and manages the infrastructure. You can write code that runs in response to events or on a schedule.

Azure Functions lets you focus on the application’s business logic, and the code executes when called by a webhook, an HTTP or REST request, a schedule, or an event. Serverless is particularly appealing because you can use the development language of your choice, ranging from PHP, Python, Node.js, or .NET languages. Azure functions scale as needed, and consumption-based pricing means that you’re charged only when the code runs.

If you don’t need a completely hosted project and only pay for what you use, choose Azure Functions triggered by web-based events, a schedule, or other Azure services.

Static websites

Static websites are quickly becoming the defacto way to publish content. Built from libraries and frameworks that don’t use server side rendering like React, Angular, or Vue, they include HTML, CSS, image assets, and Javascript in the application. Instead of serving assets from a server or servers scaled horizontally, the static assets are distributed through a Content Delivery Network (CDN). This reduces latency, resulting in faster websites and increased customer satisfaction.

Azure provides two ways to deploy static websites. Azure lets you deploy a static web application with Azure App Service and the recently released Azure Static Web Apps. Azure App Service deploys websites by uploading the website to Azure storage configured to serve web content. Azure App Service automatically builds and deploys full stack web apps to Azure from a code repository, which follows the Jamstack model for developing and deploying websites.

We’ll look at both methods and discuss the pros and cons of both methods.

Kubernetes

Kubernetes is the leading container orchestration service. Azure Kubernetes Service (AKS) is a hosted service that can deploy, scale, and manage containerized applications in a cluster. AKS fulfills many different use cases, for example:

If you have an existing application, think n-tier applications, you can containerize the components and “lift and shift” your application to run in Kubernetes.
You can deploy microservices on AKS to add horizontal scaling, load balancing, secrets management, and self-healing.
Data scientists are training machine learning models on AKS using TensorFlow and Kubeflow.
You can even use AKS as part of a build pipeline by pairing with a CI, such as Jenkins, to build containers for your application.

AKS supports many use cases by providing provisioning, scaling, and resource upgrades without requiring expertise in container orchestration. With AKS, the Kubernetes master nodes are managed by Azure, and you are only charged for the worker nodes in your cluster.

We’ll do two separate deep dives into deploying AKS clusters and deploying applications in AKS in this series.

Deploy AKS

Azure provides many ways to deploy an AKS cluster, including:

When you deploy an AKS Cluster, the worker nodes and Kubernetes master are configured with a basic configuration. You can additionally configure networking, storage, monitoring, and policies. We’ll cover configuring AKS in this series.

Deploy Applications

Deploying applications on AKS is equally flexible, and you can deploy using:

kubectl and YAML manifests
Helm, an open-source Kubernetes packaging tool
Azure Pipelines Infrastructure as Code

We’ll examine these options and do a deep dive on application deployment in a follow-up article.

DevOps integration

Before Continuous integration (CI) and Continuous Deployment (CD) became prevalent, applications were manually built, compiled, and maintained in a large codebase. CI/CD automation lets you build quickly, test, and deploy services. Both application fixes and new features are shipped quickly to your customers, and you can rapidly respond to changing customer demands. For example, if demand increases, you can quickly add resources to your application or add a serverless function that increases and decreases resources as needed.

If you’re deploying applications on Azure Functions, virtual machines, AKS, or provisioning infrastructure, the build process should be automated. Azure offers a range of services such as Azure Container Registry, AKS, and Azure Cosmos DB that can integrate with popular CI/CD pipelines like Azure DevOps, Jenkins, Github, and TeamCity.

If you’re interested in using Azure DevOps to automate infrastructure and application delivery, stay tuned!

Let’s Get Started!

So here are the top 5 things essential to know as an Azure developer:

configuring and deploying virtual machines
developing and deploying Azure Functions
deploying static websites on Azure
configuring Azure Kubernetes Service and deploying containerized applications
using Azure DevOps to automate the building and deployment of infrastructure and applications

First up and next in this series of articles is deploying Azure virtual machines with an application.

How Webiny Built a Serverless Application Framework

Adrian Smijulj — Thu, 03 Jun 2021 00:00:00 +0000

Building an open-source framework for building serverless applications has many challenges, one of which is deploying cloud infrastructure resources. In this article, learn how Webiny uses Pulumi to enable its users to easily deploy and develop applications built on top of serverless cloud technologies.

What is Webiny?

Webiny is an open-source framework for building serverless applications, completely written in TypeScript.

In other words, it is a framework that enables developers to architect, build, and deploy solutions on top of the serverless cloud infrastructure, like AWS Lambda, Amazon DynamoDB, Amazon S3, and others. Some examples of what you can build with Webiny today are:

full-stack applications
websites
REST and GraphQL APIs
microservices

With the framework, Webiny also built a content management system called the Webiny Serverless CMS. It is free, open-source, and by default, included with every new Webiny project.

Finally, it’s worth mentioning that, at the moment, Webiny exclusively works with AWS. But, bringing Webiny to other cloud providers like Microsoft Azure and Google Cloud Platform (GCP) is definitely something we’ll be looking at in the future.

Building the Framework - The Challenge

Building a serverless framework certainly has its challenges. One of which is the deployment of cloud infrastructure, which, in the world of serverless, is one of the fundamental operations developers need to do, even while the application is still in development.

Before the version 5 release, Webiny relied on an infrastructure provisioning technology called Serverless Components (not to be confused with Serverless Framework). Using a concept of components, it is a system that enables developers to deploy ready-made blocks of cloud infrastructure resources configured for a specific use-case. Some examples of components are Website, REST API, a Full Stack Application, and more.

At the time, it seemed that the overall developer experience and the simplicity of the Serverless Components product was the right choice for our users. So, we decided to go with it.

But, over time, we noticed several things that, unfortunately, made us question our decision. And if I were to choose the top four, I’d mention the following.

1. Hard to Customize

While the idea around components for different use-cases certainly sounded interesting, ultimately, it was not ideal for Webiny. We frequently received questions regarding further component configuration and customization, which was not easy to perform.

“How do I configure a different database?”, “How do I set a VPC?”, “How do I set up a specific configuration parameter for my S3 bucket?” were just some of the questions we received. The answer almost always included us having to implement support for a particular feature. To some degree, this worked, but it was really us patching the holes, which we knew would only get us so far.

2. Using YAML Instead of Code

At Webiny, we believe cloud engineering and infrastructure-as-code is the future, so, naturally, we wanted our users to use familiar code (ideally TypeScript) and development tools to define their cloud infrastructure. Within Serverless Components, the components are configured via YAML files, of which we were never really big fans, simply because of the fact that writing code instead of configurations gives more flexibility to developers.

3. Vendor Lock-In

Vendor lock-in was introduced at a later stage of the Serverless Components product development. Essentially, to deploy a component, the user is now forced to use a proprietary service that comes with it. And while, among other things, the service enables much faster deployments, from our perspective, we saw this as an additional point of friction for our users. Ideally, users should be able to set up Webiny with only an AWS account.

4. Lack of Support for Other Cloud Providers

At the time, Serverless Components’ support for different cloud providers just wasn’t there. This would be a major roadblock for us as we get to multi-cloud integration.

Discovering Pulumi

Once we realized Serverless Components wasn’t an ideal solution for Webiny, we started looking for an alternative.

The key features that we were looking for were the following:

open-source
use code (preferably TypeScript) to create cloud infrastructure resources
cloud infrastructure code should be flexible, meaning, users should be able to make adjustments to it
support for multiple cloud providers is a must
no vendor lock-in

So, the search began, and soon enough, we discovered Pulumi. And as it turned out, it not only checked all the boxes we had initially listed but much more:

different options when it comes to storing cloud infrastructure state files: a managed SaaS (pulumi.com) with a console and self-hosted, (for example Amazon S3)
ability to deploy cloud infrastructure into multiple environments using its concept of stacks
advanced features like Policy as Code and watch mode
great documentation
a vibrant community of developers and a responsive team behind the product

I probably forgot to mention a thing or two here, but most certainly, at this point, we were ready to make a move and start experimenting with Pulumi.

Integrating Pulumi with Webiny

For us, the integration of Pulumi with Webiny consisted of three steps:

figure out how to integrate Pulumi’s programming model with Webiny
integrate Pulumi CLI into Webiny CLI
figure out the optimal way of handling cloud infrastructure state files

Let’s see how we tackled each of these three steps.

Integrating Pulumi’s Programming Model With Webiny

In terms of project organization, every Webiny project consists of two key concepts: packages and project applications (or just applications).

Packages are just regular NPM packages, or in other words, folders with a package.json manifest file and some code. On the other hand, project applications are higher-level organizational units formed from one or more packages that form applications, as the name itself suggests. Applications consist of both application code and cloud infrastructure needed to run them.

A single Webiny project can contain multiple project applications, where each one has its own independent set of necessary cloud infrastructure resources that need to be deployed.

As an example, a default Webiny project includes three project applications:

API - essentially, represents your GraphQL HTTP API
Admin Area - the Admin Area (React) application
Website - the public website, a (React) application with static site generation (SSG) in the cloud

If we were to compare these by the complexity of the necessary cloud infrastructure to deploy, the Admin Area is the simplest. It only relies on a single Amazon S3 bucket and an Amazon Cloudfront distribution. On the other hand, the API project application is the most complex one as it needs to deploy multiple AWS Lambda functions, Amazon DynamoDB tables, Amazon S3 buckets, and more.

Ultimately, we decided that each project application should be a Pulumi project. With this approach, we gave developers the ability to both define and deploy respective cloud infrastructures independently. And, with the concept of stacks, they are also able to deploy them into multiple environments, which we’ll show in a moment.

Pulumi CLI and Webiny CLI

Once we understood how to use Pulumi concepts with Webiny’s project organization, the next step was integrating the Pulumi CLI with the Webiny CLI.

And although the Pulumi CLI is great, we still wanted to keep it super simple for the user and make the overall developer experience as straightforward and unified as possible. For starters, we didn’t want our users to install the Pulumi CLI manually. We wanted it to happen automatically.

We’ve created our version of the Pulumi SDK, which lets us use the Pulumi CLI more programmatically. It also enables us to make the Pulumi CLI download experience as smooth as possible. Essentially, whenever a user runs a deployment-related command, all of the necessary Pulumi CLI binaries and plugins are downloaded and stored inside the project’s node_modules folder.

Although we could’ve saved us some time by using Pulumi’s Automation API (instead of creating the mentioned Pulumi SDK), at the time, the Automation API was still in preview and not generally available. And since the setup we already had was working well, we decided to keep it and hopefully revisit the Automation API integration in the future.

Once we had that in place, we were ready for the next step, exposing a couple of fundamental deployment-related commands via the Webiny CLI. The following examples show us some of the commands users can use:

# Deploys a project application (internally runs "pulumi up")
webiny deploy {PROJECT_APPLICATION_FOLDER} --env {ENVIRONMENT}

# Previews a project application deployment (internally runs "pulumi preview")
webiny deploy {PROJECT_APPLICATION_FOLDER} --env dev --{ENVIRONMENT}

# Destroys cloud infrastructure resources previously deployed within
# the specified project application (internally runs "pulumi destroy")
webiny destroy {PROJECT_APPLICATION_FOLDER} --env {ENVIRONMENT}

# Starts watching specified project application and continuously deploys
# changes to the cloud (internally runs "pulumi watch")
webiny watch {PROJECT_APPLICATION_FOLDER} --env {ENVIRONMENT}

Note the --env argument appended to each command. With it and Pulumi’s concept of stacks, users can easily deploy their project applications into multiple environments.

Cloud Infrastructure State Files

The last piece of the puzzle was storing cloud infrastructure state files. Here we went with the following approach.

For local development, users’ cloud infrastructure state files are stored locally within their Webiny project using the Local Filesystem Backend, which we’ve seen worked great for developers.

On the other hand, for ephemeral environments spawned in CI/CD or long-lived environments like staging or production, through our documentation, we advise our users to use centralized and remote storage by using backends like Amazon S3 and even Pulumi Cloud (pulumi.com). Both backends have their pros and cons, and we let the users choose the one they want to use.

Show Me the Code

Let’s take a look at how it all works in the actual code.

As mentioned, by default, every Webiny project comes with three project applications: API, Admin Area, and Website, which are located in the api, apps/admin, and apps/website folders, respectively:

.
├── api # API
│ ├── code
│ ├── pulumi
│ ├── Pulumi.yaml
│ ├── tsconfig.json
│ └── webiny.application.ts
├── apps
│ ├── admin # Admin Area
│ │ ├── code
│ │ ├── pulumi
│ │ ├── Pulumi.yaml
│ │ ├── tsconfig.json
│ │ └── webiny.application.ts
│ └── website # Website
│ ├── code
│ ├── pulumi
│ ├── Pulumi.yaml
│ ├── tsconfig.json
│ └── webiny.application.ts
└── (...)

As we can see, every project application follows the same general organization. The two folders in each project are:

code - contains application code (one or more packages)
pulumi - contains cloud infrastructure (Pulumi) code

Furthermore, if we opened each of these pulumi folders, we’d see different cloud infrastructure resources clearly defined via multiple TypeScript classes. As a simple example, if we were to open the apps/admin/pulumi (Admin Area) folder, we’d find the following three files:

app.ts - deploys an Amazon S3 bucket that hosts the Admin Area (React) application
cloudfront.ts - deploys an Amazon CloudFront distribution for improved availability
index.ts - the Pulumi entrypoint file, imports classes defined within separate files

This code organization makes it much easier for developers to grasp the overall cloud infrastructure. It also makes it easier for them to adjust the code to their needs eventually.

A more complex example is the cloud infrastructure code in the API project application (api/pulumi), which deploys many different cloud infrastructure resources, like AWS Lambda functions, DynamoDB tables, ElasticSearch clusters, VPCs, and more.

But I will leave this up to you to check out, as pasting multiple chunks of code here might not be that productive.

Instead, I wanted to focus on a couple of other interesting and useful features that Pulumi and the infrastructure-as-code approach enabled us to create.

API Project Application - Different `dev` and `prod` Stacks

As mentioned, the API project application contains many different cloud infrastructure resources. But, as it turns out, when it comes to active development, we don’t have to unleash the full potential of the cloud, like we’re doing in staging and production environments.

For example, it probably makes no sense to deploy an Amazon ElasticSearch cluster into multiple availability zones (AZs) in most cases. A single AZ is enough for development purposes.

Another good example are VPCs and potentially NAT Gateways. Again, for development purposes, in most cases, we could probably get away without these too.

These extra resources make it slower to perform deployments from developers’ machines and incur extra costs for an organization because developers are deploying more resources, some of which are charged hourly.

So, with that in mind, we split the API project application’s cloud infrastructure into two stacks - the dev and prod. And, as you might’ve already guessed, only the prod variant will deploy absolutely all necessary cloud infrastructure resources.

What is even more interesting is the fact that this can be achieved with a simple if statement, which we placed in the index.ts entrypoint file:

// (...)

export = async () => {
 // (...)

 // WEBINY_ENV represents the environment which was passed upon running
 // the `webiny deploy` command. This is inserted automatically by the
 // Webiny CLI.
 if (process.env.WEBINY_ENV === "prod" || process.env.WEBINY_ENV === "staging") {
 // Import "prod" resources config and initialize resources.
 return await import("./prod").then(module => module.default());
 }

 // Import "dev" resources config and initialize resources.
 return await import("./dev").then(module => module.default());
};

Automatic Tagging of Cloud Infrastructure Resources

Another useful feature is the automatic tagging of the deployed cloud infrastructure resources. In other words, every taggable cloud infrastructure resource will be tagged with WbyProjectName and WbyEnvironment tags. For developers, this makes it much easier to see all of the deployed resources within their Webiny project.

We created a tagResources function, which essentially registers a global stack transformation via pulumi.runtime.registerStackTransformation function to achieve this.

It is also applied in the same index.ts entrypoint file we saw in the previous section:

import { tagResources } from "@webiny/cli-plugin-deploy-pulumi/utils";

export = async () => {
 // Add tags to all resources that support tagging.
 tagResources({
 WbyProjectName: process.env.WEBINY_PROJECT_NAME as string,
 WbyEnvironment: process.env.WEBINY_ENV as string
 });

 // (...)
};

Protect Feature

Finally, to protect our users from accidental deletions of mission-critical cloud infrastructure resources, we’ve used Pulumi’s protect feature:

The protect option marks a resource as protected. A protected resource cannot be deleted directly. Instead, you must first set protect: false and run pulumi up. Then you can delete the resource by removing the line of code or by running pulumi destroy. The default is to inherit this value from the parent resource and false for resources without a parent.

So, for Webiny users, we’ve made sure that this feature is automatically enabled for resources like DynamoDB tables, Cognito User Pools, Elasticsearch Cluster, and similar.

Future Plans

With the Webiny v5, we managed to introduce many significant improvements to both the Webiny Serverless Application Framework and Webiny Serverless CMS. However, we still have a lot of new cool things to unravel.

Webiny aims to be a multi-cloud compatible solution, and this is something Pulumi will significantly assist us with because deploying to multiple cloud providers is supported.

With that, we would also like to create various project templates, which would, for example, enable users to quickly start working on a new GraphQL or REST HTTP API, a simple React application, a microservice, and more. All of these require specific cloud infrastructure resources to be deployed, and again, this is where Pulumi will make our users’ lives much easier.

Finally, I already mentioned the Automation API as something that we might look into in the future, as it seems it’s something that can be seamlessly integrated with Webiny.

Conclusion

For Webiny, choosing Pulumi as the default infrastructure as code framework was certainly a big leap in the right direction.

It enabled us to leave the complexities of cloud infrastructure deployments to a product and company whose primary focus is exactly that. And this is what ultimately helped us completely focus on what we care about the most - creating an awesome serverless application framework.

Upcoming Workshops and Events

Sophia Parafina — Tue, 12 Jan 2021 00:00:00 +0000

It’s a new year and it’s time to level up your cloud engineering skills. Pulumi is there to get you started on your cloud engineering journey with workshops and technical sessions.

CI/CD Pipelines for Kubernetes Apps with Pulumi & Codefresh

To kickoff the year we start with a technical session that demonstrates how to continuously deliver both applications and provision infrastructure using Codefresh CI/CD. This session will demonstrate:

The basics of CI/CD using Codefresh.
How to declare cloud resources using Pulumi and your favorite programming languages.
How to define a simple pipeline for Kubernetes deployments.
Practical tests to ensure infrastructure reliability as part of your pipeline.

Taught by Pulumi and Codefresh engineers, you’ll receive a thorough understanding of CI/CD and infrastructure as code best practices.

Intro to Pulumi

If you’re new to infrastructure as code, this workshop is for you. You will learn the fundamentals of Infrastructure as Code through a series of guided exercises using Pulumi’s Cloud Engineering platform. The workshop covers setting up Pulumi, an infrastructure as code platform that uses programming languages to provision modern cloud infrastructure.

This workshop is designed to help users completely new to Pulumi become familiar with the core concepts needed to be effective with the Pulumi Infrastructure as Code platform. We will guide through the Pulumi platform with diagrams and a series of hands on exercises to help you understand the building blocks available in Pulumi.

The workshop will cover the following topics.

What is Pulumi?
How to install Pulumi
Configuring your development environment
Choose your Pulumi language
An introduction to the Pulumi state & the Pulumi Service
An introduction to Pulumi config
Storing sensitive data in Pulumi securely
Provision your first resources
Understanding stacks
An introduction to the Pulumi programming model

We have multiple workshops scheduled from January to March. Register for one today.

Jan 26: Intro to Pulumi, 9 AM PST Feb 9: Intro to Pulumi, 9 AM PST Feb 23: Intro to Pulumi, 9 AM PST March 9: Intro to Pulumi, 9 AM PST

Deploying Microservices with Pulumi and AWS Lambda

This serverless workshop demonstrates how Pulumi can accelerate provisioning of cloud infrastructure. The workshop shows how to build a set of microservices utilizing AWS’s newest Lambda features. Attendees will be guided through the process of provisioning a set of Lambda resources in AWS and see real time examples of how Pulumi’s innovative programming model helps turbocharge cloud engineering.

If you’re an active Pulumi user and building serverless applications this hands-on workshop is for you. You will learn:

How to provision a Lambda function in AWS.
How to provision infrastructure such as DynamoDB & API Gateway to support your microservice applications.
How to take advantage of the newest Lambda features.

Building a Kubernetes Platform in Amazon EKS with Pulumi

Kubernetes is one of the major cloud technologies and in this hands-on workshop, we’ll examine how Pulumi works with Kubernetes through real-world examples of managing Amazon EKS clusters. We’ll focus on building a user-friendly Kubernetes platform, installing software that makes Kubernetes easy to use for application developers.

Attendees will be guided through the process of provisioning an Amazon EKS cluster and installing platform friendly software such as the aws load balancer controller. We’ll cover the following:

How-to provision a production-ready Amazon EKS cluster with key features enabled using Pulumi
Provisioning the aws-load-balancer-controller to automate ingress creation
Install an example application to show the end-to-end user experience for users.

Get a solid foundation for provisioning Kubernetes clusters and applications by signing up.

Running Container Images in AWS Lambda

Mikhail Shilkov — Tue, 01 Dec 2020 00:00:00 +0000

TL;DR — To run a container image in AWS Lambda, build an OCI image that implements the Lambda Runtime API (most teams start from an AWS-provided base image), push it to Amazon ECR, and create a Lambda function with packageType: "Image" pointing at the image URI. Lambda containers support images up to 10 GB, up to 10 GB of memory, up to 10 GB of /tmp ephemeral storage, and a 15-minute execution ceiling. Pulumi automates the build, push, and function wiring in a single program. Pick Lambda containers when your workload is event-driven and bursty but your dependencies (binaries, ML models, system libraries) outgrow the 250 MB ZIP limit; pick AWS Fargate or ECS when you need long-running tasks, persistent connections, or multi-container pods.

How do I run a container image in AWS Lambda?

You package your function as a Linux container image, push it to Amazon ECR, and create a Lambda function whose packageType is Image. The image must implement the Lambda Runtime API, which is an HTTP protocol Lambda uses to deliver invocation events and collect responses. The simplest path is to start FROM an AWS-provided base image (public.ecr.aws/lambda/nodejs:22, public.ecr.aws/lambda/python:3.13, public.ecr.aws/lambda/java:21, and so on) because they ship with the Runtime Interface Client preinstalled. If you want to use a different OS or runtime, add the open-source Runtime Interface Client yourself.

With Pulumi, one program covers the entire path: ECR repository, Docker build and push, IAM role, Lambda function, and event source. It works in TypeScript, Python, Go, .NET, Java, or YAML. The rest of this post walks through that program with a real video-thumbnailer example.

import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";

const repo = new awsx.ecr.Repository("repo", {
 forceDelete: true,
});

const image = new awsx.ecr.Image("image", {
 repositoryUrl: repo.url,
 context: "./app",
 platform: "linux/amd64",
});

const role = new aws.iam.Role("thumbnailerRole", {
 assumeRolePolicy: aws.iam.getPolicyDocument({
 statements: [{
 actions: ["sts:AssumeRole"],
 principals: [{ type: "Service", identifiers: ["lambda.amazonaws.com"] }],
 }],
 }).then(doc => doc.json),
});

new aws.iam.RolePolicyAttachment("lambdaExecute", {
 role: role.name,
 policyArn: aws.iam.ManagedPolicy.AWSLambdaExecute,
});

const thumbnailer = new aws.lambda.Function("thumbnailer", {
 packageType: "Image",
 imageUri: image.imageUri,
 role: role.arn,
 timeout: 900,
 memorySize: 2048,
});

import pulumi_aws as aws
import pulumi_awsx as awsx

repo = awsx.ecr.Repository("repo", force_delete=True)

image = awsx.ecr.Image(
 "image",
 repository_url=repo.url,
 context="./app",
 platform="linux/amd64",
)

role = aws.iam.Role(
 "thumbnailerRole",
 assume_role_policy=aws.iam.get_policy_document(statements=[{
 "actions": ["sts:AssumeRole"],
 "principals": [{"type": "Service", "identifiers": ["lambda.amazonaws.com"]}],
 }]).json,
)

aws.iam.RolePolicyAttachment(
 "lambdaExecute",
 role=role.name,
 policy_arn=aws.iam.ManagedPolicy.AWS_LAMBDA_EXECUTE,
)

thumbnailer = aws.lambda_.Function(
 "thumbnailer",
 package_type="Image",
 image_uri=image.image_uri,
 role=role.arn,
 timeout=900,
 memory_size=2048,
)

pulumi up builds the image from ./app, pushes it to a fresh ECR repository, and deploys the Lambda — in one step, with one command, with no shell scripts.

What is a Lambda container image?

A Lambda container image is an OCI (or Docker v2) image whose entrypoint speaks the Lambda Runtime API. The image is stored in Amazon ECR, and Lambda pulls it on cold start. Functionally, a container-image function behaves the same as a ZIP-package function: it is invoked by the same triggers (S3, SQS, API Gateway, EventBridge, and so on), it bills the same way (per request and per GB-second), and it is bound by the same 15-minute execution ceiling. The only meaningful difference is how the code is packaged and what limits apply to that package.

Lambda container images were announced at re:Invent 2020 and have been generally available ever since. They are not a separate service — they are a deployment format for the same Lambda runtime.

What are the AWS Lambda container limits in 2026?

The hard limits below apply to every Lambda function regardless of packaging, except for the image-size column, which is specific to container images.

Limit	Container image	ZIP package
Maximum package size	10 GB (uncompressed)	250 MB (unzipped, including layers)
Maximum memory	10,240 MB	10,240 MB
Maximum ephemeral `/tmp`	10,240 MB	10,240 MB
Maximum execution timeout	15 minutes	15 minutes
Maximum payload (sync)	6 MB request / 6 MB response	6 MB request / 6 MB response
Architectures	`x86_64`, `arm64`	`x86_64`, `arm64`
SnapStart support	Not supported for container images	Java, Python, .NET

Sources: Lambda quotas, Lambda images, SnapStart documentation.

A note on cold starts: the public guidance from AWS is that container-image cold starts are typically comparable to ZIP cold starts when the image is well-optimized — AWS caches image layers and uses on-demand block-level loading so only the bytes touched at startup are downloaded. The biggest lever you have is image size: keep the image small, put rarely changed dependencies in lower layers, and prefer a minimal base. SnapStart, which is the most effective cold-start mitigation for Java, Python, and .NET ZIP functions, does not apply to container-image functions; if you need SnapStart, ship a ZIP.

How much do Lambda container images cost?

Container-image functions are billed identically to ZIP functions:

Requests: $0.20 per 1M requests (x86_64 and arm64 alike).
Duration: $0.0000166667 per GB-second on x86_64, $0.0000133334 on arm64 (Graviton2) — roughly 20% lower per GB-second, with up to 34% better price-performance per AWS.
Free tier: 1M requests and 400,000 GB-seconds per month, every month.
Storage: ECR charges $0.10 per GB-month for image storage; data transfer to Lambda within the same Region is free.

Always confirm rates against the live Lambda pricing and ECR pricing pages, since AWS adjusts them periodically.

How do Lambda containers compare to Fargate, ECS, and ZIP Lambdas?

The four options below all run containers, but they sit on a spectrum from “fully event-driven, scale to zero” to “long-running, always on.”

Dimension	Lambda (ZIP)	Lambda (container image)	AWS Fargate	Amazon ECS (EC2)
Packaging	ZIP up to 250 MB	OCI image up to 10 GB	OCI image (no Lambda runtime API needed)	OCI image
Cold start	Milliseconds to seconds; SnapStart available	Seconds (image-size dependent); no SnapStart	Tens of seconds for new tasks	Tens of seconds (capacity dependent)
Max execution time	15 minutes	15 minutes	Unbounded	Unbounded
Scale to zero	Yes	Yes	No (minimum task count)	No (minimum container instances)
Billing	Per-ms duration + per-request	Per-ms duration + per-request	Per-second of vCPU and memory	Per-EC2-instance-hour + per-task overhead
Networking	Optional VPC attachment	Optional VPC attachment	First-class VPC, ENIs, persistent connections	First-class VPC, ENIs, persistent connections
Native event triggers	100+ AWS services	100+ AWS services	EventBridge / Step Functions integrations	EventBridge / Step Functions integrations
Best for	Short, bursty, event-driven workloads	Same as ZIP, but with heavy dependencies, native binaries, large ML models	Long-running services, daemons, persistent workloads	Workloads needing custom EC2 sizing or GPU

The decision tree is short: if your workload runs for less than 15 minutes per invocation and you can fit it in 10 GB, Lambda is almost always cheaper and operationally simpler than Fargate. If you cannot fit in a 250 MB ZIP, use a Lambda container image. If your workload is long-running or needs persistent network connections, choose Fargate or ECS.

How do I deploy a Lambda container image with Pulumi?

Let’s walk through a complete program: a video thumbnailer that runs every time a .mp4 is uploaded to an S3 bucket. The function uses FFmpeg, a binary that is impractical to ship in a 250 MB ZIP, to extract a thumbnail and write it back to the bucket. The full source is in pulumi/examples.

Quick start

Bootstrap a project: pulumi new https://github.com/pulumi/examples/tree/master/aws-ts-lambda-thumbnailer
Edit ./app/Dockerfile and ./app/index.js with your function’s logic.
Deploy: pulumi up.

1. Define a Dockerfile

The container image is based on the AWS-provided Node.js 22 image, which preinstalls the Runtime Interface Client. It also installs the AWS CLI and FFmpeg, copies the handler, and points to index.handler as the entrypoint.

FROM public.ecr.aws/lambda/nodejs:22

RUN dnf install -y tar xz unzip \
 && curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" \
 && unzip awscliv2.zip && ./aws/install \
 && curl -O https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz \
 && tar -xf ffmpeg-release-amd64-static.tar.xz \
 && mv ffmpeg-*-amd64-static/ffmpeg /usr/local/bin/ \
 && rm -rf awscliv2.zip aws ffmpeg-*-amd64-static*

COPY index.js ${LAMBDA_TASK_ROOT}
CMD ["index.handler"]

2. Create an S3 bucket for inputs and outputs

import * as aws from "@pulumi/aws";

const bucket = new aws.s3.Bucket("bucket");

import pulumi_aws as aws

bucket = aws.s3.Bucket("bucket")

3. Build the image and push it to ECR

Pulumi Crosswalk for AWS wraps the ECR repository, the Docker build, and the push into two resources:

import * as awsx from "@pulumi/awsx";

const repo = new awsx.ecr.Repository("repo", {
 forceDelete: true,
});

const image = new awsx.ecr.Image("image", {
 repositoryUrl: repo.url,
 context: "./app",
 platform: "linux/amd64",
});

import pulumi_awsx as awsx

repo = awsx.ecr.Repository("repo", force_delete=True)

image = awsx.ecr.Image(
 "image",
 repository_url=repo.url,
 context="./app",
 platform="linux/amd64",
)

The ./app directory holds the Dockerfile and the handler. platform: "linux/amd64" is explicit so the build is reproducible from Apple Silicon and Graviton developer machines. Switch to linux/arm64 if you want the cheaper Graviton runtime.

4. Set up the IAM role

const role = new aws.iam.Role("thumbnailerRole", {
 assumeRolePolicy: aws.iam.getPolicyDocument({
 statements: [{
 actions: ["sts:AssumeRole"],
 principals: [{ type: "Service", identifiers: ["lambda.amazonaws.com"] }],
 }],
 }).then(doc => doc.json),
});

new aws.iam.RolePolicyAttachment("lambdaExecute", {
 role: role.name,
 policyArn: aws.iam.ManagedPolicy.AWSLambdaExecute,
});

role = aws.iam.Role(
 "thumbnailerRole",
 assume_role_policy=aws.iam.get_policy_document(statements=[{
 "actions": ["sts:AssumeRole"],
 "principals": [{"type": "Service", "identifiers": ["lambda.amazonaws.com"]}],
 }]).json,
)

aws.iam.RolePolicyAttachment(
 "lambdaExecute",
 role=role.name,
 policy_arn=aws.iam.ManagedPolicy.AWS_LAMBDA_EXECUTE,
)

AWSLambdaExecute is the AWS-managed policy that grants the read/write S3 access and CloudWatch Logs access this function needs. For a real production workload, scope this down to a custom policy that names the bucket explicitly.

5. Configure the Lambda function

const thumbnailer = new aws.lambda.Function("thumbnailer", {
 packageType: "Image",
 imageUri: image.imageUri,
 role: role.arn,
 timeout: 900,
 memorySize: 2048,
 architectures: ["x86_64"],
});

thumbnailer = aws.lambda_.Function(
 "thumbnailer",
 package_type="Image",
 image_uri=image.image_uri,
 role=role.arn,
 timeout=900,
 memory_size=2048,
 architectures=["x86_64"],
)

packageType: "Image" is the only setting that distinguishes a container function from a ZIP function. The imageUri is the digest-pinned URI of the image we just pushed, so any change to the Dockerfile or handler triggers a redeploy on the next pulumi up.

6. Wire up the S3 trigger

new aws.s3.BucketNotification("onNewVideo", {
 bucket: bucket.id,
 lambdaFunctions: [{
 lambdaFunctionArn: thumbnailer.arn,
 events: ["s3:ObjectCreated:*"],
 filterSuffix: ".mp4",
 }],
});

new aws.lambda.Permission("allowBucket", {
 action: "lambda:InvokeFunction",
 function: thumbnailer.name,
 principal: "s3.amazonaws.com",
 sourceArn: bucket.arn,
});

aws.s3.BucketNotification(
 "onNewVideo",
 bucket=bucket.id,
 lambda_functions=[{
 "lambda_function_arn": thumbnailer.arn,
 "events": ["s3:ObjectCreated:*"],
 "filter_suffix": ".mp4",
 }],
)

aws.lambda_.Permission(
 "allowBucket",
 action="lambda:InvokeFunction",
 function=thumbnailer.name,
 principal="s3.amazonaws.com",
 source_arn=bucket.arn,
)

That is the entire program. pulumi up provisions everything in the right order, and a subsequent edit to the Dockerfile rebuilds and rolls out a new image automatically.

Deploy Lambda containers with Pulumi

Build the image, push it to Amazon ECR, wire up the IAM role, and deploy the function from one program in TypeScript, Python, Go, or the language you already use.

Get started

When should I choose Lambda containers over ZIP packages?

Reach for a container image when one or more of the following is true:

Your dependencies exceed 250 MB unzipped. Native binaries (FFmpeg, ImageMagick, headless Chromium), large ML model weights, and full Python data-science stacks all blow past the ZIP limit quickly.
You need a custom OS, runtime, or system library. Lambda’s managed runtimes are good but opinionated. A container lets you pin a specific glibc version, install OS packages, or run a language Lambda doesn’t natively support.
Your team’s CI/CD already builds containers. Reusing your existing image-build pipeline (and its scanning, signing, and SBOM tooling) is often easier than maintaining a parallel ZIP-packaging pipeline.
You want a single artifact format across Lambda, Fargate, and ECS. A container image that runs in Lambda for short bursts can also run in Fargate for long jobs — convenient for hybrid workloads.

Stick with ZIP when your code is small, you want SnapStart, or you want the fastest possible cold starts.

You might also like:

Conclusion

AWS Lambda’s container-image support brings the industry-standard packaging format to event-driven serverless functions, and Pulumi covers the full deploy (ECR repository, image build, IAM role, function, and event source) in one program in the language you already use. If you have outgrown the ZIP package, this is the pattern you want.

Watch a demo of the thumbnailer below.

Next steps:

Browse the full Lambda + container image example in pulumi/examples.
Read the Pulumi AWS Lambda guide for event sources, IAM patterns, and packaging tips.
See AWS Lambda SnapStart with Pulumi if you need faster cold starts on a ZIP-packaged Java, Python, or .NET function.
Cut x86 costs with Lambda functions powered by Graviton2.
Get started with Pulumi for AWS today.

Deploy AWS Lambda with Elastic File System using Pulumi

Luke Hoban — Tue, 16 Jun 2020 00:00:00 +0000

Some of the code in this post is out of date. See the AWS guides for an updated overview and examples.

Ever since AWS Lambda was released in 2015, users have wanted persistent file storage beyond the small 512MB /tmp disk allocated to each Lambda function. The following year, Amazon launched EFS, offering a simple managed file system service for AWS, but initially only available to mount onto Amazon EC2 instances. Over the last few months, AWS has been extending access to EFS to all of the modern compute offerings. First EKS for Kubernetes, then ECS and Fargate for containers. Today, AWS announced that EFS is now also supported in Lambda, providing easy access to network file systems from your serverless functions.

In this post, we’ll show you how to get started using EFS with Lambda and Pulumi.

Why would you want to use EFS from Lambda? There are a few common reasons:

You need more local disk than the 512MB of /tmp space available to a single Lambda Function.
You want to share persistent storage across various functions and would prefer to use file system storage instead of blob storage or a database.
You want to share persistent storage between Lambda and other compute (like containers or ECS), and would prefer to use file storage over blob storage or a database.

Storing data to the file system is particularly natural for the kind of small functions common in Lambda - no need to use the AWS SDK - just call fs.writeFileSync() (or equivalent for your language of choice).

EFS + Lambda Concepts

To use an EFS file system from Lambda, we need a few things:

A VPC
An EFS File System in the VPC
EFS mount points in each of the subnets of our VPC
An EFS Access Point specifying how Lambda should mount the EFS volume
An AWS Lambda Function running in the same VPC and private subnets and referencing the EFS Access Point

Let’s walk through how to build the required infrastructure. We’ll use Pulumi to provision the necessary resources. You can check out the full source code in the Pulumi Examples.

Because EFS is tied to a VPC, we can only access it from a Lambda running inside a VPC. We should create a VPC with both public and private subnets to ensure the Lambda function can reach the internet (since Lambda functions inside VPCs are not assigned public IPs). We could use an existing VPC or create a new one for our application.

const vpc = new awsx.ec2.Vpc("vpc", { subnets: [{ type: "private" }, { type: "public" }] });
const subnetIds = await vpc.publicSubnetIds;

An EFS FileSystem is a regional service, but accessing it requires adding mount points to a VPC. We can add those to the private subnets where our Functions will run. Each mount target has a security group, and we can use the default security group to allow access only from other compute resources in the same security group in our VPC. We could also use an existing FileSystem and mount targets if they have already been provisioned for another application.

const filesystem = new aws.efs.FileSystem("filesystem");
const targets = [];
for (let i = 0; i < subnetIds.length; i++) {
 targets.push(new aws.efs.MountTarget(`fs-mount-${i}`, {
 fileSystemId: filesystem.id,
 subnetId: subnetIds[i],
 securityGroups: [vpc.vpc.defaultSecurityGroupId],
 }));
}

To use EFS from our Lambda Function, we need to tell Lambda how to mount the NFS volume. Specifically, we must specify the Unix user and group to use to enable access, and the location and permission scheme to use as the root folder of the mount. This is done using an EFS AccessPoint.

const ap = new aws.efs.AccessPoint("ap", {
 fileSystemId: filesystem.id,
 posixUser: { uid: 1000, gid: 1000 },
 rootDirectory: { path: "/www", creationInfo: { ownerGid: 1000, ownerUid: 1000, permissions: "755" } },
}, { dependsOn: targets });

Finally, each Lambda Function we want to run will need to be in our VPC, use our default security group, and mount the EFS file system via our AccessPoint. We can write a small function that creates Lambda Functions with this configuration and we can specify where we want to mount this into the Lambda Function, though we must mount it underneath /mnt/.

function efsvpcCallback(name: string, f: aws.lambda.Callback<awsx.apigateway.Request, awsx.apigateway.Response>) {
 return new aws.lambda.CallbackFunction(name, {
 policies: [aws.iam.ManagedPolicy.AWSLambdaVPCAccessExecutionRole, aws.iam.ManagedPolicy.LambdaFullAccess],
 vpcConfig: {
 subnetIds: vpc.privateSubnetIds,
 securityGroupIds: [vpc.vpc.defaultSecurityGroupId],
 },
 fileSystemConfig: { arn: ap.arn, localMountPath: "/mnt/storage" },
 callback: f,
 });
}

We now have our infrastructure all set up to start using Lambda with EFS. Let’s build an app!

Interact with EFS from an HTTP API

For a first simple Lambda application, we’ll expose an HTTP API that GETs files from, and POSTs files to, our EFS file system. We’ll implement this with API Gateway and two routes. Using the helper we defined above, each of these is just a few lines of code.

Note that we can use our normal Node.js fs.readFileSync and fs.writeFileSync APIs to interact with the EFS file system - no need for an AWS SDK to talk to EFS. This makes it easy to take existing software and libraries that can interact with the filesystem and run them inside Lambda.

const api = new awsx.apigateway.API("api", {
 routes: [
 {
 method: "GET", path: "/files/{filename+}", eventHandler: efsvpcCallback("getHandler", async (ev, ctx) => {
 try {
 const f = "/mnt/storage/" + ev.pathParameters!.filename;
 const data = fs.readFileSync(f)
 return {
 statusCode: 200,
 body: data.toString(),
 };
 } catch {
 return { statusCode: 500, body: "" }
 }
 }),
 },
 {
 method: "POST", path: "/files/{filename+}", eventHandler: efsvpcCallback("uploadHandler", async (ev, ctx) => {
 try {
 const f = "/mnt/storage/" + ev.pathParameters!.filename;
 const data = new Buffer(ev.body!, 'base64');
 fs.writeFileSync(f, data)
 return {
 statusCode: 200,
 body: "",
 };
 } catch {
 return { statusCode: 500, body: "" }
 }
 }),
 },
 ]
});

Once this is deployed, we can interact with the API to add files to our file system:

$ curl -X POST -d 'Hello world' $(pulumi stack output url)files/file.txt
$ curl -X GET $(pulumi stack output url)files/file.txt
Hello, world

Easy as that - our two Lambda functions are sharing data via EFS!

We can get more insight into what the file system looks like inside Lambda by exposing a REST endpoint to run commands inside the a Lambda function. This will allow us to run curl to run commands to populate and read the contents of the file system.

...
 {
 method: "POST", path: "/", eventHandler: efsvpcCallback("postHandler", async (ev, ctx) => {
 const cmd = new Buffer(ev.body!, 'base64').toString()
 const buf = cp.execSync(cmd);
 return {
 statusCode: 200,
 body: buf.toString(),
 };
 }),
 },
...

We can now interact with our EFS file system by just running shell commands. Since it’s a normal file system mount, all our standard Unix tools work as we expect. Note that the files are owned by uid/gid 1000/1000, and have the 755 permissions specified by our Access Point.

$ curl -X POST -d 'ls -la /mnt/storage' $(pulumi stack output url)
total 8
drwxr-xr-x 2 1000 1000 6144 Jun 10 05:37 .
drwxr-xr-x 3 root root 4096 Jun 10 05:35 ..
-rw-rw-r-- 1 1000 1000 21 Jun 10 05:37 file.txt

$ curl -X POST -d 'echo "<h1>Hello World</h1>" > /mnt/storage/index.html' $(pulumi stack output url)
$ curl -X POST -d 'ls -la /mnt/storage' $(pulumi stack output url)
total 12
drwxr-xr-x 2 1000 1000 6144 Jun 10 05:38 .
drwxr-xr-x 3 root root 4096 Jun 10 05:35 ..
-rw-rw-r-- 1 1000 1000 21 Jun 10 05:37 file.txt
-rw-rw-r-- 1 1000 1000 21 Jun 10 05:38 index.html

Now that we can read and write to our file system from Lambda, what about sharing it between Lambda and other compute services? We could mount our volume to EC2, EKS, ECS, or Fargate. Let’s try out Fargate by running a small nginx server that serves from the same file system we just used.

First, we create an EFS volume configuration that describes the volume we want to mount in our Fargate Task. This is similar to Lambda - we use the same AccessPoint and root directory.

const efsVolumeConfiguration: aws.types.input.ecs.TaskDefinitionVolumeEfsVolumeConfiguration = {
 fileSystemId: filesystem.id,
 authorizationConfig: { accessPointId: ap.id, },
 rootDirectory: "/www",
};

Then we can launch our nginx task. We use our EFS volume configuration to define a volume named efs in the task, and then mount that volume into the /usr/share/nginx/html path in the nginx container so it can serve content directly from the EFS file system. Note that we also have to explicitly set the platformVersion to 1.4.0 so that we can use EFS since 1.4.0 is not yet the default. We ensure that we are using the same security group that can access the file system and that we’ve opened up access to port 80 for our nginx server.

const nginx = new awsx.ecs.FargateService("nginx", {
 cluster: cluster,
 taskDefinitionArgs: {
 container: {
 image: "nginx",
 memory: 128,
 portMappings: [{ containerPort: 80, hostPort: 80, protocol: "tcp" }],
 mountPoints: [{ containerPath: "/usr/share/nginx/html", sourceVolume: "efs" }],
 },
 volumes: [{ name: "efs", efsVolumeConfiguration }],
 },
 securityGroups: [vpc.vpc.defaultSecurityGroupId, ...cluster.securityGroups],
 subnets: vpc.publicSubnetIds,
 platformVersion: "1.4.0",
});

Navigating to our Task’s public IP - we see the contents of the file we stored in our file system.

Conclusion

Support for EFS in Lambda opens up a lot of new options for managing persistent data inside serverless functions. And even more importantly, makes it easy to share data in a format that existing applications in VM instances or containers can natively work with.

In this post, we’ve shown how you can use Pulumi to manage Lambda + EFS. Pulumi makes it easy to provision and manage cloud infrastructure on any cloud, using programming languages you are familiar with, including TypeScript, Python, Go and .NET. This is a great fit for EFS and Lambda, which combine managed infrastructure with small serverless function code.

Check out the full Lambda + EFS example above in the Pulumi Examples, learn more about Lambda support for EFS in the AWS docs, and get started with Pulumi for AWS today.

Architecture as Code: Serverless

Sophia Parafina — Tue, 28 Apr 2020 00:00:00 +0000

In this fourth installment of Architecture as Code series, we’ll take a look at serverless, an architectural pattern that has quickly gained popularity among cloud practitioners. There are two reasons why serverless usage has proliferated: a cost-saving pay as you go model and elasticity that goes from zero to as many as needed to complete the task without managing servers.

Overview

Serverless is a pattern for building services without having to manage servers. You write a service and deploy it on infrastructure managed by a cloud service provider. Serverless apps can be either a simple function invoked by an event or an application in a container. They are typically ephemeral, where the application runs within a fixed time or until it completes the task. Managed implementations such as AWS Lambda and Fargate, Azure Functions, and Google Cloud Run and Cloud Functions charge for the service only when it’s invoked and create instances, or workers, based on demand.

Developers can write the application in any language that the service provider supports. That means developers can choose the language that best meets the needs of their application. For example, a compiled application may respond more quickly than a scripting language. Serverless applications can be either synchronous or asynchronous, depending on the task.

A common feature of serverless architecture is a REST interface which maps HTTP operations such as GET, PUT, POST, and DELETE to functions. A REST interface simplifies implementing REST interfaces for functions and applications. AWS offers API Gateway for building REST interfaces, while Azure and Google Cloud have out of the box solutions.

Let’s take a look at examples where we can apply Architecture as Code to serverless.

Reusing resources

While serverless simplifies and reduces management overhead for deploying functions and containers, ancillary services also need to be configured. For example, a production deployment requires creating clusters, security groups, load balancers, and IAM policies. The following Python script creates resources that can be imported into __main__.py, which deploys the function or container. It ensures that each serverless application uses a vetted set of resources that can be replicated for every deployment.

from pulumi import export, ResourceOptions
import pulumi_aws as aws
import json

# Create an ECS cluster to run a container-based service.
cluster = aws.ecs.Cluster('cluster')

# Read back the default VPC and public subnets, which we will use.
default_vpc = aws.ec2.get_vpc(default='true')
default_vpc_subnets = aws.ec2.get_subnet_ids(vpc_id=default_vpc.id)

# Create a SecurityGroup that permits HTTP ingress and unrestricted egress.
group = aws.ec2.SecurityGroup('web-secgrp',
 vpc_id=default_vpc.id,
 description='Enable HTTP access',
 ingress=[{
 'protocol': 'tcp',
 'from_port': 80,
 'to_port': 80,
 'cidr_blocks': ['0.0.0.0/0'],
 }],
 egress=[{
 'protocol': '-1',
 'from_port': 0,
 'to_port': 0,
 'cidr_blocks': ['0.0.0.0/0'],
 }]
)

# Create a load balancer to listen for HTTP traffic on port 80.
alb = aws.lb.LoadBalancer('app-lb',
 security_groups=[group.id],
 subnets=default_vpc_subnets.ids
)

atg = aws.lb.TargetGroup('app-tg',
 port=80,
 protocol='HTTP',
 target_type='ip',
 vpc_id=default_vpc.id
)

wl = aws.lb.Listener('web',
 load_balancer_arn=alb.arn,
 port=80,
 default_actions=[{
 'type': 'forward',
 'target_group_arn': atg.arn
 }]
)

# Create an IAM role that can be used by our service's task.
role = aws.iam.Role('task-exec-role',
 assume_role_policy=json.dumps({
 'Version': '2008-10-17',
 'Statement': [{
 'Sid': '',
 'Effect': 'Allow',
 'Principal': {
 'Service': 'ecs-tasks.amazonaws.com'
 },
 'Action': 'sts:AssumeRole',
 }]
 })
)

rpa = aws.iam.RolePolicyAttachment('task-exec-policy',
 role=role.name,
 policy_arn='arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy'
)

To learn more about creating reusable components, read about Pulumi’s programming model, which shows how to author components.

Polyglot applications

This example shows how to deploy multiple functions in Google Cloud Functions. Note that we have one function written in Python and another in Go. The code that deploys the function is written in python and creates a Bucket to hold the function code and deploys both functions in the same script.

from pulumi_gcp import storage, cloudfunctions
from pulumi import export, asset

bucket = storage.Bucket("bucket")

py_bucket_object = storage.BucketObject(
 "python-zip",
 bucket=bucket.name,
 source=asset.AssetArchive({
 ".": asset.FileArchive("./pythonfunc")
 }))

py_function = cloudfunctions.Function(
 "python-func",
 source_archive_bucket=bucket.name,
 runtime="python37",
 source_archive_object=py_bucket_object.name,
 entry_point="handler",
 trigger_http="true",
 available_memory_mb=128,
)

export("python_endpoint", py_function.https_trigger_url)

go_bucket_object = storage.BucketObject(
 "go-zip",
 bucket=bucket.name,
 source=asset.AssetArchive({
 ".": asset.FileArchive("./gofunc")
 }))

go_function = cloudfunctions.Function(
 "go-func",
 source_archive_bucket=bucket.name,
 runtime="go111",
 source_archive_object=go_bucket_object.name,
 entry_point="Handler",
 trigger_http="true",
 available_memory_mb=128,
)

export("go_endpoint", go_function.https_trigger_url)

The full example is available on Github.

Cold Starts

Despite the many advantages of serverless, one of the challenges of serverless is latency, more commonly known as “cold starts.” There are several ways to lessen the impact of cold starts, including:

Invoking a pool of workers - a process known as ‘warming’.
Provisioned concurrency - workers that are always warm and dedicated to a specific function.
Dynamic concurrency - workers configured to autoscale as needed.
Scheduled profile - warming is applied during peak use.
Autoscaling-based utilization - adds workers as service utilization increases.

Deploying infrastructure with code lets you implement these strategies based on your application requirements. The code snippet below is an example of provisioned concurrency; a fully worked example is available on Github.

// Read the config of whether to provision fixed concurrency for Lambda
const config = new pulumi.Config();
const provisionedConcurrentExecutions = config.getNumber("provisionedConcurrency");

// Create a Lambda function, using code from the `./app` folder.
const lambda = new aws.lambda.Function("mylambda", {
 runtime: aws.lambda.DotnetCore2d1Runtime,
 code: new pulumi.asset.AssetArchive({
 ".": new pulumi.asset.FileArchive(dotNetApplicationPublishFolder),
 }),
 timeout: 300,
 handler: dotNetApplicationEntryPoint,
 role: role.arn,
 publish: !!provisionedConcurrentExecutions, // Versioning required for provisioned concurrency
 environment: {
 variables: {
 "COUNTER_TABLE": counterTable.name,
 },
 },
}, { dependsOn: [policy] });

if (provisionedConcurrentExecutions) {
 const concurrency = new aws.lambda.ProvisionedConcurrencyConfig("concurrency", {
 functionName: lambda.name,
 qualifier: lambda.version,
 provisionedConcurrentExecutions,
 });
}

Conclusion

Serverless is an attractive alternative to architectures that require hands-on management of servers. Despite this, it does not relieve you of the tasks needed to create and manage the ancillary resources needed for a modern application in production. Architecture as code enables you to declare resources and reuse them to ensure a standard and reproducible infrastructure. Also, you can deploy multiple functions and applications with a single Pulumi project, saving time and effort required to deploy them individually. Finally, you can tune your deployment strategies to overcome cold-start issues.

Controlling AWS Costs with Pulumi and AWS Lambda

Paul Stack — Thu, 09 Apr 2020 00:00:00 +0000

Due to the nature of the product we build, the Pulumi team needs to have access to several cloud providers to develop and test the product. An increasing number of cloud providers comes with an associated ever-increasing cost.

Giving a development team the ability to launch the resources they need in the cloud is both the best and most scary thing that happens in a company. The flexibility for them to be able to achieve their goal is fantastic, but with this great freedom can come problems. We depend on developers to clean up after themselves, but sometimes that doesn’t happen. Throw in automated testing failures that leak cloud resources, and we can very quickly get ourselves into a situation where costs spiral out of control.

There are multiple ways that Pulumi can help you deal with this situation. Joe Duffy wrote a blog post on Automatically Enforcing Tagging Policies to demonstrate how Policy as Code can ensure that tags are automatically added to any Pulumi managed infrastructure. If your company doesn’t use Pulumi to build and manage their infrastructure, then you can still use Pulumi in collaboration with AWS Lambda to build and deploy an application that can monitor your cloud costs.

To keep our cloud costs down, Pulumi uses an AWS Lambda Go application that removes wasteful resources from our AWS accounts daily. If a developer needs to protect their resources from being deleted, they need to add a tag to their infrastructure of “Owner: ”. This ensures the owner of a resource can be tracked at any time.

Cloud Cost Lambda

Here’s our application in it’s simplest form.

package main

import (
 "log"
 "os"
 "strconv"

 "github.com/aws/aws-lambda-go/lambda"
 "github.com/aws/aws-sdk-go-v2/aws/external"
)

var DeleteEnabled bool

func main() {
 lambda.Start(HandleRequest)
}

func HandleRequest() {
 destroyOperations, _ := os.LookupEnv("DESTROY_ENABLED")
 deleteEnabled, err := strconv.ParseBool(destroyOperations)
 if err != nil {
 log.Print("loading DESTROY_ENABLED envvar failed")
 log.Fatal(err)
 } else {
 DeleteEnabled = deleteEnabled
 }

 cfg, err := external.LoadDefaultAWSConfig()
 if err != nil {
 log.Print("loading AWS config failed")
 log.Fatal(err)
 }

 log.Printf("Running cleanup operations in %q", cfg.Region)
 log.Printf("Are destroy operations enabled? %q", destroyOperations)

 log.Print("Finished cleanup operations")
}

To deploy this application, we need to build and package the lambda. Packaging a Go lambda requires compiling it for Linux and then zipping the lambda up. We can do this as follows:

$ GOOS=linux GOARCH=amd64 go build -o main main.go && zip deployment.zip main

After creating a new Pulumi application, we can write the following code to deploy this lambda for us:

import * as aws from "@pulumi/aws";
import * as pulumi from "@pulumi/pulumi";

const lambdaRole = new aws.iam.Role("my-lambda-role", {
 name: "sample-role",
 assumeRolePolicy: aws.iam.assumeRolePolicyForPrincipal({ Service: "lambda.amazonaws.com" }),
});

const lambda = new aws.lambda.Function("my-lambda-function", {
 name: "sample-lambda",
 runtime: aws.lambda.Go1dxRuntime,
 timeout: 900,
 role: lambdaRole,
 handler: "main",
 memorySize: 1024,
 code: new pulumi.asset.FileArchive("deployment.zip"),
 environment: {
 variables: {
 "DESTROY_ENABLED": "true"
 }
 },
});

We can deploy the application using pulumi up:

$ pulumi up
Previewing update (dev):
 Type Name Plan
 + pulumi:pulumi:Stack lambda-post-dev create
 + ├─ aws:iam:Role my-lambda-role create
 + └─ aws:lambda:Function my-lambda-function create

Resources:
 + 3 to create

Do you want to perform this update? yes
Updating (dev):
 Type Name Status
 + pulumi:pulumi:Stack lambda-post-dev created
 + ├─ aws:iam:Role my-lambda-role created
 + └─ aws:lambda:Function my-lambda-function created

Resources:
 + 3 created

Duration: 27s

We can go to the AWS Lambda console and trigger the application. To do this, we must trigger a test event. Our application doesn’t accept any inputs, so the test event appears as follows:

When we trigger the lambda, we should get an output as follows:

START RequestId: 0910c342-4c43-4946-aeca-cf57683aa36b Version: $LATEST
2020/04/07 21:37:01 Running cleanup operations in "us-east-1"
2020/04/07 21:37:01 Are destroy operations enabled? "true"
2020/04/07 21:37:01 Finished cleanup operations
END RequestId: 0910c342-4c43-4946-aeca-cf57683aa36b
REPORT RequestId: 0910c342-4c43-4946-aeca-cf57683aa36b Duration: 1.96 ms Billed Duration: 100 ms Memory Size: 1024 MB Max Memory Used: 38 MB Init Duration: 99.32 ms

Our lambda application works!! But, it’s not going to delete anything because we haven’t told it what to delete. We can start very small. Let’s tell AWS to cleanup any EBS disks that are currently unattached.

func handleUnattachedVolumes(cfg aws.Config) {
 log.Print("Running cleanup operations for unattached ebs volumes")
 svc := ec2.New(cfg)

 input := &ec2.DescribeVolumesInput{
 Filters: []ec2.Filter{
 {
 Name: aws.String("status"),
 Values: []string{"available"},
 },
 },
 }
 volumes := make([]ec2.Volume, 0)

 for {
 req := svc.DescribeVolumesRequest(input)
 res, err := req.Send(context.Background())
 if err != nil {
 log.Fatal(err)
 }

 volumes = append(volumes, res.Volumes...)

 if res.NextToken == nil {
 break
 }

 input.NextToken = res.NextToken
 }

 for _, volume := range volumes {
 log.Printf("Killing EBS Volume %q as it's unattached", *volume.VolumeId)
 if DeleteEnabled {
 delReq := svc.DeleteVolumeRequest(&ec2.DeleteVolumeInput{
 VolumeId: volume.VolumeId,
 })
 _, err := delReq.Send(context.Background())
 if err != nil {
 log.Print(err)
 log.Printf("error deleting volume %q", *volume.VolumeId)
 }
 }
 }
}

We can then update our lambda handler to include this new EBS cleanup function:

func HandleRequest() {
 destroyOperations, _ := os.LookupEnv("DESTROY_ENABLED")
 deleteEnabled, err := strconv.ParseBool(destroyOperations)
 if err != nil {
 log.Print("loading DESTROY_ENABLED envvar failed")
 log.Fatal(err)
 } else {
 DeleteEnabled = deleteEnabled
 }

 cfg, err := external.LoadDefaultAWSConfig()
 if err != nil {
 log.Print("loading AWS config failed")
 log.Fatal(err)
 }

 log.Printf("Running cleanup operations in %q", cfg.Region)
 log.Printf("Are destroy operations enabled? %q", destroyOperations)

 handleUnattachedVolumes(cfg)

 log.Print("Finished cleanup operations")
}

When we run pulumi up, Pulumi will understand that the source code of the lambda has changed:

$ pulumi up
Previewing update (dev):
 Type Name Plan Info
 pulumi:pulumi:Stack lambda-post-dev
 ~ └─ aws:lambda:Function my-lambda-function update [diff: ~code]

Resources:
 ~ 1 to update
 2 unchanged

Do you want to perform this update? yes
Updating (dev):
 Type Name Status Info
 pulumi:pulumi:Stack lambda-post-dev
 ~ └─ aws:lambda:Function my-lambda-function updated [diff: ~code]

Resources:
 ~ 1 updated
 2 unchanged

Duration: 22s

When we execute the lambda code, we will receive an error now:

START RequestId: dc34a4f0-55f3-434b-a7e4-7832132865d1 Version: $LATEST
2020/04/07 21:45:01 Running cleanup operations in "us-east-1"
2020/04/07 21:45:01 Are destroy operations enabled? "true"
2020/04/07 21:45:01 Running cleanup operations for unattached ebs volumes
2020/04/07 21:45:01 UnauthorizedOperation: You are not authorized to perform this operation.
status code: 403, request id: c03efbf0-a3b4-4dd3-805b-02cec3523e0b
END RequestId: dc34a4f0-55f3-434b-a7e4-7832132865d1
REPORT RequestId: dc34a4f0-55f3-434b-a7e4-7832132865d1 Duration: 172.66 ms Billed Duration: 200 ms Memory Size: 1024 MB Max Memory Used: 49 MB Init Duration: 121.26 ms
RequestId: dc34a4f0-55f3-434b-a7e4-7832132865d1 Process exited before completing request

We haven’t given our IAM Role permission to interact with EBS. Let’s attach a specific policy to the IAM Role that allows us to interact with EBS:

const lambdaRolePolicy = new aws.iam.Policy("my-policy", {
 name: "sample-lambda-plicy",
 policy: {
 Version: "2012-10-17",
 Statement: [{
 Action: [
 "ec2:DescribeVolumes",
 "ec2:DeleteVolume",
 ],
 Resource: "*",
 Effect: "Allow",
 }],
 },
});

const policyAttachment = new aws.iam.PolicyAttachment("my-attachment", {
 policyArn: lambdaRolePolicy.arn,
 roles: [lambdaRole],
});

After deploying this updated IAM role, we can then see the execution will be allowed to proceed:

START RequestId: 7ca595aa-0a0f-4662-ab64-835ab0b02258 Version: $LATEST
2020/04/07 21:49:28 Running cleanup operations in "us-east-1"
2020/04/07 21:49:28 Are destroy operations enabled? "true"
2020/04/07 21:49:28 Running cleanup operations for unattached ebs volumes
2020/04/07 21:49:28 Finished cleanup operations
END RequestId: 7ca595aa-0a0f-4662-ab64-835ab0b02258
REPORT RequestId: 7ca595aa-0a0f-4662-ab64-835ab0b02258 Duration: 182.97 ms Billed Duration: 200 ms Memory Size: 1024 MB Max Memory Used: 26 MB

You can continue adding new segments to the cleanup lambda to help cleanup your AWS account.

Deploying To Multiple Regions

We want to ensure that all of our regions are cleaned up; therefore, we need to change the structure of the code. We can take advantage of explicit provider configuration to allow us to target the deployment of a resource to a specific region.

Let’s assume we want to deploy our application to us-east-1, us-east-2, eu-west-1, eu-west-2 and us-west-2. We can create a map of Providers to do this as follows:

const providers: {[key: string]: aws.Provider} = {
 "us-east-1": new aws.Provider("us-east-1", {region: "us-east-1"}),
 "us-east-2": new aws.Provider("us-east-2", {region: "us-east-2"}),
 "us-west-2": new aws.Provider("us-west-2", {region: "us-west-2"}),
 "eu-west-1": new aws.Provider("eu-west-1", {region: "eu-west-1"}),
 "eu-west-2": new aws.Provider("eu-west-2", {region: "eu-west-2"}),
};

We can iterate through the map and deploy our lambda to each of the configured regions as follows:

const providers: {[key: string]: aws.Provider} = {
 "us-east-1": new aws.Provider("us-east-1", {region: "us-east-1"}),
 "us-east-2": new aws.Provider("us-east-2", {region: "us-east-2"}),
 "us-west-2": new aws.Provider("us-west-2", {region: "us-west-2"}),
 "eu-west-1": new aws.Provider("eu-west-1", {region: "eu-west-1"}),
 "eu-west-2": new aws.Provider("eu-west-2", {region: "eu-west-2"}),
};

for (const providerKey of Object.keys(providers)) {
 const provider = providers[providerKey];

 const lambda = new aws.lambda.Function(`my-lambda-function-${providerKey}`, {
 name: "sample-lambda",
 runtime: aws.lambda.Go1dxRuntime,
 timeout: 900,
 role: lambdaRole.arn,
 handler: "main",
 memorySize: 1024,
 code: new pulumi.asset.FileArchive("deployment.zip"),
 environment: {
 variables: {
 "DESTROY_ENABLED": "true"
 }
 },
 }, {provider});
}

Pulumi can then deploy the lambdas to all of the providers in the loop. If a new region requires cleaning, then it needs to be added to the map of providers.

In Conclusion

The Go application we built continues to evolve. There are new areas of our account that have started to accumulate costs. After a few iterations of the application, we were able to drop 64% of our daily AWS costs.

We look forward to hearing the results of your work in this area!

Architecture as Code

Sophia Parafina — Fri, 20 Mar 2020 00:00:00 +0000

Abstraction is key to building resilient systems because it encapsulates behavior and decouples code, letting each component perform its function independently. The same principles apply to infrastructure, where we want to declare behavior or state and not implementation details. As an industry, we’ve moved away from monolithic applications to distributed systems such as serverless, microservices, Kubernetes, and virtual machine deployments. In this article, we’ll take a closer look at the characteristics of these architectures and how Pulumi can abstract the components that comprise these systems.

Virtual Machines

Virtual machines arose from the need to use expensive hardware more efficiently. VMs are more efficient because they share the host server, decreasing the number of physical servers needed to run applications. A group of hosts can also be aggregated so that it behaves as a single virtual host with large memory and processing capabilities. Virtual machines can emulate other environments that have specific hardware requirements or capabilities that the host machine does not have, thus reducing the need for specific hardware or operating systems. Virtualization provides the host machine isolation from the guest virtual machines by limiting access to the host machine through an abstraction layer. Finally, virtualization has the added advantage of providing a fine degree of control over resources available to the virtual machine.

If virtual machines are part of your infrastructure, Pulumi lets you create virtual machines programmatically across cloud service providers. The JavaScript example below illustrates how to deploy a virtual machine on Google Cloud Platform. The script spins up a Debian based instance and creates a network with an open port 22 to let you configure and manage the machine as needed.

const gcp = require("@pulumi/gcp");
const fs = require("fs");

// Create a network
const network = new gcp.compute.Network("network");
const computeFirewall = new gcp.compute.Firewall("firewall", {
 network: network.id,
 allows: [{
 protocol: "tcp",
 ports: [ "22", "80" ],
 }],
});

// Read the configuration script
const startupScript = fs.readFileSync("config.sh", "utf-8")

// Create a Virtual Machine Instance.
const computeInstance = new gcp.compute.Instance("instance", {
 machineType: "f1-micro",
 zone: "us-central1-a",
 metadataStartupScript: startupScript,
 bootDisk: { initializeParams: { image: "debian-cloud/debian-9" } },
 networkInterfaces: [{
 network: network.id,
 // accessConfigs must include a single empty config to request an ephemeral IP.
 accessConfigs: [{}],
 }],
});

// Export the name and IP address of the Instance.
exports.instanceName = computeInstance.name;
exports.instanceIP = computeInstance.networkInterfaces.apply(ni => ni[0].accessConfigs[0].natIp);

This is the configuration script, config.sh, referenced in the code:

#!/bin/bash
echo "Hello, World!" > index.html
nohup python -m SimpleHTTPServer 80 &

When the virtual machine is created, it reads the configuration file that creates a webserver. This a basic example, but it shows how to configure virtual machines programmatically, which is useful for scaling horizontally.

Serverless

Serverless is a computing architecture characterized by server-side logic running in stateless containers that are invoked by events. They are typically ephemeral (often are available for only one call), and managed by third-party cloud providers. Serverless is also called Functions as a Service, or FaaS, and well-known implementations include AWS Lambda and Fargate, Azure Functions, and Google Cloud Run and Cloud Functions.

A significant benefit of serverless is that it’s a polyglot platform that allows developers to choose languages optimized for a task. For example, scripting languages like JavaScript or Python may be more responsive than a language such as Java. Serverless functions can support both synchronous and asynchronous calls. There are use cases that require an immediate response such as processing a video stream and instances where returning the result immediately is not critical as in an ETL batch job.

A key part of serverless architecture is an API Gateway that provides logical routes for mapping standard HTTP operations such as GET, PUT, POST, and DELETE to functions. The gateway makes development easier since these are standard and well-known interfaces. In addition to an API Gateway, the serverless platform should provide REST endpoints that allow you to manage the deploy with a CLI, portal, or an automation script. Finally, serverless architecture is extensible and supports integration with event sources and resources from the cloud provider through webhooks and other mechanisms.

The following example illustrates how to deploy two functions using TypeScript, one written in Python and the other written in Go. You can find the full example in the Pulumi GitHub repository.

import * as gcp from "@pulumi/gcp";
import { asset } from "@pulumi/pulumi";

const bucket = new gcp.storage.Bucket("bucket");

// Google Cloud Function in Python

const bucketObjectPython = new gcp.storage.BucketObject("python-zip", {
 bucket: bucket.name,
 source: new asset.AssetArchive({
 ".": new asset.FileArchive("./pythonfunc"),
 }),
});

const functionPython = new gcp.cloudfunctions.Function("python-func", {
 sourceArchiveBucket: bucket.name,
 runtime: "python37",
 sourceArchiveObject: bucketObjectPython.name,
 entryPoint: "handler",
 triggerHttp: true,
 availableMemoryMb: 128,
});

export const pythonEndpoint = functionPython.httpsTriggerUrl;

// Google Cloud Function in Go

const bucketObjectGo = new gcp.storage.BucketObject("go-zip", {
 bucket: bucket.name,
 source: new asset.AssetArchive({
 ".": new asset.FileArchive("./gofunc"),
 }),
});

const functionGo = new gcp.cloudfunctions.Function("go-func", {
 sourceArchiveBucket: bucket.name,
 runtime: "go111",
 sourceArchiveObject: bucketObjectGo.name,
 entryPoint: "Handler",
 triggerHttp: true,
 availableMemoryMb: 128,
});

export const goEndpoint = functionGo.httpsTriggerUrl;

Kubernetes

Kubernetes is a container orchestration system for deploying, scaling, and managing your application. Applications use multiple containers that contain code and dependencies needed for the code to execute. Kubernetes runs on a cluster of nodes and containers are deployed in pods. Kubernetes manages the pods by replacing containers that fail and removing containers that don’t respond. It can also scale the number of containers as needed and load balance the network traffic. These are some of the features that Kubernetes provides:

Load balancing and traffic distribution across the cluster
Service discovery
Create and manage containers
Remove failed containers and reassign their resources to a new container
Mount storage systems such as local storage and storage on public cloud providers
Automated deployments based on different deploy methods such as canary or blue/green deployments
Automated rollbacks to last known good state
Secrets management

The following example is based on the Kubernetes Guestbook example. The difference between this implementation and the original application is that instead of using YAML to declare the infrastructure, it uses a component written in TypeScript to create the service deployment which is implemented as k8sjs.

import * as k8s from "@pulumi/kubernetes";
import * as k8stypes from "@pulumi/kubernetes/types/input";
import * as pulumi from "@pulumi/pulumi";

/**
 * ServiceDeployment is an example abstraction that uses a class to fold together the common pattern of a
 * Kubernetes Deployment and its associated Service object.
 */
export class ServiceDeployment extends pulumi.ComponentResource {
 public readonly deployment: k8s.apps.v1.Deployment;
 public readonly service: k8s.core.v1.Service;
 public readonly ipAddress?: pulumi.Output<string>;

 constructor(name: string, args: ServiceDeploymentArgs, opts?: pulumi.ComponentResourceOptions) {
 super("k8sjs:service:ServiceDeployment", name, {}, opts);

 const labels = { app: name };
 const container: k8stypes.core.v1.Container = {
 name,
 image: args.image,
 resources: args.resources || { requests: { cpu: "100m", memory: "100Mi" } },
 env: [{ name: "GET_HOSTS_FROM", value: "dns" }],
 ports: args.ports && args.ports.map(p => ({ containerPort: p })),
 };
 this.deployment = new k8s.apps.v1.Deployment(name, {
 spec: {
 selector: { matchLabels: labels },
 replicas: args.replicas || 1,
 template: {
 metadata: { labels: labels },
 spec: { containers: [ container ] },
 },
 },
 }, { parent: this });

 this.service = new k8s.core.v1.Service(name, {
 metadata: {
 labels: this.deployment.metadata.labels,
 },
 spec: {
 ports: args.ports && args.ports.map(p => ({ port: p, targetPort: p })),
 selector: this.deployment.spec.template.metadata.labels,
 // Minikube does not implement services of type `LoadBalancer`; require the user to specify if we're
 // running on minikube, and if so, create only services of type ClusterIP.
 type: args.allocateIpAddress ? (args.isMinikube ? "ClusterIP" : "LoadBalancer") : undefined,
 },
 }, { parent: this });

 if (args.allocateIpAddress) {
 this.ipAddress = args.isMinikube ?
 this.service.spec.apply(spec => spec.clusterIP) :
 this.service.status.apply(status => status.loadBalancer.ingress[0].ip);
 }
 }
}

export interface ServiceDeploymentArgs {
 image: string;
 resources?: k8stypes.core.v1.ResourceRequirements;
 replicas?: number;
 ports?: number[];
 allocateIpAddress?: boolean;
 isMinikube?: boolean;
}

The main program deploys an instance of Redis and the guestbook application using containers pulled from the Google Container Registry.

import * as pulumi from "@pulumi/pulumi";
import * as k8sjs from "./k8sjs";

const config = new pulumi.Config();

const redisMaster = new k8sjs.ServiceDeployment("redis-master", {
 image: "k8s.gcr.io/redis:e2e",
 ports: [6379],
});

const redisReplica = new k8sjs.ServiceDeployment("redis-replica", {
 image: "gcr.io/google_samples/gb-redisslave:v1",
 ports: [6379],
});

const frontend = new k8sjs.ServiceDeployment("frontend", {
 replicas: 3,
 image: "gcr.io/google-samples/gb-frontend:v4",
 ports: [80],
 allocateIpAddress: true,
 isMinikube: config.getBoolean("isMinikube"),
});

export const frontendIp = frontend.ipAddress;

The original example defines the Service and Deployment in YAML. In the example above, the YAML is abstracted in a reusable component written in a modern programming language. The complete example is available on GitHub. As you can see, the ability to reuse components gets us closer to implementing architecture as code.

Microservices

Microservices are based on the idea that components encapsulate a business capability. Services are independently deployable and communicate via web service requests, which has the advantage of redeploying only one or selected services and leaving the remaining components up and running. The 12-Factor app is the canonical pattern for microservices, which is summarized by:

Use declarative methods to configure and deploy applications.
Enforce maximum portability between environments by establishing a clean contract with the underlying operating system.
Deploy on modern cloud platforms that diminish the need for servers and the requisite system administration.
Establish continuous deployment by keeping development, testing, and production stages similar.
Use platforms with the ability to scale without significant changes to architecture, development, or tooling.

In addition to Kubernetes, there are other platforms for deploying container-based applications. AWS Elastic Container Service, Google Cloud Run, and Azure Container Service provide alternatives to Kubernetes for container orchestration. The following code snippet demonstrates deploying a container with a Ruby application in Google Cloud Run. Note that the image is built using a local Docker engine and pushed to the Google Cloud Repository. The code shows that you can also set memory limits and concurrency for the container. The full example is available on GitHub.

import * as docker from "@pulumi/docker";
import * as gcp from "@pulumi/gcp";
import * as pulumi from "@pulumi/pulumi";

// Location to deploy Cloud Run services
const location = gcp.config.region || "us-central1";

// -------------------------------------- //
// Deploy a custom container to Cloud Run //
// -------------------------------------- //

// Build a Docker image from our sample Ruby app and put it to Google Container Registry.
// Note: Run `gcloud auth configure-docker` in your command line to configure auth to GCR.
const imageName = "ruby-app";
const myImage = new docker.Image(imageName, {
 imageName: pulumi.interpolate`gcr.io/${gcp.config.project}/${imageName}:v1.0.0`,
 build: {
 context: "./app",
 },
});

// Deploy to Cloud Run. Some extra parameters like concurrency and memory are set for illustration purpose.
const rubyService = new gcp.cloudrun.Service("ruby", {
 location,
 template: {
 spec: {
 containers: [{
 image: myImage.imageName,
 resources: {
 limits: {
 memory: "1Gi",
 },
 },
 }],
 containerConcurrency: 50,
 },
 },
});

// Open the service to public unrestricted access
const iamRuby = new gcp.cloudrun.IamMember("ruby-everyone", {
 service: rubyService.name,
 location,
 role: "roles/run.invoker",
 member: "allUsers",
});

// Export the URL
export const rubyUrl = rubyService.status.url;

Conclusion

We’ve covered the major infrastructure architectural patterns use for deploying modern applications. Whether you use Virtual Machines, Serverless, Kubernetes or Microservices with containers the goal is to create reusable components that abstract the configuration details and enable plug-and-play architecture. In subsequent articles, we’ll take an in-depth look at each pattern and how to implement them on major providers using modern languages.

Scheduling Serverless

Cyrus Najmabadi — Tue, 25 Feb 2020 00:00:00 +0000

Scheduling events has long been an essential part of automation; many tasks need to run at specific times or intervals. You could be checking StackOverflow for new questions every 20 minutes or compiling a report that is emailed every other Friday at 4:00 pm. Today, many of these tasks can be efficiently accomplished in the cloud. While each cloud has its flavor of scheduled functions, this post steps you through an example using AWS CloudWatch with the help of Pulumi.

Taking out the trash

Let’s assume we have an S3 Bucket similar to the Recycle Bin on your desktop. It contains discarded items that we might not be ready to delete yet, but because we are paying for storage, we want to empty this bucket every Friday at 6:00 pm EST.

import * as aws from "@pulumi/aws";
import { ObjectIdentifier } from "aws-sdk/clients/s3";

// Create an AWS resource (S3 Bucket)
const trashBucket = new aws.s3.Bucket("trash");

We’re now ready to create our Lambda function and schedule it. Pulumi simplifies creating Lambda functions by allowing us to provide a handler function to an event such as cloudwatch.onSchedule. To learn more, check out our Lambdas as Lambdas post. Let’s create a cloudwatch.EventRuleEventHandler async function that will get all the items from the trash bucket and delete them.

// A handler function that will list objects in the bucket and bulk delete them
const emptyTrash: aws.cloudwatch.EventRuleEventHandler = async (
 event: aws.cloudwatch.EventRuleEvent
) => {
 const s3Client = new aws.sdk.S3(); //creates interface to service
 const bucket = trashBucket.id.get();

 const { Contents = [] } = await s3Client //get list of objects in bucket
 .listObjects({ Bucket: bucket })
 .promise();
 const objects: ObjectIdentifier[] = Contents.map(object => {
 return { Key: object.Key! };
 });

 await s3Client //delete objects
 .deleteObjects({
 Bucket: bucket,
 Delete: { Objects: objects, Quiet: false }
 })
 .promise()
 .catch(error => console.log(error));
 console.log(
 `Deleted ${Contents.length} item${
 Contents.length === 1 ? "" : "s"
 } from ${bucket}.`
 );
};

Now that we have our handler function, we can create a CloudWatch event that fires based on the specified schedule, which is every Friday at 6:00 pm EST. We’ll need to represent that as a Schedule Expression. CloudWatch events support cron and rate expressions. If we wanted our function to run on a set interval (i.e., every 20 minutes), we would choose a rate expression. Since we want fine-grained schedule control, we’ll be using a cron expression. You can learn more at the Schedule Expressions documentation. Because time for cron jobs uses UTC, we’ll need to schedule our event to fire on Friday at 11:00 pm UTC.

// Schedule the function to run every Friday at 11:00pm UTC (6:00pm EST)
// More info on Schedule Expressions at
// https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/ScheduledEvents.html
const emptyTrashSchedule: aws.cloudwatch.EventRuleEventSubscription = aws.cloudwatch.onSchedule(
 "emptyTrash",
 "cron(0 23 ? * FRI *)",
 emptyTrash);

// Export the name of the bucket
export const bucketName = trashBucket.id;

Now, run pulumi up to deploy your new scheduled function.

$ pulumi up
Previewing update (dev):
…

Updating (dev):

 Type Name Status
 pulumi:pulumi:Stack aws-ts-scheduled-function-dev
 + └─ aws:cloudwatch:EventRuleEventSubscription emptyTrash created
 + ├─ aws:cloudwatch:EventRule emptyTrash created
 + ├─ aws:iam:Role emptyTrash created
 + ├─ aws:iam:RolePolicyAttachment emptyTrash-32be53a2 created
 + ├─ aws:lambda:Function emptyTrash created
 + ├─ aws:lambda:Permission emptyTrash created
 + └─ aws:cloudwatch:EventTarget emptyTrash created

Outputs:
 bucketName: "trash-1cde441"

Resources:
 + 7 created
 2 unchanged

Duration: 14s

Test it out by adding some items to your bucket and see if they’re deleted. You can use pulumi logs to view the console.log from the handler function. If you can’t wait until Friday, experiment with changing the cron expression and rerun pulumi up. You can find the full code in our GitHub repository.

Want to know more?

This simple example demonstrates what you can do with serverless functions, such as AWS Lambda, without having to set up your infrastructure. Check out more serverless examples in the Pulumi Examples to see what you can build.

The Beginner's Guide to AWS Serverless Step Functions

Sophia Parafina — Thu, 13 Feb 2020 00:00:00 +0000

AWS Step Functions lets you build applications by connecting AWS services. Daisy-chaining steps into a workflow simplifies application development by creating a state machine diagram which shows how services are connected to each other in your application. We’ll go into the details of creating a lambda function, IAM roles and policies, and creating a workflow. Once we have the example deployed, we’ll walk through the process of adding another function and step to the workflow. Included in the walkthrough is a discussion of one of the aspects of the Pulumi programming model. The goal of this article is to provide a foundation for building your application using serverless workflows.

Getting Started with AWS Step Functions

Let’s start with a hello world function in an AWS Lambda. You can start the example from Github by selecting the Deploy with Pulumi button. A new project page is created automatically in the Pulumi Service, select Create Project to get started. Follow the steps to copy the project locally. As with any Python project, we recommend creating a virtual environment to store project-specific dependencies.

$ python3 -m venv venv
$ source venv/bin/activate
$ pip3 install -r requirements.txt

Before we run the example, let’s look at the code. A step function is made up of a lambda function and a state machine that runs the function. We first need to define the permissions for the code. Both the lambda and the state machine require an IAM execution role and IAM access policy to access AWS services and resources. The iam.py module establishes the roles and policies and the AWS region, which are imported in __main__,py.

The lambda returns “Hello world!” when called. Note that the handler is the name of the module, hello, and the name of the function, handler.

hello_world_fn = lambda_.Function('helloWorldFunction',
 role=iam.lambda_role.arn,
 runtime="python3.7",
 handler="hello.handler",
 code=pulumi.AssetArchive({
 '.': pulumi.FileArchive('./step_hello')
 })
)

The state machine controls the workflow. It executes using the IAM role we defined and has all the permissions necessary to execute the tasks defined. The definition specifies the start state and all the other states. In our case, there’s only one state, HelloWorld.

state_defn = state_machine = sfn.StateMachine('stateMachine',
 role_arn=iam.sfn_role.arn,
 definition=hello_world_fn.arn.apply(lambda arn: """{
 "Comment": "A Hello World example of the Amazon States Language using an AWS Lambda Function",
 "StartAt": "HelloWorld",
 "States": {
 "HelloWorld": {
 "Type": "Task",
 "Resource": "%s",
 "End": true
 }
 }
 }""" % arn)
)

To run the example run and select yes.

$ pulumi up
Previewing update (dev):

 Type Name Plan
 pulumi:pulumi:Stack stepfunctions-py-dev
 + ├─ aws:iam:Role lambdaRole create
 + ├─ aws:iam:Role sfnRole create
 + ├─ aws:iam:RolePolicy lambdaRolePolicy create
 + ├─ aws:iam:RolePolicy sfnRolePolicy create
 + ├─ aws:lambda:Function helloWorldFunction create
 + └─ aws:sfn:StateMachine stateMachine create

Outputs:
 + state_machine_arn: output<string>

Resources:
 + 6 to create
 1 unchanged

Do you want to perform this update?
 yes
> no
 details

Select yes to run the example, but note that the function doesn’t specify where to return Hello world!. We can execute the function from the AWS cli, but to see how it works, we’ll need to launch the AWS Console.

Call the function first.

aws stepfunctions start-execution --state-machine-arn $(pulumi stack output state_machine_arn)

Then check the AWS console to see if the lambda was executed.

Clicking on the Name displays the step details, which include the input and out, i.e., ‘Hello world!’

The AWS console provides a graph of your workflow and the details of each step. Clicking on Output shows “Hello world” meaning that the call to the lambda function was successful.

Expand AWS Step Functions Workflow

Now that we have a basic flow, let’s add another step. To keep it simple we’ll create another simple function that returns “Nice to meet you.” To do this, make a file called nice.py and add the function.

def handler(event, context):
 return "Nice to meet you"

We will also need to import Output from pulumi along with the other imports. We’ll use it later.

from pulumi import Output

Next, add the new lambda function, nice_fn. Note that we can reuse the same IAM role and policy we use in the hello_world_fn function.

nice_fn = lambda_.Function('niceFunction',
 role=iam.lambda_role.arn,
 runtime="python3.7",
 handler="nice.handler",
 code=pulumi.AssetArchive({
 '.': pulumi.FileArchive('./step_hello')
 })
)

Finally, we need to update the state machine with a new workflow that includes our new function. The state machine takes three arguments, a name, an IAM role, and the workflow as a JSON document. The code to get the ARN names is Pulumi specific and requires an explanation if you are not familiar with the Pulumi programming model.

Instance properties of a resource are called outputs, which behave much like promises. This method is necessary because outputs may not be fully available until a resource is fully provisioned. To get the value of a resource, Pulumi has the apply method, which has a call back that receives the raw value and computes a new output. Apply is how we get the lambda ARN names in the workflow document. The Output.all method joins a list of Outputs that we can apply to a Python lambda.

We add a new step function by changing the HelloWorld step to call the ‘Nice’ step with the tag ”Next” : “Nice”. We add the “Nice” step and end the workflow. Here’s how the state machine is configured when adding another step.

state_defn = state_machine = sfn.StateMachine('stateMachine',
 role_arn=iam.sfn_role.arn,
 definition=Output.all(hello_world_fn.arn,nice_fn.arn).apply(lambda args: """{
 "Comment": "A Hello World example of the Amazon States Language using an AWS Lambda Function",
 "StartAt": "HelloWorld",
 "States": {
 "HelloWorld": {
 "Type": "Task",
 "Resource": "%s",
 "Next": "Nice"
 },
 "Nice": {
 "Type": "Task",
 "Resource": "%s",
 "End": true
 }
 }
 }""" % (args[0], args[1]) )
)

Run pulumi up to update the deployment and call it using the AWS client again.

aws stepfunctions start-execution --state-machine-arn $(pulumi stack output state_machine_arn)

You can see the results in the AWS Console. The workflow graph shows the new step, and the output from nice_fn is “Nice to meet you.”

What’s next

This example is just a simple workflow intended as an introduction. In future articles, we’ll expand on this basic model to include custom lambda functions and AWS services. If python isn’t your preferred programming language, this example is also available in typescript. Additionally, check out the examples repository to see what you can build next.

Deploy Serverless Containers on Google Cloud Run with Pulumi

Mikhail Shilkov — Tue, 04 Feb 2020 00:00:00 +0000

Google Cloud Run is the latest addition to the serverless compute family. While it may look similar to existing services of public cloud, the feature set makes Cloud Run unique:

Docker as a deployment package enables using any language, runtime, framework, or library that can respond to an HTTP request.
Automatic scaling, including scale to zero, means you pay for what you consume with no fixed cost and no management overhead.
HTTP load-balancing out of the box simplifies the usage.

Cloud Run is targeted very specifically at stateless web applications. It uses ephemeral containers, and each execution is limited to 15 minutes.

Today, we will deploy our first Cloud Run services with Pulumi. Then, we’ll discuss pricing and compare Cloud Run to the competition.

Hello Cloud Run

We’ll start by deploying a pre-built container image provided by Google to a Cloud Run service.

To follow along, create a new Pulumi project with the Get Started with Google Cloud guide.

Enable Cloud Run for the project

As with any other service of Google Cloud, you need to enable it for the target project before the first deployment. You can do so with the gcp.projects.Service resource:

const enableCloudRun = new gcp.projects.Service("EnableCloudRun", {
 service: "run.googleapis.com",
});

Note: If you destroy the Pulumi stack and delete the projects.Service resource, the Cloud Run service will be disabled again. If you use Cloud Run in multiple Pulumi stacks, you should move the service management to a central location, i.e., a shared stack.

Choose a location

Cloud Run is a regional service: all container instances run in a single location of your choice.

Set the region setting to one of the currently supported locations: us-central1 (Iowa), us-east1 (South Carolina), europe-west1( Belgium), or asia-northeast1 (Tokyo).

pulumi config set gcp:region <region>

Then, the program can read the value and reuse it for all resources:

// Location to deploy Cloud Run services
const location = gcp.config.region || "us-central1";

Deploy a Cloud Run service

Google provides a pre-deployed “Hello Cloud Run” image at gcr.io/cloudrun/hello. The following resource deploys that image to your GCP project with the default settings.

const helloService = new gcp.cloudrun.Service("hello", {
 location,
 template: {
 spec: {
 containers: [
 { image: "gcr.io/cloudrun/hello" },
 ],
 },
 },
}, { dependsOn: enableCloudRun });

Expose unrestricted HTTP access

By default, Google does not expose HTTP endpoints of a Cloud Run service to the Internet. To make it publicly available, you should grant the roles/run.invoker role to allUsers.

const iamHello = new gcp.cloudrun.IamMember("hello-everyone", {
 service: helloService.name,
 location,
 role: "roles/run.invoker",
 member: "allUsers",
});

Note that the Contributor role is required for the user who executes Pulumi deployment to grant the access.

Try it out

Each Cloud Run service automatically gets a run.app subdomain. You can export the exact URL as a Pulumi output.

// Export the URL
export const helloUrl = helloService.status.url;

Run pulumi up to deploy all the resources.

$ pulumi up
...
Outputs:
~ helloUrl: "https://hello-585ad15-q4wszdxb2a-ew.a.run.app"

Navigate to the URL to see the welcome screen:

Congratulations, your first Cloud Run service is up and running. Now, it’s time to deploy some custom code.

Deploy a Custom Application

Cloud Run can run any Docker container so that you can write the application code in a language of your choice. I picked Ruby as an example of a language that is not supported by Cloud Functions.

Create a Ruby web app

Let’s create an app subfolder and place all application files there. The first file app.rb is a Hello World web application in Ruby.

require 'sinatra'

set :bind, '0.0.0.0'

get '/' do
 target = ENV['TARGET'] || 'Pulumi'
 "Hello #{target}!\n"
end

Gemfile and Gemfile.lock files configure the required gems (packages).

Define a Docker image

Dockerfile defines the container image to deploy. It is based on a generic Ruby image, copies the application files, and runs the web application.

# Use the official lightweight Ruby image.
FROM ruby:2.7-slim

# Install production dependencies.
WORKDIR /usr/src/app
COPY Gemfile Gemfile.lock ./
ENV BUNDLE_FROZEN=true
RUN bundle install

# Copy local code to the container image.
COPY . ./

# Run the web service on container startup.
CMD ["ruby", "./app.rb"]

Note that nothing in this web application is specific to Cloud Run.

Build and publish the Docker image

Cloud Run can only deploy images from Google Cloud Registry (GCR). Therefore, our Pulumi program needs to build the Docker image with the sample Ruby application and push it to GCR.

Run gcloud auth configure-docker in your command line to configure your local Docker installation to use GCR endpoints.

Run npm i @pulumi/docker to install the Pulumi Docker SDK. Then, add the docker.Image resource to the Pulumi program.

const myImage = new docker.Image("ruby-image", {
 imageName: pulumi.interpolate`gcr.io/${gcp.config.project}/my-ruby-app:v1.0.0`,
 build: {
 context: "./app",
 },
});

Pulumi takes care of building the image in the app folder and uploading it to GCR.

Deploy to Cloud Run

Now, you can deploy another Cloud Run service and point it to the custom image.

const rubyService = new gcp.cloudrun.Service("ruby", {
 location,
 template: {
 spec: {
 containers: [{
 image: myImage.imageName,
 resources: {
 limits: {
 memory: "1Gi",
 },
 },
 }],
 containerConcurrency: 50,
 },
 },
}, { dependsOn: enableCloudRun });

This snippet illustrates two additional configuration values. The memory limit defines the amount of RAM available to each container. Container concurrency defines how many requests each container instance may process in parallel at any given time. Both parameters influence the price of the deployment, as explained in the Pricing section below.

Currently, it’s not possible to use partial vCPU or multiple vCPUs per instance: your instance has always one vCPU assigned. You can adjust the memory from 128Mb to 2Gb.

Finally, don’t forget to enable unrestricted access and export the public URL.

const iamRuby = new gcp.cloudrun.IamMember("ruby-everyone", {
 service: rubyService.name,
 location,
 role: "roles/run.invoker",
 member: "allUsers",
});

export const rubyUrl = rubyService.status.url;

After deploying the program, you should be able to access the application with an HTTP call.

$ curl "$(pulumi stack output rubyUrl)"
Hello Pulumi!

You can find the complete example in the Pulumi examples Github repository.

Pricing

Cloud Run pricing is entirely consumption-based. After you exhaust the free tier, you start paying for four components, combined:

Per CPU time ($24.00 / million seconds)
Per memory consumption ($2.50 / million GB-seconds)
Per request ($0.40 / million requests)
Egress traffic, same as any other Google Cloud usage

Both time-based measurements are rounded up to the nearest 100 ms. All the provisioned memory is charged, the actual Mb/Gb consumption doesn’t matter.

It’s important to understand that the CPU and memory metrics are calculated per active instance (host, provisioned container), not per request. Each Cloud Run instance can handle multiple requests concurrently. Overlapping executions aren’t double charged: billable time begins with the start of the first request and ends at the end of the last request.

The following picture illustrates the pricing for three executions running on the same container host.

Multiple requests can share the allocated CPU and memory, so it makes sense to set the concurrency setting as high as possible for a given application (but not higher). This model is a big difference to Cloud Functions, which are charged for each request independently.

Comparing Cloud Run to Other Services

Despite the existence of multiple services that look somewhat similar, Cloud Run is unique in its capabilities.

Google Cloud Run vs Functions

Google Cloud Functions (GCF) service deploys snippets of code as functions, while Cloud Run deploys a web application packaged as a Docker image. Currently, GCF only supports three runtimes (Node.js, Python, and Go), while Cloud Run can run practically any language and any runtime.

GCF has a notion of events and triggers: it can natively integrate with Pub/Sub, Cloud Storage, Cloud Firestore. Cloud Run is all about handling HTTP requests: Any connection to another service has to go via HTTP.

Google Cloud Run vs AWS Fargate

AWS Fargate deploys container images. It requires an ECS cluster to run on and imposes more configuration burden on the user, including networking, load balancing, auto-scaling, and service discovery. Pulumi Crosswalk for AWS can help with these tasks.

Fargate is capable of hosting long-running workloads. Therefore, Fargate’s scaling model is not tied to individual requests, and there is no scale-to-zero out of the box.

With Cloud Run, there’s no notion of a cluster to manage, the scaling and billing models are based on individual requests. However, the workloads are HTTP-only, with a maximum duration of 15 minutes per call.

Google Cloud Run vs Azure Container Instances

Azure Container Instances (ACI) can also run arbitrary containers and has built-in HTTP endpoints. However, there’s no auto-scaling: you get a single host for each instance. Also, there’s no load balancing capability across multiple instances.

Cloud Run is elastically scalable, which positions it much better to host applications with variable workloads.

Conclusion

Cloud Run is a service with a unique combination of serverless ease of use and pricing with the full power of container deployments. It is easier to manage than Fargate, has automatic scaling missing from ACI, and greater flexibility compared to GCF. With concurrent requests model, Cloud Run is well-positioned to be significantly cheaper than Cloud Functions for some scenarios.

If you are excited, follow the example to Get Started with Cloud Run and Pulumi.

AWS Serverless Analytics

Evan Boyle — Thu, 30 Jan 2020 00:00:00 +0000

Whether it’s an IoT installation, a website, or a mobile app, modern software systems generate a trove of usage and performance data. While it can be daunting to collect and manage, surfacing data empowers the business to make informed product investments. In this article, we’ll explore the following:

An overview of the traditional Redshift analytics stack on AWS, the use cases it excels at, and where it falls apart.
An alternative architecture utilizing serverless and streaming.
How to implement this architecture as code in a reusable library with Pulumi.
How to automate the development loop when writing Pulumi libraries.

If you’d like to follow along, you can clone and run the reference implementation. If you’re new to Pulumi, you can follow this guide to get started.

ACME E-Commerce

ACME is a burgeoning online retailer. They ship goods around the world to millions of happy customers. ACME has recently started an incubation project to test out running ads on its website. They are unsure of how customers will react or whether their merchants will be interested in participating, so they staff a small team of 5 tasked with launching an MVP.

ACME wants to track the performance of its ads, understand how its customers are interacting with them, and look for patterns that can inform business decisions and further investment. With such a small team, they look for the path of least resistance as analytics is just a small slice of the overall work to be done to launch on their aggressive timeline.

The Traditional Analytics Stack

A familiar SQL interface and the ease of loading data through built-in features like S3 COPY have made Redshift the goto choice for bootstrapping an analytics stack on new projects. It’s low friction to get from zero to something, and Redshift scales to petabytes.

Like all tools, Redshift has targeted use cases that it excels at. Broadly speaking, it’s niche is heavy BI and analytics workloads, reading more than you write. Redshift can serve millisecond-level queries on massive datasets through a horizontally scalable, distributed processing architecture. At the core of this is a reliance on sorting and data distribution mechanisms. Schemas are configured by administrators to distribute the data across the cluster in a variety of methods that optimize for different use cases, such as highly denormalized tables, or tables making heavy use of joins. Distributed data is stored in a specified sort order on disk, allowing the query planner to make optimizations.

New data is sorted locally when a COPY command is run, but this doesn’t mean that the global sort order is maintained. A VACUUM and ANALYZE command must be run to re-sort the data, rebuild indexes, and update metadata used by the query planner. It can be a resource-intensive process relative to how much data you’ve loaded, and how it’s distributed amongst your existing sort key range. This limits the effective use of Redshift.

ACME Ads MVP

The team defines its initial product. They’ll begin with testing ads in the search results of ACME’s North American website. When customers search for “Soap”, advertisers can bid to have their product show up first. They’ve picked a cost per click (CPC) billing model, where advertisers pay a small sum whenever a user clicks on their ad. To support this, they’ve designed an initial architecture:

They’ve defined the following key events to track:

Impression: Emitted when the ad is rendered on a customer device.
Click: Emitted when the customer clicks on the product shown in the ad.

The shopping web app has a mechanism to deliver these events into an S3 bucket. On an as-needed basis, engineers on the team can manually run a COPY command to refresh the data in the cluster. They do this on a monthly cadence to generate invoices to send to advertisers, and sometimes more frequently to generate ad-hoc reports on ad performance.

Scaling a Real Business

The team is delighted to find that the ads are performing well. Advertisers see a return on their investment, and data shows that the impact on the consumer experience is minimal. The ACME team has scaled the number of advertisers from five in the beta program, to over a thousand. The team struggled with the ad-hoc nature of its MVP analytics stack and has invested heavily in automation:

Loading of event data (pre-impression, clicks, etc.) into Redshift has been automated into an hourly cron that runs an incremental COPY.
Billing is automated to run on an hourly basis, to limit the amount of time an advertiser can be allowed to serve ads after exceeding their allocated budget.

The automated load, in addition to the data generated by the recently hired Product Manager, Business Intelligence Analyst, and Data Scientist, strained the Redshift deployment. It has quickly become the most expensive part of their system, not just in terms of dollars spent, but operator time as well. An overloaded cluster leads to a variety of operational issues that the team must learn to detect and mitigate:

Data delivery is regularly delayed: COPY commands that used to take minutes are taking over an hour when the cluster is under load. Dependent cron jobs over the data need to be updated to handle delays.
The cluster must be resized regularly to keep up with growth: The team often uses Elastic Resize operations, which require a few minutes of downtime and only allow the cluster to increase in size by factors of two. Upgrading node types to better match the workload requires downtime as a new cluster is stood up, and data is copied over.
Ad-hoc workloads interfere with production: Business users write sophisticated queries that accidentally bring the cluster to a standstill. The team puts a complicated WLM policy in place to try and isolate ad-hoc from production, but this is more of an art than a science.

Designing the V2 Analytics Stack

The team understands how to keep the analytics stack limping along at this point, but the cost in both hardware and developer time is too high. They decide to pursue a redesign with the following goals in mind:

Little to no operator intervention required. Developer time is their most valuable asset, and going forward, they want to focus engineering on delivering new features to advertisers, and new experiences to customers.
Scale the analytics stack to 1B events per day and beyond as the program grows in terms of advertisers, and geography.
Enable workload isolation between production services and ad-hoc business intelligence users.

JSON events are queued in Kinesis, consumed by a Firehose that validates their schema against the definition in Glue, then converted into parquet, and finally stored in S3 in hourly folders based on arrival time. Structured data in S3 and a Glue metastore are flexible primitives that end-users and production systems can build on. Both production systems and ad-hoc users can bring their own compute or take advantage of serverless solutions like Athena (the AWS serverless version of Presto) to query over the data with isolation. The operator overhead is low because it relies almost entirely on serverless offerings. This architecture brings some additional benefits:

Scale the read and write workloads independently. Redshift was overprovisioned to handle the mismatch of a continuous write workload at peak hours. A decrease in operating cost was the result of adopting this architecture.
Pay as you go. Firehose, S3, Lambda, and Athena are all billed based on usage. Kinesis is the exception charging per shard hour, which for most deployments would be provisioned for peak load.
Configurable latency. Firehose can dump batches of data in intervals of 1-15 minutes. If low latency is a concern, you can achieve it (see the end for caveats on compaction). We don’t have to worry about an increased write workload impacting reads.
Malformed records won’t block the data ingestion pipeline. Redshift COPY has a maxerror that allows skipping over bad records, but there isn’t any automation to isolate and alert on those records. Firehose writes unsuccessful events to a separate directory and allows processing to continue.

While our new serverless streaming architecture brings a slew of benefits that enable ease of operation and the next level of scale, there are some drawbacks. For one, there is substantially more infrastructure to configure than the single Redshift cluster. How do we implement this in a repeatable fashion?

Architecture as Code with Pulumi

We want repeatable infrastructure that we can reliably develop, test, and deploy. Creating a single Redshift instance through the console was manageable, but our serverless architecture utilizes multiple instances of five AWS services. Serverless solutions frequently require many more granular resources, and thus the complexity of the infrastructure as code increases, and we need more robust tools to manage this complexity - just like we manage software complexity.

Pulumi takes care of the complexity by bringing software development best practices like abstraction, encapsulation, and modularity to infrastructure. You can use design patterns in your favorite programming language to provision modern cloud infrastructure. Component Resources are our tools for encapsulating low-level cloud infrastructure into Architecture as Code that is shareable across teams.

With Pulumi the code we write looks very similar to our architecture diagram:

import { ServerlessDataWarehouse, StreamingInputTableArgs }
 from "./datawarehouse";

// declare our shared schema for clicks and impressions
const columns = [
 {
 name: "id",
 type: "string",
 },
 {
 name: "session_id",
 type: "string",
 },
 {
 name: "event_time",
 type: "string",
 },
];

const impressionsTableName = "impressions";
const clicksTableName = "clicks";

const tableArgs: StreamingInputTableArgs = {
 columns,
 inputStreamShardCount: 1,
 region: "us-west-2",
};

// create two tables with kinesis input streams,
// writing data into hourly partitions in S3.
const dataWarehouse = new ServerlessDataWarehouse("analytics_dw", { isDev })
 .withStreamingInputTable(impressionsTableName, genericTableArgs)
 .withStreamingInputTable(clicksTableName, genericTableArgs);

// Make input streams available to external programs as stack outputs
export const impressionInputStream = impressionsInputStream.name;
export const clickInputStream = clicksInputStream.name;

In a dozen lines of code, we’ve provisioned two of our desired tables, “clicks” and “impressions”, using our serverless streaming input architecture. It takes care of everything, including implicitly creating an arrival time partition scheme with the key “inserted_at”. We export our Kinesis input streams as stack outputs that can be referenced in other projects, such as the instrumentation within our ad server or consumer-facing web app.

While on the surface this Pulumi component is described imperatively, it produces a declarative output in the form of a state file that can be managed locally, in an object store like S3, or by the Pulumi Service backend. Running a ‘pulumi up’ shows that we’ve created 45 AWS resources, and lists our stack outputs to the console.

After putting some sample events into Kinesis, we can use Athena to query our impressions table:

What does ServerlessDataWarehouse look like under the hood? Just like any other typescript class:

export class ServerlessDataWarehouse extends pulumi.ComponentResource {

 public dataWarehouseBucket: aws.s3.Bucket;
 public queryResultsBucket: aws.s3.Bucket;
 public database: aws.glue.CatalogDatabase;
 private tables: { [key: string]: aws.glue.CatalogTable } = {};
 private inputStreams: { [key: string]: aws.kinesis.Stream } = {};

 constructor(name: string,
 args: DataWarehouseArgs = {},
 opts: pulumi.ComponentResourceOptions = {}
 ) {
 super("serverless:data_warehouse", name, opts);

 const bucketArgs: BucketArgs | undefined = args.isDev ?
 { forceDestroy: true } : undefined;

 const dataWarehouseBucket = new aws.s3.Bucket("datawarehouse-bucket",
 bucketArgs, { parent: this });
 const queryResultsBucket = new aws.s3.Bucket("query-results-bucket",
 bucketArgs, { parent: this });

 const database = args.database || new aws.glue.CatalogDatabase(name, {
 name,
 }, { parent: this });


 this.dataWarehouseBucket = dataWarehouseBucket;
 this.queryResultsBucket = queryResultsBucket;
 this.database = database;
 }
}

The ServerlessDataWarehouse class creates a shared S3 data bucket, Athena query results bucket, and Glue database. We’ve exposed a fluent method, .withStreamingInputTable(), that creates the Glue table, Kinesis input stream, Firehose destination stream, and the Lambda cron that registers the hourly partitions. Many of these pieces themselves are also encapsulated into higher-order components.

public withStreamingInputTable(name: string, args: StreamingInputTableArgs):
ServerlessDataWarehouse {
 const {
 partitionKeyName, columns,
 inputStreamShardCount, fileFlushIntervalSeconds
 } = args;
 const partitionKey = partitionKeyName || "inserted_at";

 const partitionKeys: input.glue.CatalogTablePartitionKey[] = [{
 name: partitionKey,
 type: "string",
 }];

 const tableArgs: TableArgs = {
 columns,
 partitionKeys,
 };

 this.withTable(name, tableArgs);

 const streamArgs: InputStreamArgs = {
 destinationBucket: this.dataWarehouseBucket,
 shardCount: inputStreamShardCount,
 databaseName: this.database.name,
 tableName: name,
 fileFlushIntervalSeconds,
 };

 const { inputStream } = new InputStream(`inputstream-${name}`,
 streamArgs, { parent: this });
 this.inputStreams[name] = inputStream;

 const registrarArgs: PartitionRegistrarArgs = {
 database: this.database,
 partitionKey,
 region: args.region,
 dataWarehouseBucket: this.dataWarehouseBucket,
 athenaResultsBucket: this.queryResultsBucket,
 table: name,
 scheduleExpression: args.partitionScheduleExpression,
 };
 const partitionRegistrar = new HourlyPartitionRegistrar(
 `${name}-partitionregistrar`, registrarArgs, { parent: this });

 return this;
}

Our withStreamingInputTable call makes use of another fluent API, .withTable(), that encapsulates the logic around various storage and SerDe details required for storing different formats in glue (Parquet vs. JSON) in a way that Athena can understand:

public withTable(name: string, args: TableArgs): ServerlessDataWarehouse {
 if (this.tables[name]) {
 throw new Error(`Duplicate table! Name: ${name}`);
 }

 const dataFormat = this.validateFormatAndGetDefault(args.dataFormat);

 const table = this.createTable(name, args.columns,
 dataFormat, args.partitionKeys);
 this.tables[name] = table;

 return this;
}

private createTable(name: string,
 columns: input.glue.CatalogTableStorageDescriptorColumn[],
 dataFormat: DataFormat,
 partitionKeys?: input.glue.CatalogTablePartitionKey[]):
 aws.glue.CatalogTable {
 const location = getS3Location(this.dataWarehouseBucket, name);

 const parquetStorageDescriptor = {
 location,
 inputFormat:
 "org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat",
 outputFormat:
 "org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat",
 serDeInfo: {
 parameters: { "serialization.format": "1" },
 name: "ParquetHiveSerDe",
 serializationLibrary:
 "org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe",
 },
 columns,
 };

 const jsonStorageDescriptor = {
 location,
 inputFormat:
 "org.apache.hadoop.mapred.TextInputFormat",
 outputFormat:
 "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat",
 serDeInfo: {
 name: "OpenXJSONSerDe",
 serializationLibrary: "org.openx.data.jsonserde.JsonSerDe",
 },
 columns,
 };

 const storageDescriptor = dataFormat === "JSON" ?
 jsonStorageDescriptor : parquetStorageDescriptor;
 return new aws.glue.CatalogTable(name, {
 name: name,
 databaseName: this.database.name,
 tableType: "EXTERNAL_TABLE",
 storageDescriptor,
 partitionKeys,
 }, { parent: this });
}

Encapsulating core functionality like this enables other analytics use cases, and can act as an escape hatch for more complex ones. Users can create a data warehouse with component owned and managed streaming tables, use .withTable() to enable the same database to own the table, and then write custom external logic to populate that table.

// create a static fact table
const factTableName = "facts";
const factColumns = [
 {
 name: "thing",
 type: "string",
 },
 {
 name: "color",
 type: "string",
 },
];

const factTableArgs: TableArgs = {
 columns: factColumns,
 dataFormat: "JSON",
};

dataWarehouse.withTable("facts", factTableArgs);

// Load a static facts file into the facts table.
const data = `{"thing": "sky", "color": "blue"}\n` +
 `{ "thing": "seattle sky", "color": "grey"}\n` +
 `{ "thing": "oranges", "color": "orange"}`;

const factJSON = new aws.s3.BucketObject("factsFile", {
 bucket: dataWarehouse.dataWarehouseBucket,
 content: data,
 key: `${factTableName}/facts.json`,
});

Here we create a static fact table that never changes. While this population mechanism is simple, you could imagine something much more sophisticated, like a sequence of EMR jobs instead.

Creating a Batch Table API

We now have a very opinionated streaming table API (withStreamingInputTable), and an escape hatch (withTable). Can we provide something for the general batch use case? It is quite a significant problem, but we can come up with a useful API if we constrain it a bit:

Execute user-specified code on a user-specified interval.
The runtime platform is AWS Lambda. Job must execute within Lambda resource constraints for time, memory, CPU, and disk (15 mins, 3 GB, and 500 MB at the time of writing).
The user must specify policies to attach to the lambda execution role for access to resources needed during the job (S3, Athena, and others).

public withBatchInputTable(name: string, args: BatchInputTableArgs):
ServerlessDataWarehouse {
 const {
 columns, partitionKeys, jobFn, scheduleExpression,
 policyARNsToAttach, dataFormat
 } = args;

 const tableArgs: TableArgs = {
 columns,
 partitionKeys,
 dataFormat,
 };

 // create a table
 this.withTable(name, tableArgs);

 const lambdaCronArgs: LambdaCronJobArgs = {
 jobFn,
 scheduleExpression,
 policyARNsToAttach,
 };

 // configure a lambda to execute on a schedule and
 // run the function to write records to the table
 const batchInputJob = new LambdaCronJob(name, lambdaCronArgs,
 { parent: this });

 return this;
}

Our .withBatchInputTable() API creates a table and schedules a lambda that writes data into the table in batches. In our example below, we create a function that queries our clicks and impressions streaming tables over a given hour and writes a JSON file with aggregations. In fifty lines of code, we can publish higher-level statistics about our streaming tables that can be consumed by business users:

// Configure batch input table 'aggregates'
const aggregateTableName = "aggregates";

const aggregateTableColumns = [
 {
 name: "event_type",
 type: "string",
 },
 {
 name: "count",
 type: "int",
 },
 {
 name: "time",
 type: "string",
 },
];

// function that lambda runs.
// writes records to table that summarize clicks and impressions
const aggregationFunction = async (event: EventRuleEvent) => {
 // configure an athena client for our DW
 const athena = require("athena-client");
 const bucketUri = `s3://${athenaResultsBucket.get()}`;
 const clientConfig = {
 bucketUri,
 };
 const awsConfig = {
 region,
 };
 const athenaClient = athena.createClient(clientConfig, awsConfig);
 const date = moment(event.time);
 // query impressions and clicks tables for hourly event counts
 const partitionKey = date.utc().format("YYYY/MM/DD/HH");
 const getAggregateQuery = (table: string) =>
 `select count(*) from ${databaseName.get()}.${table} ` +
 `where inserted_at='${partitionKey}'`;
 const clicksPromise =
 athenaClient.execute(getAggregateQuery(clicksTableName))
 .toPromise();
 const impressionsPromise =
 athenaClient.execute(getAggregateQuery(impressionsTableName))
 .toPromise();

 const clickRows = await clicksPromise;
 const impressionRows = await impressionsPromise;
 // write the event counts to our table in S3 in JSON format
 const clickCount = clickRows.records[0]["_col0"];
 const impressionsCount = impressionRows.records[0]["_col0"];
 const data = `{ "event_type": "${clicksTableName}", `+
 `"count": ${clickCount}, "time": "${partitionKey}" }\n` +
 `{ "event_type": "${impressionsTableName}", ` +
 `"count": ${impressionsCount}, "time": "${partitionKey}"}`;
 const s3Client = new S3();
 await s3Client.putObject({
 Bucket: dwBucket.get(),
 Key: `${aggregateTableName}/${partitionKey}/results.json`,
 Body: data,
 }).promise();
};

//configure resource access policies required by the function
const policyARNsToAttach: pulumi.Input<ARN>[] = [
 aws.iam.ManagedPolicies.AmazonAthenaFullAccess,
 aws.iam.ManagedPolicies.AmazonS3FullAccess,
];

const aggregateTableArgs: BatchInputTableArgs = {
 columns: aggregateTableColumns,
 jobFn: aggregationFunction,
 scheduleExpression,
 policyARNsToAttach,
 dataFormat: "JSON",
};

dataWarehouse.withBatchInputTable(aggregateTableName, aggregateTableArgs);

While Lambda’s limits restrict the utility of this method, the concept is powerful. Pulumi gives you the building blocks to define your abstractions. Instead of Lambda, the execution plane could be ECS, Kubernetes, or EMR.

Automating the Dev Environment

Developing software is an activity in seeking a specific state. For functions with simple inputs and outputs, this is a matter of well-factored code and unit test coverage. You can and should write this sort of test in your Pulumi applications. But often, the state that you’re seeking can’t be validated in unit tests. It raises many questions such as, “Am I stitching these cloud services together in a valid way? Will data flow from input service through to output destination? If I try to tighten down this IAM role, will everything still work?”

Our workflow is to make a change, run a pulumi up, and then verify that data ingested into Kinesis is read out the other end by Athena. A simple shell script using stack outputs and a query in the AWS console are enough to get started here:

#/bin/sh

aws kinesis put-record \
--stream-name $(pulumi stack output impressionInputStream) \
--region $(pulumi config get aws:region) \
--data '{"session_id": "1", "id": "2", "event_time": "2020-01-23T20:38:42.151Z"}' \
--partition-key '1'

This workflow works well enough, but it has a few problems. First of all, it requires a bunch of manual work. We have to run a pulumi up, wait for it to complete, execute this script, logon to the AWS console, execute an Athena query, and wait for the results to come back. We can automate the data ingestion with more Pulumi in our dev stack.


export class EventGenerator extends pulumi.ComponentResource {
 constructor(name: string,
 args: EventGeneratorArgs,
 opts?: pulumi.CustomResourceOptions
 ) {
 super("serverless:event_generator", name, opts);

 const { eventType, inputStreamName } = args;

 // puts a sample record into our input stream
 const eventGenCallback = (event: EventRuleEvent) => {
 const AWS = require("aws-sdk");
 const uuid = require("uuid/v4");
 const kinesis = new AWS.Kinesis();
 const records: any = [];

 const sessionId = uuid();
 const eventId = uuid();
 const record = {
 Data: JSON.stringify({
 id: eventId,
 session_id: sessionId,
 event_time: event.time,
 }),
 PartitionKey: sessionId,
 };
 records.push(record);

 kinesis.putRecords({
 Records: records,
 StreamName: inputStreamName.get(),
 }, (err: any) => {
 if (err) {
 console.error(err);
 }
 });
 };

 const lambdaCronArgs: LambdaCronJobArgs = {
 jobFn: eventGenCallback,
 scheduleExpression: "rate(1 minute)",
 policyARNsToAttach: [
 aws.iam.ManagedPolicies.AmazonKinesisFullAccess,
 ],
 };

 // schedule a lambda to run our function once a minute.
 const eventGenerator = new LambdaCronJob(`${eventType}-eventGenerator`,
 lambdaCronArgs, { parent: this });
 }
}

This helper creates a Lambda that publishes sample events to Kinesis on a minutely cron. It doesn’t completely automate our validation, but it’s certainly a step closer. Now when we run an incremental update, data automatically propagates through the system. We can use the AWS console to execute Athena queries and observe the output.

Incremental updates in our dev loop are fast, but when dealing with AWS services, entities like IAM roles and Glue schemas can be cached by services like Firehose. It’s entirely possible to make an update that appears to work for a few minutes, just to have it fail minutes later when some implementation detail clears its cache.

To bring 100% automation and certainty, we can write an isolated integration test that stands up new infrastructure, executes our tests, and then tears everything down.

jest.setTimeout(360000);

let runner: PulumiRunner;
const region = "us-west-2";

beforeAll(async () => {
 const config: { [key: string]: string } = {
 "aws:region": region,
 "aws-ts-serverless-datawarehouse:dev": "true",
 };

 const pulumiProjDir = resolve("./");
 runner = new PulumiRunner(config, pulumiProjDir);
 const setupResult = await runner.setup();
 if (!setupResult.success) {
 throw new Error(`Pulumi setup failed, aborting: ${setupResult.error}`);
 }
});

afterAll(async () => {
 const teardownResult = await runner.teardown();
 if (!teardownResult.success) {
 throw new Error(
 `Teardown failed. Test stack has leaked: ${teardownResult.error}`
 );
 }
});

test("WithStreamingInput integrtion test", async () => {
 // assert we see the expected stack outputs
 expect(runner.getStackOutputKeys().length).toBe(6);
 const db = runner.getStackOutput("databaseName");
 const clickTable = runner.getStackOutput("clickTableName");
 const impressionTable = runner.getStackOutput("impressionTableName");
 const bucket = runner.getStackOutput("athenaResultsBucket");

 // query our tables on a retry loop until we find records or timeout and fail
 const clickPromise = verifyRecordsInTable(db, clickTable, bucket);
 const impressionPromise = verifyRecordsInTable(db, impressionTable, bucket);

 const [clickTableHasRecords, impressionTableHasRecords] = await
 Promise.all([clickPromise, impressionPromise]);

 expect(clickTableHasRecords).toBe(true);
 expect(impressionTableHasRecords).toBe(true);
});

This test gives us a great deal of confidence that our changes are valid. While it certainly does a lot, it’s a little slow. We can run this suite with npm run test:int & tput bel to notify us upon completion while we work in the background, but four minutes might not meet the latency bar for core inner dev loop.

Here we’ve seen three different ways to use Pulumi to automate and improve the inner dev loop. Simple scripting with stack outputs to get started, automation of that script using Pulumi, and full blow integration testing to bring 100% confidence in tricky changes. All three are tools in our arsenal to be applied appropriately.

Wrapping Up

ACME re-evaluated their analytics architecture and shipped the next-generation serverless streaming solution that can take them to the next click-stop, and well through the next order of magnitude. Along the way, they learned how to reason about streaming data and examined the underlying assumptions about time and completeness of data in their existing system. ACME developed its streaming architecture using Pulumi and learned how to automate and create reproducible developer environments along the way. The outcome isn’t just an instance of the serverless streaming architecture, but a component library that implements best practices. That investment in time is bundled up and shipped in a format that the next generation of incubators at ACME can pick up and run with.

If you’d like to get started delivering best practices in infrastructure across your organization, get started with Pulumi free today. Clone and run the reference implementation of the Serverless Data Warehouse to build your own serverless analytics stack.

Architectural Caveats

Like every architecture, the serverless streaming model has limitations and pitfalls.

Processing Time vs. Event Time: Our data is now ordered and indexed by when it’s processed as opposed to when the events occur. We need to reason about data arriving late and make sure that our queries and dependent systems account for spikes in latency. For a detailed primer, see Streaming 101.
Data Freshness & Compaction: While it is possible to emit files by the minute to achieve low latency, it can lead to performance issues. In practice, this sort of setup can cause an explosion of tiny files that degrade the performance of query engines. See this article on optimizing file sizes for Athena for more details. A standard solution is to run an hourly compaction job that rewrites data into optimally sized chunks, leaving only the most recent hour or two of data in smaller files.
Cross Account Athena: Due to Glue’s limitations, you cannot grant cross-account access for Athena directly. There are workarounds like temporary credentials and using assume role.
Service Limits: Just as with Redshift, nothing is truly a black box. Athena has a maximum number of concurrent queries, and S3 has default limits on the rate at which you can list and get objects. As you scale, make sure you understand your service limits and reach out to AWS support to proactively increase limits where appropriate.
Query Translation: If you’re migrating an existing system from Redshift to Athena, the queries won’t quite map 1:1. For one, there’s the need to restructure queries to limit first on arrival time, and then narrow down on specific event time. And some of the traditional SQL operations like ORDER BY and GROUP BY will perform differently. See the section on query optimization for more details.
Unbounded Athena Queries: Athena charges $5 per TB scanned during a query. Most business users aren’t used to the concept of pay per query. It’s quite easy to issue a ‘select *’ query that can cost a great deal of money. Guard rails like a middleware that parses queries and ensures usage of a partition key may be necessary to control cost if exposing Athena as an interface to a broader audience.

Provisioned Concurrency: Avoiding Cold Starts in AWS Lambda

Mikhail Shilkov — Thu, 19 Dec 2019 00:00:00 +0000

AWS Lambda cold starts (the time it takes for AWS to assign a worker to a request) are a major frustration point of many serverless programmers. In this article, we will take a look at the problem of latency-critical serverless applications, and how Provisioned Concurrency impacts the status-quo.

Concurrency Model of AWS Lambda

Despite being serverless, AWS Lambda uses lightweight containers to process incoming requests. Every container, or worker, can process only a single request at any given time.

Overlapping executions land on separate workers

Because of this, the number of concurrent requests defines the number of required workers that a specific AWS Lambda function needs to serve a response at any given moment.

Cold Starts

How does AWS know how many workers it needs to run for a given function? Well, it doesn’t know in advance. AWS allocates new workers on-demand as the Lambda gets invoked.

Whenever Lambda receives a request but it has no idle workers, the control plane assigns a new generic worker to it. The worker then has to download the custom code or binaries of your Lambda and load them into memory before it can service the request. This process takes time, which significantly increases response latency.

The issue of sporadically slow responses caused by the need to increase the pool of workers is known as Cold Start. Cold starts are consistently the top concern about the applicability of serverless technologies to latency-sensitive workloads. There are numerous articles about the problem, including many articles I have written in the Cold Starts section on my website

Warming

Cold starts don’t occur for the majority of requests because AWS Lambda reuses workers between subsequent invocations. However, if a particular worker is idle for too long (usually, several minutes), AWS may decide to recycle and return it to the generic pool.

A trick known as Lambda Warmers uses kept-alive workers to reduce the frequency of cold starts. The idea is to have a CloudWatch timer that fires every few minutes and sends N parallel requests to the target Lambda. If all those requests land at the same time, AWS has to provision at least N workers to process them. The actual operation doesn’t have to do any useful work, but it resets the recycling timer back to zero.

There are a few drawbacks with this approach:

If a valid request comes at the same time as warming requests, it might hit a cold start.
Extra logic is needed in the Lambda code to detect warming requests and short-circuit them instead of doing useful work.
CloudWatch rules require additional setup and management.
The result is still best-effort: Lambda still recycles workers from time to time.

You can find more details about how Pulumi can help with some of these issues in AWS Lambda Warmer as Pulumi Component.

Provisioned Concurrency

At re:Invent 2019, AWS introduced Lambda Provisioned Concurrency—a feature to work around cold starts. The base concurrency model doesn’t change. However, you can request a given number of workers to be always-warm and dedicated to a specific Lambda.

Here is an example of configuring the provisioned concurrency with Pulumi in TypeScript:

const lambda = new aws.lambda.Function("mylambda", {
 code: new pulumi.asset.FileArchive("./app"),
 handler: "handler",
 role: role.arn,
 publish: true,
});

const fixed = new aws.lambda.ProvisionedConcurrencyConfig("fixed-concurrency", {
 functionName: lambda.name,
 qualifier: lambda.version,
 provisionedConcurrentExecutions: 2,
});

The snippet sets the provisioned concurrency for mylambda to a fixed value of 2. Note that concurrency is provisioned per Lambda version, and it can’t be set for the $LATEST version alias. Therefore, I set the publish attribute of my Lambda to true and reference a specific version of the Lambda in provisioning.

Dynamic Provisioned Concurrency

A fixed level of provisioned concurrency works well for stable workloads.

Fixed provisioned concurrency for uniform workloads

However, many workloads fluctuate a lot. Extreme elasticity and lack of configuration parameters have always been the essential benefits of AWS Lambda. It works great if you can tolerate the cold starts that come during scale-out. If not, you can explore more advanced scenarios for provisioning concurrency dynamically.

Instead of choosing a permanently fixed value, you can configure provisioned concurrency to autoscale. The first required component is the autoscaling target:

const resourceId = pulumi.interpolate`function:${lambda.name}:${lambda.version}`;
const target = new aws.appautoscaling.Target("target", {
 resourceId,
 serviceNamespace: "lambda",
 scalableDimension: "lambda:function:ProvisionedConcurrency",
 minCapacity: 1,
 maxCapacity: 10,
});

The second component is the autoscaling policy, which defines the conditions when scaling starts. There are two important ways to adapt your concurrency provisioning to traffic patterns.

Scheduled profile

Quite often, increases in request rates are partially predictable. For example, usage increases during business hours and decreases at night.

Provisioned concurrency matches predictable workload changes

The following snippet sets two scheduled rules that switch between two levels of provisioned concurrency every day.

function scheduledConcurrency(name: string, cron: string, capacity: number) {
 return new aws.appautoscaling.ScheduledAction(`schedule-${name}`, {
 resourceId,
 serviceNamespace: "lambda",
 scalableDimension: "lambda:function:ProvisionedConcurrency",
 scalableTargetAction: {
 minCapacity: capacity,
 maxCapacity: capacity,
 },
 schedule: `cron(${cron})`,
 }, { dependsOn: [target]});
}

scheduledConcurrency("day", "0 8 * * ? *", 10);
scheduledConcurrency("night", "0 18 * * ? *", 2);

The example defines a helper function and calls it twice to set two schedules with different parameters.

Autoscaling based on utilization

If your workload pattern is less predictable, you can configure autoscaling for provisioned concurrency based on the measured utilization.

Provisioned concurrency matches the variation in workload

Here is a basic example of a dynamic scaling policy.

const scaledConcurrency = new aws.appautoscaling.Policy("autoscaling", {
 resourceId,
 serviceNamespace: "lambda",
 scalableDimension: "lambda:function:ProvisionedConcurrency",
 policyType: "TargetTrackingScaling",
 targetTrackingScalingPolicyConfiguration: {
 targetValue: 0.8,
 predefinedMetricSpecification: {
 predefinedMetricType: "LambdaProvisionedConcurrencyUtilization",
 },
 },
}, { dependsOn: [target]});

Currently, there are issues with autoscaling based on the metrics, which makes provisioned concurrency scale less effectively than what the chart above shows. Hopefully, Amazon will fix these issues soon.

AWS Pricing

The pricing information in this section is outdated. For the most current AWS Lambda pricing, including Provisioned Concurrency costs, please refer to the official AWS Lambda pricing page.

While hand-crafted Lambda warmers are virtually free, provisioned concurrency can be costly. The pricing model for Provisioned Concurrency differs from the standard on-demand Lambda model: Instead of purely per-call billing, AWS charges per hour for provisioned capacity.

You pay for the Provisioned Concurrency you configure, even if a worker handled zero requests, plus a reduced rate for actual usage of those pre-provisioned instances.

Comparison of the cost of a 1GB worker for two billing models

The equilibrium point is at 60% utilization. Note that because the billed duration is rounded up to the nearest 100 ms for each execution, the utilization is not limited to 100%. A series of sequential executions can be processed by a single worker. If each execution is 10 ms, the charge is still 100 ms, and the total utilization can be as high as 1000% in terms of the chart above!

Finally, be careful to clean up your resources after any experiments: Leaving running workers with provisioned concurrency can be expensive! Using pulumi destroy removes resources after you finish experimenting.

Conclusion

Provisioned concurrency brings the long-awaited solution to cold starts in AWS Lambda. However, it comes at a cost in terms of a direct impact on billing and management overhead to provision concurrency at optimal levels.

For high-load functions where utilization is continuously above a known level of requests, it makes sense to set the provisioned concurrency to that level to save money and have the guarantee of warm workers.

If your Lambda hosts a latency-sensitive API, especially with runtimes like Java and .NET, you should strive to find the right balance between the percentage of requests that result in a cold start and the money spent on concurrency. Autoscaling should help once AWS has fixed the initial glitches that slipped into their current services.

If you want to try this for yourself, we’ve updated the Serverless App example to show off the scenario of configurable AWS Lambda provisioned concurrency.

Pulumi Watch: Fast Inner Loop Development for Infrastructure

Luke Hoban — Mon, 02 Dec 2019 00:00:00 +0000

A big part of our vision with Pulumi is to bring application developers and infrastructure teams closer together in the cloud. That includes both providing infrastructure teams with better software engineering tools, as well as providing developers with easier access to cloud infrastructure. We are often inspired by looking at great software engineering experiences in other development stacks and applying them to the cloud infrastructure space. Whether it be general-purpose languages and rich IDEs, testing and package management, or components and rich APIs, at Pulumi, we’ve repeatedly applied successful development tools and practices to the challenges of building and scaling modern cloud infrastructure.

As we look across other areas of successful application development, one of the most critical drivers of creativity and productivity is the speed of iteration. When developers can make a change and see or experience the results nearly instantaneously, they can explore solutions quicker, can take more creative risks (and thus learn more), and can ultimately iterate more quickly toward their desired result. From fast build times as a driving force in the design of the Go language, to the DevTools in Chrome, to Edit-and-Continue in Visual Studio, to the watch modes on tools like TypeScript, this idea of enabling quick or near-instantaneous feedback as code changes are made is a driving force behind many of the most successful developer experiences across a wide variety of software development environments.

Introducing Pulumi Watch

In Pulumi 1.5, we released a preview of a new pulumi watch command. This command provides a mode for developing cloud infrastructure that speeds up the rate of iteration and allows cloud developers to focus on their code and infrastructure, instead of on the mechanics of their deployments. With pulumi watch, cloud infrastructure feels like it’s truly at your fingertips.

The video above shows an example of working with some S3 resources. Every time I hit “save” in my editor, pulumi watch automatically deploys any changes to my infrastructure. If I add a new BucketObject resources and hit save, the object appears in S3. If I add a loop around it and hit save, 10 new objects appear and the original object is removed. And if I add an event handler, the 5 Lambda and IAM resources needed are deployed in seconds and are hooked up to the bucket. You don’t have to switch back and forth between an editor and a terminal, and you don’t have to go through heavyweight infrastructure deployment tools like Terraform or wait on slow external services like CloudFormation or CI/CD. Instead, changes appear directly in the cloud within seconds of hitting enter. You can try things out, experiment, learn, and rapidly iterate on your cloud infrastructure.

This new watch mode isn’t a replacement for a standard pulumi preview/pulumi update. For production infrastructure deployments, it is critical to understand precisely what changes will happen before applying them, to integrate with CI/CD systems, to be able to do targeted updates, and to apply policy to guard the production environment against misconfiguration. But that same level of care isn’t necessarily the right way to approach building new features in a development stack. And it’s in these development environments that pulumi watch opens up a new experience for infrastructure development. Although they are shorter periods relative to the full life of an infrastructure deployment, they are the majority of the time that creativity and creation is applied to cloud infrastructure, and thus represent a key opportunity to empower developers.

Not all cloud resources can be deployed or updated in seconds - but many of the most popular modern cloud primitives are built on platforms that enable and encourage quick iteration - and in particular both serverless and Kubernetes. Watch mode can be used during the development of any infrastructure, but can really shine in these areas where the cloud platforms natively support fast iteration.

Kubernetes Infrastructure + Applications

For Kubernetes, pulumi watch allows us to deploy changes to our cluster automatically as we author our Kubernetes resources. Change the number of replicas for our Deployment and hit save and the new pods scale-out. Add a Service to expose the deployment and public endpoint is available immediately. This pairs beautifully with the recently released Pulumi Kubernetes Extensions (kx) library, which allows you to build Kubernetes applications with just a few lines of code - avoiding a lot of the boilerplate that slows down typical Kubernetes development when working with YAML. We can see this combination in action in the video below:

Both of the previous videos also highlight another interesting element of modern cloud infrastructure development. In both the S3 example and the Kubernetes example, the Pulumi program deployed not just the infrastructure, but also the application code and assets. By authoring and delivering both infrastructure and application code together, we remove another aspect of the silos that separate developers from infrastructure, and in doing so, unlock even faster iteration. This is a pattern that Pulumi enables in a wide variety of unique ways, from Lambdas as lambdas, to build-and-push of Docker images, to deploying assets for static sites.

From Zero to Cloud Application in 90 Seconds

We can bring these ideas together, and show another example of what’s possible with watch mode using a serverless application example. As with so many things these days, a tweet inspired this example:

I want a website that does one thing: displays the number of times it’s been viewed. What is the shortest number of steps you can think of to get it running in the cloud?

— Slava Akhmechet (@spakhm)

Indeed, such simple tasks are often surprisingly complex and cumbersome with today’s infrastructure tools. But with pulumi watch and the Pulumi Cloud Framework, we can go from nothing to having this application deployed in AWS (or Azure!) in under 90 seconds.

Conclusion

When using pulumi watch, working with cloud infrastructure feels so different - it feels malleable, it feels accessible, and it feels like it’s right at your fingertips. By building on the same robust incremental deployment infrastructure as the rest of Pulumi, you get the same access to any cloud, and to a variety of languages and all the other benefits of Pulumi’s modern infrastructure as code tooling. Pulumi watch triggers a deploy on any save - including both of files in the Pulumi program itself, as well as other assets that the Pulumi program may reference like application source code or static assets. Pulumi watch also displays runtime logs for supported platforms (Lambda, ECS, and Cloudwatch currently, more coming soon), and can be paired with another new feature, Pulumi Query, to get customized status about the runtime environment during rapid iteration.

The pulumi watch command is available in preview now when the PULUMI_EXPERIMENTAL environment variable is set to true. We are just getting started - and see many more opportunities to tighten the iteration cycle further, to support more logging capabilities, to integrate more deeply with query, and to be applied to more modern infrastructure scenarios.

Give it a try today and drop us a note about where you’d like to see this go for Pulumi 2.0!

Azure Functions on Kubernetes with KEDA

Mikhail Shilkov — Thu, 10 Oct 2019 00:00:00 +0000

Azure Functions is a managed service for serverless applications in the Azure cloud. More broadly, Azure Functions is a runtime with multiple hosting possibilities. KEDA (Kubernetes-based Event-Driven Autoscaling) is an emerging option to host this runtime in Kubernetes.

In the first part of this post, I compare KEDA with cloud-based scaling and outline the required components. In the second part, I define infrastructure as code to deploy a sample KEDA application to an Azure Kubernetes Service (AKS) cluster.

The result is a fully working example and a high-level idea of how it works. Kubernetes expertise is not required!

Autoscaling Primer

When you deploy an Azure Function, it runs within the Azure Functions runtime. The runtime is a host process which knows how to pull events from an event source (defined by the function trigger) and pass those to your function:

However, one instance of runtime rarely provides adequate processing capacity. If you only get one message per day, having an instance always running is wasteful. If you get thousands of events per second, one instance won’t be able to process all of them.

Automatic horizontal scaling solves the problem. At any point in time, several identical workers are crunching the events. The number N is optimized continuously to fit the current workload by adding new workers and removing underutilized ones.

Components of an automatically scaled application

Instance Provisioner is an extra component in the auto-scaled system. It monitors the stream of metrics from the event source and decides to add or remove workers. A massive standby pool of idle generic workers provides the workforce. Such a generic worker pulls the artifact of the assigned function, plugs it into the runtime, and starts processing events.

Azure Functions Consumption Plan

Consumption Plan is the serverless hosting option where Azure manages all the scaling components internally. Let’s consider an example of an Azure Function triggered by a Storage Queue:

Azure Functions Consumption Plan

The deployment artifact is just the code packaged as a zip archive and uploaded to Blob Storage. Scale Controller is an internal Azure component that observes the target queue and allocates Function App instances based on the queue length fluctuations. Each instance bootstraps itself with a zip file, connects to the queue, and pulls messages to process.

The cloud provider manages all the components of the system, so developers can focus on writing business logic code. It can be as simple as a JavaScript callback:

queue.onEvent("MyHandler", async (context, msg) => {
 console.log("New message: " + JSON.stringify(msg));
});

KEDA

Over the last few years, Kubernetes has gained traction across many industries. KEDA is provides a way to design and run event-driven applications inside a Kubernetes cluster. KEDA implements the autoscaling components in terms of Kubernetes tools.

The target Function App is packaged together with the Azure Functions runtime into a custom Docker image and published to a registry. A Kubernetes deployment utilizes that image and configures the parameters to connect it to the target event source (for instance, a queue):

The application can then run on one or many instances, or pods in Kubernetes terms. Kubernetes has a built-in component to scale out the pods: Horizontal Pod Autoscaler (HPA). By default, HPA makes scaling decisions based on the processor utilization of existing pods. CPU turns out to be a poor metric for event-driven applications: many workloads are not CPU-bound, so the scale-out won’t be aggressive enough to keep the queue empty. Therefore, KEDA introduces a ScaledObject—a custom resource which pulls metric values from the event source and feeds them as a custom metric to HPA:

Kubernetes-based Event-Driven Autoscaling

At the time of the initial KEDA preview announcement and until Kubernetes version 1.16, Horizontal Pod Autoscaler wasn’t able to scale a deployment down to zero pods. Therefore, KEDA includes an extra controller to disable the deployment if the event source is empty and re-enable it when new events come in. This duty might be delegated back to HPA in the future versions.

Now, when we know what KEDA is and how it works, it’s time to deploy a Function App!

Deploy a Function App with KEDA and Pulumi

Here is the list of the components required to run a Function App in Kubernetes with KEDA:

Azure Kubernetes Service cluster
Azure Container Registry
Helm chart keda-edge with KEDA service and custom resource definition
Azure Storage Queue as a sample event source
Docker image of the queue processor application
Deployment of this image
Scaled Object custom resource

All these components can be defined and deployed within a single Pulumi program. Below I highlight the main blocks of the program. Navigate to the following section to look at a short and reusable component.

AKS Cluster

KEDA can run on any Kubernetes cluster, but I choose to do so on managed AKS. My custom AksCluster component defines a cluster, including the required Active Directory and networking configuration. It makes multiple assumptions, so I only need to specify the main properties:

const resourceGroup = new azure.core.ResourceGroup("keda-sample");

const aks = new AksCluster("keda-cluster", {
 resourceGroupName: resourceGroup.name,
 kubernetesVersion: "1.13.5",
 vmSize: "Standard_B2s",
 vmCount: 3,
});

aks.cluster now has all the required output values, for instance, aks.cluster.kubeConfigRaw configuration.

Shared Cluster Components

The next group of components needs to be deployed just once.

Azure Container Registry is not part of the AKS cluster. Its purpose is to host Docker images of applications.

const registry = new azure.containerservice.Registry("registry", {
 resourceGroupName: args.resourceGroup.name,
 adminEnabled: true,
 sku: "Premium",
});

A Helm chart kedacore/keda-edge deploys the KEDA service and the ScaledObject custom resource definition.

const keda = new k8s.helm.v2.Chart("keda-edge", {
 repo: "kedacore",
 chart: "keda-edge",
 version: "0.0.1-2019.07.24.21.37.42-8ffd9a3",
 values: {
 logLevel: "debug",
 },
}, { providers: { kubernetes: aks.provider } });

Event Source

KEDA supports multiple types of event sources, and the current list is available here. An event source is supported if there is a scaler which knows how to pull metrics out of it and turn them into a custom metric for HPA.

My example uses an Azure Storage Queue as the event source:

const storageAccount = new azure.storage.Account("kedasa", {
 resourceGroupName: resourceGroup.name,
 accountTier: "Standard",
 accountReplicationType: "LRS",
});

const queue = new azure.storage.Queue("kedaqueue", {
 storageAccountName: storageAccount.name,
});

Docker Image

The same Pulumi program is capable of building a Docker image and uploading it to the Container Registry.

const dockerImage = new docker.Image("image", {
 imageName: pulumi.interpolate`${registry.loginServer}/${args.queue.name}:v1.0.0`,
 build: {
 context: "./functionapp",
 },
 registry: {
 server: registry.loginServer,
 username: registry.adminUsername,
 password: registry.adminPassword,
 },
});

The image refers to the folder with a Dockerfile in it and uses the registry variables to fill the credentials.

Deployment

Now, we can define a Deployment which uses the Docker image to run our Function App on Kubernetes pods.

const appLabels = { app: name };
const deployment = new k8s.apps.v1.Deployment(name, {
 apiVersion: "apps/v1",
 kind: "Deployment",
 metadata: {
 labels: appLabels,
 },
 spec: {
 selector: { matchLabels: appLabels },
 template: {
 metadata: {
 labels: appLabels,
 },
 spec: {
 containers: [{
 name,
 image: dockerImage.imageName,
 env: [{ name: "queuename", value: args.queue.name }],
 envFrom: [{ secretRef: {name: secretQueue.metadata.name } }],
 }],
 imagePullSecrets: [{ name: args.service.registrySecretName }],
 },
 },
 },
}, { provider: aks.provider });

The deployment refers to the secret value which stores the connection string to our target Storage Queue:

const secretQueue = new k8s.core.v1.Secret("queue-secret", {
 data: {
 queueConnectionString:
 args.storageAccount.primaryConnectionString.apply(c => Buffer.from(c).toString("base64")),
 },
}, { provider: aks.provider });

Scaled Object

Finally, we need to deploy an instance of the ScaledObject custom resource, which takes care of feeding the metrics from the queue to the Horizontal Pod Autoscaler.

const scaledObject = new k8s.apiextensions.CustomResource("scaledobject", {
 apiVersion: "keda.k8s.io/v1alpha1",
 kind: "ScaledObject",
 metadata: {
 labels: { deploymentName: name },
 },
 spec: {
 scaleTargetRef: { deploymentName: name },
 triggers: [{
 type: "azure-queue",
 metadata: {
 type: "queueTrigger",
 connection: "queueConnectionString",
 queueName: args.queue.name,
 name: "myQueueItem",
 },
 }],
 },
}, { provider: aks.provider });

Reusable Components

We can simplify the program further by creating reusable Pulumi components for a Cluster, a Service, and a Function app. After those are done, here is all the code required to deploy a Kubernetes cluster with a Function App and KEDA.

// Create an AKS K8s cluster
const aks = new AksCluster("keda-cluster", {
 resourceGroupName: resourceGroup.name,
 kubernetesVersion: "1.13.5",
 vmSize: "Standard_B2s",
 vmCount: 3,
});

// Deploy shared components of KEDA (container registry, kedacore/keda-edge Helm chart)
const service = new KedaService("keda-edge", {
 resourceGroup,
 k8sProvider: aks.provider,
});

// Deploy a Function App which subscribes to the Storage Queue
const app = new KedaStorageQueueHandler("queue-handler", {
 resourceGroup,
 storageAccount,
 queue,
 service,
 path: "./functionapp",
});

You can find the full code of the components, the sample program, and steps to run it in this example.

Conclusion

If your company is betting on Kubernetes as the standard application platform and you see high value in event-driven fine-grained applications, KEDA might be an exciting option for the future. As of March 31, 2020, Kubernetes Event-driven Autoscaling (KEDA) has joined the CNCF as an official sandbox project.

Pulumi is a great tool to get your KEDA application running quickly and effortlessly. You can combine Docker image creation, Kubernetes cluster provisioning, Helm chart installation, and KEDA deployment in a single program, written in your favorite general-purpose language.

Get started with Azure Kubernetes Service (AKS) Cluster and Azure Functions with KEDA.

Ten Pearls With Azure Functions in Pulumi

Mikhail Shilkov — Wed, 21 Aug 2019 00:00:00 +0000

In this post, we’ll take a look at 10 “pearls”—bite-sized code snippets—that demonstrate using Pulumi to build serverless applications with Azure Functions and infrastructure as code. These pearls are organized into four categories, each demonstrating a unique scenario:

Function App Deployment: Deploy an existing Azure Functions application using infrastructure as code.
HTTP Functions as Callbacks: Mix JavaScript or TypeScript functions with your infrastructure definition to produce strongly-typed, self-contained, serverless HTTP endpoints.
Cloud Event Handling: Leverage a variety of event sources available to Azure Functions with lightweight event handlers.
Data Flows with Function Bindings: Take advantage of function bindings—declarative connectors to Azure services.

Our Azure Functions pearls are:

Function App Deployment

Deploy a Function App written in .NET or any other supported runtime
Configure Functions to run on an Elastic Premium Plan

HTTP Functions as Callbacks

Define Node.js Functions as inline callbacks
Implement REST APIs as multiple Functions
“Warm” the Functions to avoid Cold Starts

Cloud Event Handling

Process events from Azure Event Hub
Subscribe to Blob creation with Azure Event Grid
Run a Function every time an Azure resource is modified

Data Flows with Function Bindings

Push a message to a Storage Queue with an output binding
Pull a Storage Table row with an input binding

Function App Deployment

Azure Functions can be written in many languages, and Pulumi supports whatever choice you make. You can take any existing serverless application and deploy it using infrastructure-as-code.

1. Deploy a .NET Function App

[ Runnable Example ]

Many Function Apps are .NET applications created with native tooling like Visual Studio or the Functions CLI. Pulumi can simply deploy such an application in only a few lines of JavaScript:

const dotnetApp = new azure.appservice.ArchiveFunctionApp("http-dotnet", {
 resourceGroupName: resourceGroup.name,
 archive: new pulumi.asset.FileArchive("./app/bin/Debug/netcoreapp2.1/publish"),
 appSettings: { runtime: ".NET" },
});

There are only four things required:

the Function App name (“http-dotnet”)
the resource group it belongs to
the path to the compiled .NET assemblies, and
the desired runtime

Pulumi takes care of the rest for you. It handles the following tasks:

Creating a Storage Account and a Blob Container
Zipping up the binaries and uploading them to the blob container
Calculating a SAS token
Preparing a Consumption Plan and a Function App using this Consumption Plan
Configuring the required application settings, including a reference to the zip archive with the SAS token

While a few values are required, you are not restricted to the default behavior and can change any setting.

2. Run Functions using an Elastic Premium Plan

While the Consumption Plan is the ultimate serverless option, there are other ways to host Azure Functions. Azure also offers the Premium Plan: a combination of the power and guarantees of a fixed App Service Plan with the elasticity of Consumption.

If you want to take advantage of a Premium Plan, go ahead and define it as a Pulumi resource, then link to it in the Function App definition:

const premiumPlan = new azure.appservice.Plan("my-premium", {
 resourceGroupName: resourceGroup.name,
 sku: {
 tier: "Premium",
 size: "EP1",
 },
 maximumElasticWorkerCount: 20,
});

const javaApp = new azure.appservice.ArchiveFunctionApp("http-java", {
 resourceGroupName: resourceGroup.name,
 plan: premiumPlan,
 archive: new pulumi.asset.FileArchive("./java/target/functions/fabrikam"),
 appSettings: { runtime: "java" },
});

In this example, I deployed a Java application. The Premium Plan has a couple of configuration knobs: the instance size and the maximum scale-out limit.

HTTP Functions as Callbacks

Node.js is another runtime supported by Azure Functions. Like before, you can use the ArchiveFunctionApp class to deploy the application from an external folder. However, Pulumi’s Node.js SDK provides a way to mix the code of your functions directly into your infrastructure definition.

3. Define Node.js Functions as Inline Callbacks

[ Runnable Example ]

A TypeScript or a JavaScript function becomes an Azure Function deployed to the cloud:

const greeting = new azure.appservice.HttpEventSubscription('greeting', {
 resourceGroupName: resourceGroup.name,
 callback: async (context, req) => {
 return {
 status: 200,
 body: `Hello ${req.query['name'] || 'World'}!`,
 };
 }
});

export const url = greeting.url;

You are free to use any NPM module inside the callback, and the dependencies will be automatically packaged into the deployment artifact. Alternatively, you can extract the callback function into a separate file or package and import it from your infrastructure code.

While mixing infrastructure and application code in the same file may seem counterintuitive, it provides many benefits:

Combined code binaries (data plane) and infrastructure (control plane) as a single unit of deployment
Eliminate the need for boilerplate configuration like host.json and function.json files
Robust typing out-of-the-box: For instance, you can flawlessly “dot into” the content and req object above.

You can read more about the motivation in Serverless as Simple Callbacks with Pulumi and Azure Functions.

The previous example deployed a Function App with a single Function. However, Azure supports applications with multiple Functions bundled together.

4. REST APIs as Multiple Functions

It’s common to use Azure Functions for implementing RESTful APIs. As an API may consist of multiple endpoints, we may need to combine several HTTP Functions into a single deployment unit.

To achieve that, we define an HttpFunction object per Azure Function, each with its callback and settings. Then, we pass an array of these objects into a MultiCallbackFunctionApp constructor. Each Function definition may have a specific route and HTTP methods to handle:

const get = new azure.appservice.HttpFunction("Read", {
 route: "items",
 methods: ["GET"],
 callback: async (context, request) => {
 const items = await repository.list();
 return { status: 200, body: items };
 },
});

const post = new azure.appservice.HttpFunction("Add", {
 route: "items",
 methods: ["POST"],
 callback: async (context, request) => {
 const id = await repository.add(request.body);
 return { status: 201, body: { id } };
 },
});

const app = new azure.appservice.MultiCallbackFunctionApp("multi-app", {
 resourceGroupName: resourceGroup.name,
 functions: [get, post],
});

You can combine as many Functions as you want, including functions of different types, as shown in the next example.

5. Function “Warming” with a Timer Function

[ Runnable Example ]

Scheduled jobs are another frequent use case for serverless functions. It’s possible to define a cron expression and get the code executed at designated intervals.

The Consumption Plan disposes a worker if no Function runs in about 20 minutes. After disposal, the next execution will cause a cold start, and the response latency, will increase.

In order to avoid this disposal, we can combine a target HTTP Function with a Timer Function. The body of the Timer Function is empty, since its sole purpose is to trigger the Function App and keep the worker “warm”:

const warmer = new azure.appservice.TimerFunction("warmer", {
 schedule: "0 */5 * * * *",
 callback: async () => {},
});

const http = new azure.appservice.HttpFunction("hello", {
 callback: async (context, req) => {
 return {
 status: 200,
 body: "Hello World!",
 };
 }
});

const app = new azure.appservice.MultiCallbackFunctionApp("always-warm-app", {
 resourceGroupName: resourceGroup.name,
 functions: [http, warmer],
});

It’s easy to imagine a custom component WarmedFunctionApp which appends a standard Timer Function to an array of Functions passed to its constructor.

Cloud Event Handling

While HTTP is a widespread use case, Azure Functions support many other trigger types too. The previous post featured Storage Queues and ServiceBus Topics. Pulumi supports Timers, Events Hubs, Event Grid, Storage Blobs, Service Bus Queues, and Cosmos DB Change Feed events, too! Let’s see how that looks using Azure Event Hubs.

6. Process Events from Azure Event Hubs

[ Runnable Example ]

Azure Event Hubs is a fully-managed log-based messaging service comparable to Apache Kafka. In contrast to a self-hosted Kafka cluster, it only takes a few lines of JavaScript to create an Event Hub and start processing events:

const eventHub = new eventhub.EventHub("my-hub", {
 resourceGroupName: resourceGroup.name,
 namespaceName: namespace.name,
 partitionCount: 2,
 messageRetention: 7,
});

eventHub.onEvent("MyHubEvent", async (context, msg) => {
 console.log("Event Hub message: " + JSON.stringify(msg));
});

Every time a new event comes in, be it once per hour or a thousand times a second, the Function is executed. Azure manages the scale-out for you.

[ Runnable Example ]

Azure Event Grid is another trigger type for Azure Functions. It is a dispatcher service for distributing events from many other Azure services as well as external data sources.

A classic example is subscribing to events from Azure Blob Storage. The Event Grid subscription connects to a given Storage Account and provides several handy options to filter the event stream. The following example subscribes to all JPG files created in any container of the account:

const storageAccount = new azure.storage.Account("eventgridsa", {
 resourceGroupName: resourceGroup.name,
 accountReplicationType: "LRS",
 accountTier: "Standard",
 accountKind: "StorageV2",
});

azure.eventgrid.events.onGridBlobCreated("OnNewBlob", {
 storageAccount,
 subjectFilter: {
 caseSensitive: false,
 subjectEndsWith: ".jpg",
 },
 callback: async (context, event) => {
 context.log(`Subject: ${event.subject}`);
 context.log(`File size: ${event.data.contentLength}`);
 },
});

The event object is strongly typed: The snippet above logs the file size, but there are many other properties at your disposal. Code completion makes the discovery process painless.

It’s worth mentioning that Pulumi does much of the work behind the scenes here, allowing you to focus on the important parts of the task at hand. In particular, Pulumi retrieves the appropriate secret key from Azure Functions ARM API and creates an Event Grid subscription which points to a specific webhook containing that key.

8. Respond to Resource-level Events

[ Runnable Example ]

Here is a good illustration of the power of Event Grid. A callback function gets triggered for each change to any resource belonging to the target Resource Group:

const resourceGroup = new azure.core.ResourceGroup("eventgrid-rg");

eventgrid.events.onResourceGroupEvent("OnResourceChange", {
 resourceGroupName: resourceGroup.name,
 callback: async (context, event) => {
 context.log(`Subject: ${event.subject}`);
 context.log(`Event Type: ${event.eventType}`);
 context.log(`Data: ${JSON.stringify(event.data)}`);
 },
});

This simple piece of code is an easy launching point for many automation, auditing, and governance scenarios.

Data Flows with Function Bindings

Azure Functions come with a powerful system of bindings. So far, we only saw examples of using event sources as trigger bindings. However, Azure also supports a powerful set of so called “input” and “output” bindings.

9. Output Bindings

[ Runnable Example ]

Output bindings enable developers to easily forward the data from an Azure Function to an arbitrary destination in a declarative manner. For instance, if a queue handler needs to send a message to another queue, we don’t have to use cloud SDKs. Instead, we can return the message-to-be-sent from the callback and wire it to the output queue. Here is a quick example:

const incoming = new azure.storage.Queue("queue-in", {
 storageAccountName: storageAccount.name,
});

const outgoing = new azure.storage.Queue("queue-out", {
 storageAccountName: storageAccount.name,
});

incoming.onEvent("NewMessage", {
 outputs: [outgoing.output("queueOut")],
 callback: async (context, person) => {
 return {
 queueOut: `${person.name} logged into the system`,
 };
 },
});

Two elements play together here:

The outputs property defines an output binding with the name queueOut and the destination to outgoing
The queueOut property of the resulting object contains the output message

Note that the binding name must match the output property.

10. Input Bindings

[ Runnable Example ]

Input bindings pull extra bits of information and pass them as input parameters to the callback. The exact usage depends on the trigger and binding types and might be tricky to get right with JSON configuration files. Here is one example of wiring done in a Pulumi program:

const values = new azure.storage.Table("values", {
 storageAccountName: storageAccount.name,
});

const getFunc = new azure.appservice.HttpEventSubscription('get-value', {
 resourceGroupName: resourceGroup.name,
 route: "{key}",
 inputs: [
 values.input("entry", { partitionKey: "lookup", rowKey: "{key}" }),
 ],
 callback: async (context, request, entry) => {
 return {
 status: 200,
 body: entry.value,
 };
 },
});

There are three crucial bits here:

The route property of this HTTP Function contains a template parameter key. The runtime extracts the actual key value from the HTTP request.
The inputs option contains a reference to a Storage Table with a hardcoded partitionKey and a rowKey bound to the key template parameter. At execution time, a row is retrieved based on the combination of the keys. The entry, if found, is passed to the callback.
The callback has three input parameters, while all previous examples had two. The third parameter contains the retrieved row.

It’s possible to have multiple input and output bindings, and any combination of those.

Wrapping Up

In this post, we’ve seen some of the exciting things you can do with Azure Functions in Pulumi. Developers use serverless functions as a glue between managed cloud services. Pulumi offers a compelling way to define the links between these pieces of cloud infrastructure in a simple and expressive way.

For a streamlined Pulumi walkthrough, including language runtime installation and cloud configuration, see the Azure Quickstart, check more examples at Pulumi Azure GitHub, and join the Pulumi Community Slack.

Globally-distributed Serverless App in 100 Lines of Code

Mikhail Shilkov — Tue, 02 Jul 2019 00:00:00 +0000

Pulumi is excellent at connecting multiple cloud components into a cohesive application. In my previous post, I introduced the way to mix JavaScript or TypeScript serverless functions directly into the cloud infrastructure programs.

Today, I will build a serverless application with both the data store and the HTTP endpoint located close to end users to ensure prompt response time. The entire application runs on top of managed Azure services and is defined as a single Pulumi program in TypeScript.

Baseline

I’m going to build a URL shortener: a simple HTTP endpoint which accepts a shortcode in the URL and then redirects a user to the full URL associated with the given short code.

I start simple and make a non-distributed version of the application first. It consists of two main components: a Cosmos DB container to store URL mappings and two Azure Functions to handle HTTP requests:

Let’s define this infrastructure in a Pulumi program.

Cosmos DB Collection

Arguably, Cosmos DB is overkill in such a simple scenario: we just need a key-value store, so Table Storage would suffice. However, Cosmos DB comes handy at the multi-region setup later on.

Next to a standard resource group definition, I create a Cosmos DB account with location set to a single region and Session consistency level:

import * as pulumi from "@pulumi/pulumi";
import * as azure from "@pulumi/azure";
import { CosmosClient } from "@azure/cosmos";

const location = "westus"; // To be changed to multiple configurable locations

let resourceGroup = new azure.core.ResourceGroup("UrlShorterner", {
 location,
});

let cosmosdb = new azure.cosmosdb.Account("UrlStore", {
 resourceGroupName: resourceGroup.name,
 location,
 geoLocations: [{ location, failoverPriority: 0 }],
 offerType: "Standard",
 consistencyPolicy: {
 consistencyLevel: "Session",
 maxIntervalInSeconds: 5,
 maxStalenessPrefix: 100,
 },
});

Cosmos DB has a hierarchical structure of Accounts, Databases, and Containers. Let’s define a database and a SQL collection to store our URL mappings:

const database = new azure.cosmosdb.SqlDatabase("Database", {
 resourceGroupName: resourceGroup.name,
 accountName: account.name,
});

const collection = new azure.cosmosdb.SqlContainer("Urls", {
 resourceGroupName: resourceGroup.name,
 accountName: account.name,
 databaseName: database.name,
});

Function App

Serverless Azure Functions are going to handle the HTTP layer in my application. I use the technique of serverless functions as callbacks to define Azure Functions inline inside my Pulumi program:

const fn = new azure.appservice.HttpEventSubscription("GetUrl", {
 resourceGroup,
 location,
 route: "{key}",
 callback: async (_, request: azure.appservice.HttpRequest) => {
 const endpoint = cosmosdb.endpoint.get();
 const masterKey = cosmosdb.primaryMasterKey.get();

 const client = new CosmosClient(
 { endpoint, key, connectionPolicy: { preferredLocations: [location] } });
 const container = client.database(database.name.get()).container(collection.name.get());

 const key = request.params['key'];
 try {
 const response = await container.item(key.toString()).read();
 return {
 status: 301,
 headers: { "location": response.body.url },
 body: '',
 };
 } catch (e) {
 return { status: 404, body: 'Short URL not found' }


});

export const url = fn.url;

I supply the template {key} as route parameter so that the function accepts wildcard URLs and extract the wildcard value as a key parameter available inside the request object. Then, I use the received key to look up the full URL. The URL is returned to the client as a header of the 301 Redirect response. 404 Not Found is returned in case the requested document does not exist—the naughty Cosmos SDK throws an error in this scenario.

Note how the Cosmos DB parameters endpoint and primaryMasterKey are used directly inside the function. There is no need to manage the keys manually or explicitly put them into application settings: the generated keys are injected into the code at the time of deployment. In practice, a better idea is to store the key in a KeyVault and read it via application settings, but I’ll leave this for a separate article.

A function to add a URL to the database looks quite similar to the above:

const fn = new azure.appservice.HttpEventSubscription("AddUrl", {
 resourceGroup,
 methods: ["POST"],
 callback: async (_, request: azure.appservice.HttpRequest) => {
 const endpoint = cosmosdb.endpoint.get();
 const masterKey = cosmosdb.primaryMasterKey.get();

 const client = new CosmosClient(
 { endpoint, key, connectionPolicy: { preferredLocations: [primaryLocation] } });
 const container = client.database(database.name.get()).container(collection.name.get());

 await container.items.create(request.body);
 return { status: 200, body: 'Short URL saved' };

});
export const addEndpoint = fn.url;

The key differences are:

Using POST as the accepted HTTP method
Creating a new item in Cosmos DB
Using request.body as the payload to save

I might need to add some validation in a real application—skipped for now.

Trying It Out

Now, when a user navigates to an address like https://geturl-xyz.azurewebsites.net/api/URLKEY, they get redirected to the full URL associated with the key. The URL is not quite short yet, but we could make it so with a custom domain configuration.

Presumably, we expect our URL shortener to be popular around the world. Therefore, one key aspect is to make sure that the service responds fast and enables a smooth user experience. Let’s measure the response time of our first version from different spots around the world:

Location	Response time
San Francisco	133 ms
New York	272 ms
London	383 ms
Frankfurt	420 ms
Tel-Aviv	470 ms
Hong-Kong	437 ms
Brisbane	449 ms

The further we get from the West US region, the slower the responses become.

How can we do better?

Bring Compute and Data Closer to Users

It’s hard to beat the speed of light, so if we want to respond fast, we need to bring both code and data close to any target user.

Luckily, the most involved aspect of that—the data distribution—can be handled by Cosmos DB multi-region accounts. We need to take care of deploying the code in multiple locations and routing the traffic to the nearest one. Here is the plan:

I don’t care too much about the latency of Add URL function, so I left it out of the picture.

Configuring the locations

The above picture shows three Azure regions, but since I’m using a general-purpose programming language, I can handle any number of locations in the same way. Actually, I put the list of target regions in Pulumi config file and read it at execution time:

const config = new pulumi.Config();
const locations = config.require("locations").split(',');
const primaryLocation = locations[0];

The order of regions defines the priority.

Multi-region Cosmos DB Account

Since locations is a simple array, I can map this array to produce the array of geoLocations to be set for my Cosmos DB:

let cosmosdb = new azure.cosmosdb.Account("url-cosmos", {
 location: primaryLocation,
 geoLocations: locations.map((location, failoverPriority) => ({
 location,
 failoverPriority
 })),
 /* ... other properties stay the same ... */

Right here, the full power of a programming language at my fingertips!

Multi-region Serverless App

Azure Traffic Manager is a DNS-based traffic load balancer that enables you to distribute traffic optimally to services across global Azure regions while providing high availability and responsiveness.

Sounds like what I need for my global application! Let’s define a Traffic Manager profile:

const profile = new azure.trafficmanager.Profile("UrlShortEndpoint", {
 resourceGroupName: resourceGroup.name,
 trafficRoutingMethod: 'Performance',
 dnsConfigs: [{
 relativeName: "shorturls",
 ttl: 60,
 }],
 monitorConfigs: [{
 protocol: 'HTTP',
 port: 80,
 path: '/api/ping',
 }]
});

I set the routing method to Performance:

Select Performance when you have endpoints in different geographic locations and you want end users to use the “closest” endpoint in terms of the lowest network latency.

Now, I need to create a Function App in each of the target regions. That’s easy to achieve with a for..of loop:

for (const location of locations) {
 const fn = new azure.appservice.HttpEventSubscription(`GetUrl-${location}`, {
 resourceGroup,
 location,
 route: "{key}",
 callback: /* ... the function stays the same ... */
 });
}

Finally, I set up an Endpoint for each Function App to bind them to the Traffic Manager. I do so in the same loop as above:

for (const location of locations) {
 const fn = new azure.appservice.HttpEventSubscription(`GetUrl-${location}`, {
 /* ... function definition as shown above ... */
 });

 const app = fn.functionApp;

 new azure.trafficmanager.Endpoint(`tme${location}`, {
 resourceGroupName: resourceGroup.name,
 profileName: profile.name,
 type: 'azureEndpoints',
 targetResourceId: app.id,
 target: app.defaultHostname,
 endpointLocation: app.location,
 });
}

I assign the properties of each Function App to the corresponding Endpoint with a simple object initializer expression.

Everything is in place. I can export the Traffic Manager endpoint:

export const endpoint = pulumi.interpolate `http://${profile.fqdn}/api/{key}`;

Time to re-run the latency tests.

Results

I deployed the distributed infrastructure to five Azure regions: West US, East US, West Europe, East Asia, and Australia East. The results are shown in the table below, next to the values from the initial run:

Location	Single region	Multiple regions
San Francisco	133 ms	140 ms
New York	272 ms	152 ms
London	383 ms	150 ms
Frankfurt	420 ms	130 ms
Tel-Aviv	470 ms	307 ms
Hong-Kong	437 ms	149 ms
Brisbane	449 ms	529 ms

We got a much smoother distribution of response time. There’s no region close enough to Tel-Aviv, so its latency is still high-ish. I checked Brisbane, and the traffic was somehow directed to the East US region, so the latency hasn’t improved at all. For today, I guess I’ll stick to the hypothesis “Australian internet is slow” and advise to always test your target scenarios without trusting the providers blindly.

More importantly, I walked you through a scenario of developing an end-to-end application with Pulumi. I highlighted the power of leveraging familiar techniques coming from the TypeScript realm. You can find the full code in Pulumi examples and more on Cosmos DB in the Azure Provider.

Cloud brings superpowers to developer’s hands. You just need to use those efficiently.

Mapbox IOT-as-code with Pulumi Crosswalk for AWS

Chris Toomey — Wed, 12 Jun 2019 00:00:00 +0000

Guest Author: Chris Toomey, Solution Architect Lead @ Mapbox

With 8 billion+ connected IoT devices and 2 billion GPS-equipped smartphones already online, logistics businesses are tracking assets at every step in the supply chain. At this scale and complexity, it is imperative to have a flexible way to ingest, process, and act upon this data, without sacrificing security or best practices.

To meet this need, Mapbox has created an Asset Tracking Solution that uses Pulumi’s open source JavaScript libraries (AWS, AWSX) available with multi-language support with Pulumi Crosswalk for AWS. Pulumi Crosswalk for AWS is an open source framework that streamlines creation, deployment and management of AWS services with built-in AWS Best Practices and minimal lines of code in common programming languages.

In this blog, we will show snippets of the Javascript code that embraces the power of Pulumi to program AWS service APIs to create the Mapbox solution. To see the full architecture in action with a live bike race across America, please refer to this webinar recorded on June 13th 2019 and the Mapbox asset tracking solution. Also refer to this blog of the Race across America showcased live during the webinar tomorrow.

Prerequisites: Install Pulumi; Install Node.js and Setup AWS

The diagram represents how Mapbox’s solution design on AWS services is built with Pulumi AWS and AWSX libraries in Javascript. Data is ingested in the Asset tracking IOT solution with a REST API. Mapbox uses its stream processor to perform enrichments such as geofencing, traffic-aware ETA calculations, and high-prevision elevation. This data, exposed by the API is backed by a high performance database (DynamoDB) to enable visualization in realtime on the map client.

The fun part – Pulumi Service console helps you map your cloud architecture shown above as a connected set of DAG resources so you need not remember what you deployed as your cloud environments scale:

Lets now work through the solution flow with some sample snippets of code. To get access to the complete piece of code, please connect with Mapbox solution team here and Pulumi team here. We will be happy to help you sample test this solution on AWS!

STEP 1: Create a Pulumi project using an AWS JavaScript template

$ brew install pulumi/tap/pulumi
$ mkdir asset-tracking && cd asset-tracking
$ pulumi new aws-javascript
$ yarn install
$ ls
Pulumi.dev.yaml Pulumi.yaml index.js node_modules package-lock.json package.json

STEP 2: Create the ingestion REST API with AWS Services

To visualize data the system has to accept large volumes of data from multiple sources into the rest of the system. Mapbox uses Pulumi’s AWS & AWSX libraries to build an IOT ingestion rule and forwards it to Kinesis streams. A sample index.js looks like this:

"use strict";
const pulumi = require("@pulumi/pulumi");
const aws = require("@pulumi/aws");
const awsx = require("@pulumi/awsx");

// Create Kinesis stream for ingestion
const ingestStream = new aws.kinesis.Stream("ingestAssets", {
 shardCount: 1,
 retentionPeriod: 72
});

// Create IoT Rule to push into Kinesis stream
const iotRole = new aws.iam.Role("iotRole", {
 assumeRolePolicy: JSON.stringify({
 Version: "2012-10-17",
 Statement: [
 {
 Effect: "Allow",
 Principal: {
 Service: "iot.amazonaws.com"
 },
 Action: "sts:AssumeRole"
 }
 ]
 })
});

const iotRolePolicy = new aws.iam.RolePolicy("iotRolePolicy", {
 policy: pulumi.interpolate`{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Effect": "Allow",
 "Action": [
 "kinesis:*"
 ],
 "Resource": "${ingestStream.arn}"
 }
 ]
 }`,
 role: iotRole.id
});

const iotRule = new aws.iot.TopicRule("iotTrigger", {
 description: "Pass from IoT Core to Asset Tracking",
 name: "iotAssetIngest",
 enabled: true,
 kinesis: {
 partitionKey: "id",
 roleArn: iotRole.arn,
 streamName: ingestStream.name
 },
 sql: "SELECT * FROM 'topic'",
 sqlVersion: "2015-10-08"
});

Kinesis streams are used to ingest data into multiple AWS services defined as IOT rules, Firehose and Lambda functions. Each ingest stream has the right IAM role and policy to allow Kinesis to send the data into these AWS services.

The code above does not shown the Javascript code for Lambda functions used by Mapbox streamprocessor and Firehose all created with Pulumi AWS and AWSX libraries.

To get this piece of code, please connect with Mapbox solution team here and Pulumi team here.

STEP 3: Create a consumption API for the map client to consume from a data source

Once, data is flowing through the system we build another API to query DynamoDB, transform that data, and provide it to our mapping client in geojson. With Pulumi AWSX, we create a APIGateway endpoint API that can query this data out of DynamoDB. The sample code in index.js will look like this:

To get this piece of code, please connect with Mapbox solution team here and Pulumi team here.

// Create DynamoDB Table
const assetTable = new aws.dynamodb.Table("assetTable", {
 attributes: [
 {
 name: "id",
 type: "S"
 },
 {
 name: "ts",
 type: "N"
 }
 ],
 hashKey: "id",
 rangeKey: "ts",
 ttl: {
 attributeName: "expiration",
 enabled: true
 },
 billingMode: "PAY_PER_REQUEST"
 });

// Create API to read DynamoDB
const endpoint = new awsx.apigateway.API("assetQuery", {
 routes: [
 {
 path: "/",
 method: "GET",
 eventHandler: (request, ctx, cb) => {
 const AWS = require("aws-sdk");
 const ddb = new AWS.DynamoDB.DocumentClient({
 apiVersion: "2012-10-08"
 });
 const tableName = assetTable.name.value;
 const params = {
 TableName: tableName
 };
 ddb.scan(params, (err, data) => {
 const features = data.Items.map(item => {
 const point = turf.point([item.longitude, item.latitude], {
 id: item.id,
 speed: item.speed
 });
 return point;
 });
 const featureCollection = turf.featureCollection(features);
 cb(undefined, {
 statusCode: 200,
 body: Buffer.from(
 JSON.stringify(featureCollection),
 "utf8"
 ).toString("base64"),
 isBase64Encoded: true,
 headers: { "content-type": "application/json" }
 });
 });
 }
 }
 ],
 stageName: "dev"
});

Once we’ve created the API, we start defining our routes and then in-line we can define a Lambda function. It can use NPM modules and Pulumi will handle bundling all the dependencies.

This endpoint assumes we are querying Dynamo - and so we are going to use the DynamoDB Document Client. Using that straightforward scan syntax, we query the table and then start to parse our results. Since every data point has a latitude and longitude as GeoJSON, the data is transformed using Turf.js before sending to the client. Even more beautiful is how Pulumi handles all the heavy lifting of packaging, deploying, and giving you the endpoint to query immediately.

STEP 4: Hook up a map!

Now we have data coming and going from our Solution, we just have to hook it up to a Mapbox map. This is actually quite simple now that Pulumi has built us a REST API.

Following this example, we paste in the endpoint from Pulumi and pick our interval.

Additional Resources

For more information from Mapbox check out the Mapbox Blog. For more solutions from Pulumi please visit our Blog or head over to the Pulumi Examples repository. For AWS solutions visit the AWS partner blog site.

Introducing Pulumi Crosswalk for AWS: The Easiest Way to AWS

Luke Hoban — Mon, 10 Jun 2019 00:00:00 +0000

Some of the code in this post is out of date. See the AWS guides for an updated overview and examples.

Amazon Web Services provides an incredible platform for developers to build cloud-native applications, and is used by millions of customers of all sizes. The building block services that AWS offers enable teams to offload undifferentiated heavy-lifting to AWS. To maximally benefit from these services though, cloud engineering teams must learn how to compose all of these building blocks together to build and deliver their own applications. Today, this is still too hard. Getting from your laptop to a production-ready AWS deployment frequently takes days or weeks instead of minutes or hours. And AWS building block services frequently leave you to re-implement (and re-discover) best-practices instead of providing these as smart defaults.

Pulumi Crosswalk for AWS is a new open source library of infrastructure-as-code components that make it easier to get from zero to production on AWS, easier to adopt AWS best practices by default, and easier to evolve your AWS infrastructure as your application needs mature.

Crosswalk for AWS offers higher-level components for a wide range of the AWS platform - from serverless (Lambda, API Gateway and more) to containers (ECS, Fargate) to Kubernetes (EKS) and across networking (VPCs, SecurityGroups), monitoring (Cloudwatch Dashboards and Alarms) and load balancing (ALB and NLB). Because Crosswalk builds on the primitive building blocks of the AWS platform, you can use 100% of what AWS offers, and can seamlessly move between high-level components and low-level platform primitives, mixing-and-matching as needed.

In some of these domains, there have been successful tools that have helped developers more easily work with one specific aspect of AWS - Serverless Framework for serverless, Docker Compose for ECS, eksctl for EKS. But these tools have traditionally been limited to one specific kind of compute, limiting what users can accomplish, fragmenting development of complete cloud solutions, and leading to sharp cliffs when application needs evolve. Crosswalk for AWS offers the same kind of simpler and more productive developer interface as these tools, but applied uniformly across the entire AWS platform. Simple things are simple, and arbitrarily complex things are possible.

Whether you are starting a new SaaS project at a startup, migrating an enterprise application to cloud-native infrastructure, or delivering a new service as part of an established cloud application, Crosswalk for AWS provides the easiest way to get from zero to production, and to then evolve with your projects’ needs. You can get started with Crosswalk for AWS today!

Note: Crosswalk for AWS projects are authored using the Pulumi infrastructure-as-code tools. Pulumi allows you to define your infrastructure using code. As your infrastructure code evolves, Pulumi makes the minimal required changes to your cloud resources, ensuring that your infrastructure is updated without downtime (if possible).

From Zero to Production And Beyond with Crosswalk for AWS

Serverless (Lambda, API Gateway)

When getting started with a new cloud project, the starting point is often just serving a simple API on the internet. With just a few lines of code and tens of seconds to deploy, Crosswalk for AWS gives us our API.

import * as awsx from "@pulumi/awsx";

const api = new awsx.apigateway.API("api", {
 routes: [{
 method: "GET",
 path: "/",
 eventHandler: async (ev, ctx) => {
 return {
 statusCode: 200,
 body: JSON.stringify(ev),
 };
 }
 }],
})

export const endpoint = api.url;

By building on top of great AWS building blocks like Lambda, API Gateway, IAM and more, we avoid needing to worry about infrastructure, pay nearly zero fixed costs, and gain the ability to iterate quickly. With just a few more lines of code, we can wire up to our own domain, add authorization, provision a database, or chain together a more complex event-driven application.

Containers (ECS, Fargate)

As our needs grow, we might find we need to deploy containers - either custom applications or existing Docker images. Again, just a few lines of code gets our container running in production in AWS.

import * as awsx from "@pulumi/awsx";

// Create the cluster
const cluster = new awsx.ecs.Cluster("cluster");

// Configure a Load Balancer
const alb = new awsx.lb.ApplicationLoadBalancer("lb", {
 securityGroups: cluster.securityGroups,
});
const web = alb.createListener("web", { port: 80, external: true });

// Define the scaled-out Fargate service
const appService = new awsx.ecs.FargateService("app-svc", {
 cluster,
 taskDefinitionArgs: {
 container: {
 image: awsx.ecs.Image.fromPath("app", "./app"),
 portMappings: [ web ],
 },
 },
 desiredCount: 5,
});

export const url = web.endpoint.hostname;

Using ECS, Fargate, ECR and ELB, we get a robust production-ready container deployment - horizontally scaled out, load-balanced, and integrated with a private image repository. With just a few more lines of code, we can add autoscaling, customize our ECS cluster, or wire through advanced container configuration (volumes, environment variables, and more).

Networking (VPC)

To adopt best practices security posture, we may want to move our cluster into a custom VPC, and run our containers in private subnets. We can define and configure our VPC, and then redeploy our containers into this new VPC with ease:

import * as awsx from "@pulumi/awsx";

const vpc = new awsx.ec2.Vpc("vpc", {
 numberOfAvailabilityZones: 3,
});

const cluster = new awsx.ecs.Cluster("cluster", {
 vpc: vpc,
});

// ... same as before ...

The Vpc component builds on best-practices Virtual Private Cloud design patterns based on AWS guidance and documentation. By building in simple defaults for routing, subnet structure, NATs and multi-AZ you can get started quickly without having to re-discover these best practices. And as your needs grow, you can deeply customize and evolve your VPC structure - defining custom CIDR blocks, customizing Internet and NAT Gateways or additional additional private subnets.

Kubernetes (EKS)

As our project continues to evolve, we may want to move our container workloads into Kubernetes using EKS. We can stand up an EKS cluster with just a few lines of code, then customize our cluster and deploy Kubernetes workloads into our cluster.

import * as awsx from "@pulumi/awsx";
import * as eks from "@pulumi/eks";

// Create a VPC for our cluster.
const vpc = new awsx.ec2.Vpc("vpc");

// Create the EKS cluster itself.
const cluster = new eks.Cluster("cluster", {
 vpcId: vpc.id,
 subnetIds: vpc.privateSubnetIds,
 instanceType: "m5.large",
 desiredCapacity: 2,
 minSize: 1,
 maxSize: 2,
 deployDashboard: true,
});

// Export the cluster's kubeconfig.
export const kubeconfig = cluster.kubeconfig;

Building on EKS we get all the benefits of managed Kubernetes paired with the platform capabilities of AWS. With Crosswalk for AWS, we can get started quickly, and then evolve to take advantage of all of these platform features, like customized node groups, private networking, and even deploying Kubernetes YAML and Helm charts to our cluster.

Monitoring (CloudWatch)

Getting our infrastructure up and running in AWS is great, but we also want to be able to monitor it. We can add a few lines to any of the solutions above to define metrics, dashboards, alerting and more to monitor our AWS infrastructure.

import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";

// Create a bucket and a function to log all new object created events
const bucket = new aws.s3.Bucket("b");
const subscription = bucket.onObjectCreated("newObject", async (ev) => {
 console.log(JSON.stringify(ev));
});

// Create a CloudWatch Dashobard for our functions invocations
const dashboardName = "funcDashboard";
const dashboard = new awsx.cloudwatch.Dashboard(dashboardName, {
 widgets: [
 new awsx.cloudwatch.LineGraphMetricWidget({
 title: "Lambda invocations",
 width: 14,
 metrics: awsx.lambda.metrics.invocations({
 function: subscription.func, statistic: "Sum", period: 60, }), }), ], });

// Export the URL of the dashboard in the AWS console
export const dashboardUrl =
 `https://${aws.config.region}.console.aws.amazon.com/cloudwatch/home?` +
 `region=${aws.config.region}#dashboards:name=${dashboardName}`;

With CloudWatch being deeply integrated into all AWS services, we can easily build up robust logging, alerting and dashboarding solutions across our AWS infrastructure. With just a few more lines of code we can get logs for our functions and containers, create alarms when we cross critical thresholds, and create rich dashboard layouts all of which will version along with our infrastructure!

Early Feedback

We’ve been working on pieces of Crosswalk for AWS since before we launched Pulumi, and we’re excited to be launching Crosswalk for AWS today to bring these together for all AWS users. Over the last few months, users have built on Crosswalk for AWS for all of the scenarios above and more.

Tableau has used Crosswalk for AWS to accelerate their Kubernetes migration on AWS, building on the networking and EKS capabilities of Crosswalk for AWS.

“We’ve been happily using Pulumi’s EKS support for more than three months now. Our team was looking for an end-to-end solution to tame the complexity of Kubernetes on AWS and ensure we adhere to AWS best practices. Pulumi’s Crosswalk effort has equipped our team to scale far better, our delivery is now automated, and we can now deliver new application and infrastructure features with much faster turn-around, sometimes in just a few hours. Pulumi is a key contributor to our team’s improvement in productivity.” - Pankaj Dhingra, Sr Director, Cloud Engineering, Tableau

Compute Software has used Crosswalk for AWS to go from zero to production, and then to evolve their infrastructure in AWS, all without needing a dedicated cloud infrastructure engineering team.

“We use Crosswalk for AWS with Fargate. It’s been instrumental in getting us into production. Deploying to the cloud is necessarily complex. Crosswalk streamlines the process by exposing the controls you care about, always allowing you to dive deeper when needed.” - Kenny Williams, Co-Founder, Compute Software

We’re excited by the success these teams and many more have already had with Crosswalk for AWS, but even more excited about the potential to truly democratize access to the AWS platform and unlock highly productive cloud development for all kinds of engineering organizations.

What’s Next?

We’re still just scratching the surface of what’s possible with Crosswalk. We’re continuing to evolve the open source projects to simplify additional features of the AWS platform. And we’re also working to bring the same approach to other cloud platforms, including Azure, GCP and Kubernetes.

We believe that making it easier for cloud engineers to develop and deploy natively in the cloud will unlock creativity and empower developers to take advantage of more of the rich capabilities of AWS. We can’t wait to see what you build with Crosswalk!

Crosswalk for AWS is free and open source, and you can get started today.

For more on Crosswalk for AWS see:

Serverless as Simple Callbacks with Azure Functions

Mikhail Shilkov — Tue, 07 May 2019 00:00:00 +0000

Today’s guest post is from Mikhail Shilkov, a Microsoft Azure MVP and early Pulumi user and contributor - enjoy!

Serverless compute services, like Azure Functions, offer an amazing power to application developers to leverage: highly available, automatically scaled, low-ceremony, pay-per-value functions created in several lines of code.

So, what’s the simplest way to take a Node.js function and deploy it to Azure cloud as an HTTP endpoint? How about this little tutorial:

1. Create a new Pulumi program:

$ pulumi new azure-typescript

2. Define an HTTP endpoint in `index.ts`:

import * as azure from '@pulumi/azure';

const resourceGroup = new azure.core.ResourceGroup('example', { location: 'West US' });

const greeting = new azure.appservice.HttpEventSubscription('greeting', {
 resourceGroup,
 callback: async (context, req) => {
 return {
 status: 200,
 body: `Hello ${req.query['name'] || 'World'}!`
 };
 }
});

export const url = greeting.url;

3. Deploy:

$ pulumi up
 
Updating:
   Type
 + pulumi:pulumi:Stack
 + ├─ azure:appservice:HttpEventSubscription
 + │ ├─ azure:storage:Account
 + │ ├─ azure:appservice:Plan
 + │ ├─ azure:storage:Container
 + │ ├─ azure:storage:ZipBlob
 + │ └─ azure:appservice:FunctionApp
 + └─ azure:core:ResourceGroup
 
Outputs:
   url: "https://greetingc21a23fe.azurewebsites.net/api/greeting"
Resources:
   + 8 created

4. Access your function via HTTP:

$ curl https://greetingc21a23fe.azurewebsites.net/api/greeting?name=Pulumi
Hello Pulumi!

With 12 lines of code and two CLI commands, I’ve created all Azure resources required to host a serverless function without an explicit configuration of Azure services. Okay, I had to define a location for my resource group, but that could also be accomplished via pulumi config.

Pulumi compiled my TypeScript function, serialized it to a JavaScript file, created the bindings in a function.json file, hosting configuration in a host.json file, uploaded all these assets to Blob Storage, and configured a Consumption Plan and a Function App to run my function. An automated and reproducible deployment in less than two minutes.

Beyond Hello-World

The power of Node.js comes from the richness of its library ecosystem. There’s an NPM package for everything, so you definitely want to use those.

Serverless-function-as-callback imports dependencies in a transparent way. Install the NPM packages of your choice and use them inside the callback:

import * as moment from 'moment';

const greeting = new azure.appservice.HttpEventSubscription('greeting', {
 resourceGroup,
  callback: async (context, req) => {
 return {
 status: 200,
 body: `Hello ${req.query['name'] || 'World'} at ${moment().format('LLLL')}!`
 };
 }
});

The packages get zipped up inside the deployment artifact automatically so that the Function App can find them at the startup. So there’s no need to manually figure out how to produce the archive, get it uploaded, and maintain it as your libraries get updated.

Stay tuned for another blog post with a full implementation of a serverless URL shortener application deployed into multiple Azure regions for fast response time and improved resiliency.

Not Only HTTP

Your application might not be a bunch of HTTP functions. You probably want to leverage queues for asynchronous message passing. How about defining a callback on the queue object itself:

const storageAccount = new azure.storage.Account("storage", {
   resourceGroupName: resourceGroup.name,
   location: resourceGroup.location,
   accountReplicationType: "LRS",
   accountTier: "Standard",
});

const queue = new azure.storage.Queue("myqueue", {
  resourceGroupName: resourceGroup.name,
  storageAccountName: storageAccount.name
});

queue.onEvent("newMessage", async (context, msg) => {
   // code to process 'msg' however you want here
   console.log("Message received: " + msg.toString());
});

Alternatively, define a ServiceBus topic and immediately subscribe to the messages:

import * as servicebus from "@pulumi/azure/eventhub";

const namespace = new servicebus.Namespace("test", {
   resourceGroupName: resourceGroup.name,
   sku: "standard",
});

const topic = new servicebus.Topic("mytopic", {
   resourceGroupName: resourceGroup.name,
   namespaceName: namespace.name,
});

export const subscription = topic.onEvent("mysubscription", async (context, msg) => {
   console.log("Received: " + msg.toString());
});

In addition, get notified when a new PNG image is uploaded to a Blob Container:

const storageContainer = new azure.storage.Container("images-container", {
  resourceGroupName: resourceGroup.name,
  storageAccountName: storageAccount.name,
  name: "images",
});

storageContainer.onBlobEvent("newImage", {
   callback: async (context, blob) => {
       console.log("File size: " + context.bindingData.properties.length);
   },
   filterSuffix: ".png",
});

In all these examples, you get a fully configured Function App on Consumption Plan ramped up and bound to the trigger of choice. Your callback runs in the cloud handling every event with no manual work of hooking these different components together.

Using a general-purpose language like TypeScript provides one consistent approach to defining and delivering serverless applications and infrastructure as one cohesive application.

We strive to make cloud development as simple as everyday JavaScript development that made the language so successful. A lot is happening behind the scenes, but the code still looks like “normal code”. Composition of cloud resources should be as straightforward as hooking up components in any traditional application.

Looking Ahead

Pulumi serverless programming model for Azure Functions is just ramping up. There is only a handful of trigger types supported right now, and some configuration parameters are not exposed yet.

So, today is the perfect time to chime in and join the discussion! Help us answer the questions:

Is this programming model beneficial to your scenarios?
Which trigger types do you want to be supported?
How should we package multiple functions into Function App(s)?
Do you need input and output bindings to be supported, and if yes, in which shape?

Feel free to create an issue on GitHub, tag us on Twitter, or join our community Slack channel.

Happy serverless programming!

Secure APIs with Auth0 Lambda Authorizers Using Pulumi & AWS

Erin Krengel — Wed, 24 Apr 2019 00:00:00 +0000

Creating serverless applications just got even easier! You can now protect your application APIs in just three easy steps. We’ve already posted about how easy it is to create serverless apps in Pulumi. Now, we’re helping you simplify protecting those apps with API Gateway and Lambda authorizers.

With Pulumi’s new AWSX package, you can quickly define a Lambda and an AWS Lambda authorizer to protect it. We’re once again harnessing the power of Lambdas as Lambdas to allow developers to focus on writing code.

Today, we will walkthrough creating a simple serverless app using AWS and Pulumi. We will simplify implementing the OAuth protocol by using Auth0 and AWS Lambda authorizers to authorize users. Auth0 provides a universal authentication and authorization platform for applications. It has become an extremely popular platform for user management because Auth0 makes OAuth easy.

Concepts

Before we get started, let’s go over some concepts we’ll be using.

API Gateway

An API Gateway takes clients requests and then re-routes them to the appropriate backend service. Typically, a gateway serves as the “front door” for an API and is used for traffic management as well as flexible security controls.

Using AWS API Gateway, you can build a fully managed collection of resources which integrate with AWS Lambda and various other AWS resources. To control who can call your API, you can use IAM permissions, an Amazon Cognito User Pool or set up custom logic using a Lambda authorizer.

OAuth

OAuth is an industry-standard for token-based authorization. Using OAuth, users can grant third-party applications or websites access to their information on another website without disclosing their password. Google and Facebook are two very common examples of sites that permit users to share their personal information with websites or applications using OAuth.

Lambda Authorizers

An AWS Lambda authorizer is a Lambda that provides access control to an API. It contains custom logic for authorizing requests to an endpoint. If a Lambda authorizer is configured, API Gateway routes a client’s call to the Lambda first. The Lambda authorizer runs its custom logic and returns a Policy and principal ID, which are used by API Gateway to determine if the call to the backend is allowed.

Step-by-Step Guide To Creating a Lambda Authorizer

Using a Lambda authorizer, we can implement the authorization flow using Auth0 to handle our Access Tokens.

API Gateway can be used to protect many different resources, and in the following example we will front an AWS Lambda. Users authenticate with Auth0 and then use the token they’ve obtained to make a request to our Lambda via our API Gateway, which authorizes the request by validating the token with Auth0. Some of the code examples have been stubbed or shortened for this blog post. The full code example can be found in our Pulumi examples repo.

Below is a diagram of the architecture we will set up. In Step 1, we will create our API Gateway and Hello World Lambda. In Step 2, we will set up the logic for our Lambda authorizer. And finally in Step 3, we will bring it all together by telling API Gateway to use our authorizer.

1 - Define Your Routes

Using Pulumi’s AWSX package, you can easily define routes to serverless apps as well as routes to static resources. We will create a very simple Hello world! backend Lambda, but in practice you can have this do much more interesting things.

import * as awsx from "@pulumi/awsx";

// Create our API and reference the Lambda authorizer
const api = new awsx.apigateway.API("myapi", {
    routes: [{
        path: "/hello",
        method: "GET",
        eventHandler: async () => {
            return {
                statusCode: 200,
                body: "<h1>Hello world!</h1>",
            };
        },
    }],
});

2 - Write Your Custom Auth Logic

Just as Pulumi allows you to define your Lambda inline, you can define your custom authorization code. We will define a small wrapper for our custom logic and write the rest of the code as a function called authenticate. Pulumi will package up all our runtime code and create an AWS Lambda for us.

import * as awsx from "@pulumi/awsx";
import * as pulumi from '@pulumi/pulumi';

import * as jwksClient from 'jwks-rsa';
import * as jwt from 'jsonwebtoken';
import * as util from 'util';

const config = new pulumi.Config();
const jwksUri = config.require("jwksUri");
const audience = config.require("audience");
const issuer = config.require("issuer");

const authorizerLambda = async (event: awsx.apigateway.AuthorizerEvent) => {
    try {
        return await authenticate(event);
    }
    catch (err) {
        console.log(err);
        // Tells API Gateway to return a 401 Unauthorized response
        throw new Error("Unauthorized");
    }
}

/**
 * Below is all code that gets added to the Authorizer Lambda. The code was copied and
 * converted to TypeScript from
 * [Auth0's GitHub Example](https://github.com/auth0-samples/jwt-rsa-aws-custom-authorizer)
 */

function getToken(event: awsx.apigateway.AuthorizerEvent): string {
    // Stubbed function to extract and return the Bearer Token from the Lambda event parameter
}

// Check if the Token is valid with Auth0
async function authenticate(event: awsx.apigateway.AuthorizerEvent): Promise<awsx.apigateway.AuthorizerResponse> {
    const token = getToken(event);

    const decoded = jwt.decode(token, { complete: true });
    if (!decoded || typeof decoded === "string" || !decoded.header || !decoded.header.kid) {
        throw new Error('invalid token');
    }

    const client = jwksClient({
        cache: true,
        rateLimit: true,
        jwksRequestsPerMinute: 10, // Default value
        jwksUri: jwksUri
    });

    const key = await util.promisify(client.getSigningKey)(decoded.header.kid);
    const signingKey = key.publicKey || key.rsaPublicKey;
    if (!signingKey) {
        throw new Error('could not get signing key');
    }

    const verifiedJWT = await jwt.verify(token, signingKey, { audience, issuer });
    if (!verifiedJWT || typeof verifiedJWT === "string" || !isVerifiedJWT(verifiedJWT)) {
        throw new Error('could not verify JWT');
    }
    return awsx.apigateway.authorizerResponse(verifiedJWT.sub, 'Allow', event.methodArn);
}

3 - All Together Now

Now let’s tie it all together! We will modify the route we created in Step 1 to include our authorizer.

import * as awsx from "@pulumi/awsx";

// Create our API and reference the Lambda authorizer
const api = new awsx.apigateway.API("myapi", {
    routes: [{
        path: "/hello",
        ...
        authorizers: awsx.apigateway.getTokenLambdaAuthorizer({
            authorizerName: "jwt-rsa-custom-authorizer",
            header: "Authorization",
            handler: authorizerLambda,
            identityValidationExpression: "^Bearer [-0-9a-zA-Z._]*$",
            authorizerResultTtlInSeconds: 3600,
        }),
    }],
});

The Finished Product

When we run pulumi up, Pulumi will create our resources and provide us with our protected endpoint.

$ pulumi up
...
     Type Name Status Info
 + pulumi:pulumi:Stack lambda-authorizer-dev created 1 message
 + ├─ aws:apigateway:x:API myapi created
 + │ ├─ aws:iam:Role myapifc45ff03 created
 + │ ├─ aws:iam:RolePolicyAttachment myapifc45ff03-32be53a2 created
 + │ ├─ aws:lambda:Function myapifc45ff03 created
 + │ ├─ aws:apigateway:RestApi myapi created
 + │ ├─ aws:apigateway:Deployment myapi created
 + │ ├─ aws:lambda:Permission myapi-62a1b306 created
 + │ └─ aws:apigateway:Stage myapi created
 + ├─ aws:iam:Role jwt-rsa-custom-authorizer-authorizer-role created
 + ├─ aws:iam:Role jwt-rsa-custom-authorizer created
 + ├─ aws:iam:RolePolicyAttachment jwt-rsa-custom-authorizer-32be53a2 created
 + ├─ aws:lambda:Function jwt-rsa-custom-authorizer created
 + └─ aws:iam:RolePolicy jwt-rsa-custom-authorizer-invocation-policy created
Outputs:
    url: "https://XXXXX.execute-api.us-west-2.amazonaws.com/stage/"
Resources:
    + 14 created
Duration: 28s

You’ll notice there’s a ton of resources we didn’t explicitly define. AWSX provisions the appropriate Lambdas, roles, and more for you. If required, all of these resources can be overridden in your Pulumi program by explicitly defining them.

Let’s give try out our new API.

We can now curl our endpoint without a token and should get a 401 Unauthorized response.

$ curl $(pulumi stack output url)hello
{"message":"Unauthorized"}

We can curl our endpoint with an invalid token and should once again get a 401 Unauthorized response.

$ curl $(pulumi stack output url)hello -H "Authorization: Bearer invalid"
{"message":"Unauthorized"}

Finally, we expect a 200 response when we obtain a token from Auth0 and use it to call our API. We can get a token by visiting the API Details page for our API and clicking the Test tab. This provides us with a valid token for testing purposes only. The way tokens are acquired by users or client apps will depend on the type of application (i.e. native mobile app, single-page app, web app, etc.) being developed. For more information on authenticating users, visit Auth0’s documentation.

Using the test access token, the API will return a 200 response: Hello world!

$ curl $(pulumi stack output url)hello -H "Authorization: Bearer <VALID_TOKEN>"
<h1>Hello world!</h1>

For the complete code example, visit our examples repo.

Keep It Going

While we used OAuth and Auth0 in this blog post, AWS Lambda authorizers are an extremely flexible tool that allow you to implement whatever custom authorization logic you require. AWS also supports Cognito Authorizers, which use AWS Cognito User Pools, and we will share even more examples soon.

There are a vast amount of resources for deploying your code into the cloud, and we want to make it easier for developers to make the most of those resources. Pulumi’s AWSX package provides reusable abstractions to enable developers to get their code production ready faster.

Get going with these resources:

Simple Serverless programming with Google Cloud Functions

Cyrus Najmabadi — Wed, 10 Apr 2019 00:00:00 +0000

Pulumi lets you create, deploy, and manage Google Cloud applications and infrastructure in familiar languages like JavaScript, TypeScript, and Python, and without needing to learn new DSLs or YAML templating solutions. This means great productivity and getting to use tools you already know and love. Since serverless is all about focusing more on your application code, and less on infrastructure and configuration toil, we absolutely love Google Functions.

The Simplest Way to Serverless

In fact, serverless has never been simpler than it is when you combine Pulumi with Google Cloud Functions. Want to serve a simple HTTP API with no fixed costs? It’s just a few lines of code – and no, we’re not hiding any YAML here:

import * as gcp from "@pulumi/gcp";

let greeting = new gcp.cloudfunctions.HttpCallbackFunction("greeting", (req, res) => {
    // Change this code to fit your needs!
    res.send(`Greetings from ${req.body.name || 'Google Cloud Functions'}!`);
});

export const url = greeting.httpsTriggerUrl;

Thanks to Pulumi’s language heritage, we know how to take that function, serialize it, and publish it to a Google Bucket and Object, eliminating those tedious manual steps behind a single pulumi up CLI invocation.

Or perhaps a pubsub topic that runs some custom code on every message received. This sounds like a perfect use case for an event-driven style of programming, so why don’t we write it that way!

// Create a PubSub Topic
let requests = new gcp.pubsub.Topic("requests");
requests.onMessagePublished("newMessage", (data) => {
    // Print out a log message for every message on the Topic.
    // Change this code to fit your needs!
    console.log(Buffer.from(data.data, "base64").toString());
});

Because Pulumi knows how to manage infrastructure as code, it can provision and manage not only the Google Cloud Function, but also the pubsub topic resource itself, for each creation and management.

Or perhaps, we’d like a function that respond to any uploads of new objects to your storage bucket:

// Create a Storage Bucket
let bucket = new gcp.storage.Bucket("data");
bucket.onObjectFinalized("newobject", (data) => {
    // Print out a log message for every upload to the Bucket.
    // Change this code to fit your needs!
    console.log(`New file uploaded: ${data.name}`);
});

A Complete Google Functions Slack Bot

For an idea of how you might fit these together to make a real-world cloud application, let’s look a simple skeleton structure for a Slack Bot using Pulumi and Google Cloud Functions:

// secure config tokens to use to validate incoming messages as well as authenticate ourself to slack
const config = new pulumi.Config("mentionbot");
const slackToken = config.get("slackToken");
const verificationToken = config.get("verificationToken");

// A topic that we can enqueue slack events to so they can be processed in batch later on
const messageTopic = new gcp.pubsub.Topic("messages");

// Create an http endpoint that slack will use to push events to us.
const endpoint = new gcp.cloudfunctions.HttpCallbackFunction("bot", {
    callbackFactory: () => {
        const app = express();
        app.use(bodyParser.json());
        app.post("/events", (req, res) => {
            // Importantly: This is the code that will run in your serverless GCP cloud function!

            const body = req.body;

            // Process the body as appropriate. If it's something we need to respond to immediately
            // (like a verification request), then do so. Otherwise, add the message to our pubsub
            // topic to be processed later:
            const pubSub = new PubSub();
            const topic = pubSub.topic(messageTopic.name.get());
            topic.publish(Buffer.from(JSON.stringify(body)));

            // Quickly respond with success so that slack doesn't retry.
            res.status(200).end();
        });

        return app;
    }
});

messageTopic.onMessagePublished("processTopicMessage", async (data) => {
    // Actually handle the 'data' in the pubsub message.
    // Importantly: This is the code that will run in your serverless GCP cloud function!
});

// Give this url to slack to let them know where to post their events to.
export const url = endpoint.httpsTriggerUrl;

This is the real code for a complete SlackBot application running on GCP, from the cloud resources to the serverless code, all within a unified Pulumi application! Customizing this for your own use case is as simple as changing the code in the two JavaScript arrow-functions.

To see the complete project, take a look at our @mentionbot example. That example will listen for mentions of your name and will notify you of them in a channel of your choosing, giving you you a persistent timeline you can go back to look at to make sure you can find all these messages.

Although it’s a simple example, there are a lot of moving parts this takes care of that you would normally be responsible for. This includes:

Figuring out the shape (the input/output-types) for your Cloud Functions, and then creating an appropriate program that exports the right entrypoint that matches. In this case, because we’ve exposed the right abstractions (like HttpCallbackFunction and Topic.onMessagePublished), the arrow-functions you pass in will all have the right types, and your program will be typechecked by TypeScript.
Creating separate Cloud Functions for each serverless callback. One for listening and responding to the initial Slack events and the second for processing the messages in the Topic. Here, you can write a single Pulumi App where all the code can be placed how you like it (in this case in a single file).
Packaging each callback up in the appropriate structure Cloud Functions expects, including how to get all your dependencies in place.
Creating a Storage Bucket for all of your packaged programs to live in.
Uploading each packaged program to a Bucket Object in that bucket.
Configuring the appropriate triggers on your Cloud Functions stating how they should be triggered (for example, in response to an HTTP trigger or a Pub/Sub trigger).
Including the right information in the function so you can interact with your other cloud resources in the Pulumi App. Without this, you would need to find a way to include that data in each Cloud Function’s environment variables (or just hardcode them in ‘1’) so that your program can access the rest of your cloud infrastructure. In the above example, you can see how you can just reference your resources directly (like the PubSub Topic) directly from your Cloud Function callback. Pulumi makes sure this all works, and that the data you use is available in that Cloud Function when it finally is triggered.
Figuring out a safe and secure way to encode and access secrets for your Cloud Function. Here, we can use Pulumi’s Config Secrets to safely encrypt and manage secrets for your Cloud Function code.

Not to mention that by doing all of that, you can achieve continuous deployment Pulumi and Google Cloud Build.

Updating Your Google Functions Code

This is a lot to figure out to manage cloud applications and infrastructure. If you want to tweak things even slightly you might need to go make many manual changes and updates to ensure everything is properly updated. With Pulumi, all this complexity is handled with a single update! Just run pulumi up , and it figures out the rest.

Let’s just take a look at what Pulumi does when you tweak a few things in the program:

$ pulumi up
Updating (slackbot):
     Type Name Status Info
     pulumi:pulumi:Stack slackbot
 +- ├─ gcp:pubsub:Topic messages replaced [diff: +labels]
     │ └─ gcp:cloudfunctions:CallbackFunction processTopicMessage
 +- │ ├─ gcp:storage:BucketObject processTopicMessage replaced [diff: ~name,source]
 ~ │ └─ gcp:cloudfunctions:Function processTopicMessage updated [diff: ~eventTrigger,sourceArchiveObject]
     └─ gcp:cloudfunctions:CallbackFunction mentionbot
 +- ├─ gcp:storage:BucketObject mentionbot replaced [diff: ~name,source]
 ~ └─ gcp:cloudfunctions:Function mentionbot updated [diff: ~sourceArchiveObject]
Outputs:
    url: "https://***.cloudfunctions.net/mentionbot"
Resources:
    ~ 2 updated
    +-3 replaced
    2 changes. 5 unchanged
Duration: 52s

One command later, and your entire stack is updated properly!

Winding Down

The cloud provides tremendous potential, and we want to make it easy for developers to tap into that potential. Using Pulumi, it’s easy to blur the line between business logic and serverless resources, bringing the focus back to the problems you care about. One codebase that’s easy to create, update, and maintain.

To check things out, get started today:

PS: If you’re interested in how Pulumi manages to take a JavaScript/TypeScript => function and somehow analyze and transform it into a form that Cloud Functions can use, please see our deep dive on this topic in: Lambdas as Lambdas: The magic of simple serverless Functions. We’re leveraging the same great programming language and analysis framework to power our GCP solution here.

Getting Started on Google Cloud Platform with Pulumi

Luke Hoban — Tue, 09 Apr 2019 00:00:00 +0000

Google Cloud is one of the most exciting cloud platforms available today, with a breadth of powerful infrastructure services from Google Container Engine (GKE) and Google Cloud Functions to Cloud Firestore and Cloud Spanner.

Pulumi is the most productive tooling available today for teams building cloud applications and infrastructure, in your favorite languages. Add them together, and teams can easily take maximum advantage of Google Cloud Platform’s rich features, productively, with a combined platform that makes it easy to collaborate, share, and reuse.

In this post, we’ll highlight some of the top features of Pulumi that make it a joy to work with Pulumi + Google Cloud:

Infrastructure as code for the Full Google Cloud Platform
Working with Kubernetes and Google Container Engine (GKE)
Super Simple Serverless with Google Cloud Functions
Continuous Delivery with Google Cloud Build
Managing Deployment with the Pulumi Service

Infrastructure as Code for the Full Google Cloud Platform

Pulumi lets you define and deploy infrastructure as code using your favorite programming languages, including JavaScript, Python, and Go. Pulumi provides access to the full Google Cloud Platform – every resource (including beta resources!) is available directly in your programming language and IDE. This delivers the software engineering benefits of your programming language, combined with the reliable and repeatable deployments of infrastructure as code.

In this example, we define a new network, firewall, and compute instance, using Python:

import pulumi
from pulumi_gcp import compute

disk = {
    'initializeParams': {
        'image': "centos-cloud/centos-7-v20190116"
    }
}

addr = compute.address.Address(resource_name='poc')

network = compute.Network("network")
network_interface = [
    {
        'network': network.id,
        'accessConfigs': [{'nat_ip': addr.address}],
    }
]

firewall = compute.Firewall("firewall", network=network.self_link, allows=[{
    'protocol': "tcp",
    'ports': ["22", "80"]
}])

instance = compute.Instance('poc', name='poc', boot_disk=disk, machine_type="f1-micro",
                            network_interfaces=network_interface)

pulumi.export('external_ip', addr.address)

Infrastructure can be deployed and updated using the pulumi CLI, with each update driving toward the desired state declared by the Pulumi program. Before each update, you can see a preview of the changes that will be made to your target infrastructure, and only after confirming will it proceed (with a full audit history):

$ pulumi update
Previewing update (luke):
     Type Name Plan Info
     pulumi:pulumi:Stack gcp-instance-nginx-luke
 ~ ├─ gcp:compute:Firewall firewall update [diff: ~allows]
 +- └─ gcp:compute:Instance poc replace [diff: ~bootDisk,name]
Resources:
    ~ 1 to update
    +-1 to replace
    2 changes. 3 unchanged
Do you want to perform this update? Yes
...

Learn more about working with Pulumi and Google Cloud infrastructure in this GCE Web Server tutorial.

Working with Kubernetes and Google Container Engine (GKE)

Pulumi also makes it easy to deploy resources to Kubernetes with access to the full Kubernetes API, across all object types and versions, using the same programming language and IDE experience. You no longer have to switch tooling at the boundary between GCP and Kubernetes. Instead, Pulumi gives you a single programming model for both.

In addition to the preview and auditing mentioned above, the Pulumi CLI also provides rich status and repeatable deployments to Kubernetes through its support for dependency-based orchestration of deployments:

And if you need a Kubernetes cluster, spinning one up with GKE using Pulumi is a breeze. Best of all you have easy access to the full breadth of configurability you might need: networking configuration, multiple NodePools, or clusters in multiple regions/zones. And you can easily add additional GCP infrastructure like managed databases, DNS configuration, or other resources needed to have a fully functional Kubernetes compute environment.

This example spins up a GKE cluster and uses Pulumi’s configuration system for easy parameterization:

from pulumi import Config
from pulumi_gcp.container import Cluster

# Read in some configurable settings for our cluster:
config = Config(None)

# nodeCount is the number of cluster nodes to provision. Defaults to 3 if unspecified.
NODE_COUNT = config.get('node_count') or 3
# nodeMachineType is the machine type to use for cluster nodes. Defaults to n1-standard-1 if unspecified.
# See https://cloud.google.com/compute/docs/machine-types for more details on available machine types.
NODE_MACHINE_TYPE = config.get('node_machine_type') or 'n1-standard-1'
# username is the admin username for the cluster.
USERNAME = config.get('username') or 'admin'
# password is the password for the admin user in the cluster.
PASSWORD = config.require('password')

# Now, actually create the GKE cluster.
k8s_cluster = Cluster('gke-cluster',
    initial_node_count=NODE_COUNT,
    node_version='latest',
    min_master_version='latest',
    master_auth={ 'username': USERNAME, 'password': PASSWORD },
    node_config={
        'machine_type': NODE_MACHINE_TYPE,
        'oauth_scopes': [
            'https://www.googleapis.com/auth/compute',
            'https://www.googleapis.com/auth/devstorage.read_only',
            'https://www.googleapis.com/auth/logging.write',
            'https://www.googleapis.com/auth/monitoring'
        ],
    },
)

Learn more about using Pulumi with Kubernetes and GKE in this Hello GKE tutorial.

Super Simple Serverless with Google Cloud Functions

Serverless has never been simpler than it is with Pulumi + Google Cloud Functions. Thanks to Pulumi’s “magic functions”, we can write callbacks in our language of choice, using a natural event-driven style, and Pulumi will handle the rest.

For example, want to serve a simple HTTP API with no fixed costs? It’s just a few lines of code:

import * as gcp from "@pulumi/gcp";

let greeting = new gcp.cloudfunctions.HttpCallbackFunction("greeting", (req, res) => {
    res.send(`Greetings from ${req.body.name || 'Google Cloud Functions'}!`);
});

export const url = greeting.httpsTriggerUrl;

Or perhaps a pubsub topic that runs some custom code on every message received:

// Create a PubSub Topic
let requests = new gcp.pubsub.Topic("requests");
// Print out a log message for every message on the Topic
requests.onMessagePublished("newMessage", (data) => {
    console.log(Buffer.from(data.data, "base64").toString());
});

And for quick inner loop development, the pulumi logs command supports pulling logs for your Google Cloud Functions from StackDriver, aggregating all compute logs in your program into one unified stream:

$ pulumi logs
Collecting logs for stack dev since 2019-04-08T12:02:48.000-07:00.
 2019-04-08T12:37:34.000-07:00[ newMessage-9d41c7a] Function execution started
 2019-04-08T12:37:34.000-07:00[ newMessage-9d41c7a] {"hello": "world"}
 2019-04-08T12:37:34.000-07:00[ newMessage-9d41c7a] Function execution took 70 ms, finished with status: 'ok'

Learn more about Pulumi support for Google Cloud Functions in this Simple Serverless Slackbot example.

Continuous Delivery with Google Cloud Build

Because Pulumi is infrastructure-as-code, the projects you deploy with Pulumi are robust and repeatable. Spinning up a testing or staging environment is simple, and continuously deploying infrastructure changes from commits and pull requests to your code is as easy as adding a few lines to your CI/CD system.

For teams working with Google Cloud Build, Pulumi deployments can be integrated into your continuous delivery with just a few steps. And for teams working with GitHub, Pulumi’s GitHub App gives you rich feedback on what changes will be deployed as part of a Pull Request, so you can see and approve infrastructure changes before they happen:

Learn more about integrating Pulumi with CI/CD in our continuous delivery documentation.

Managing Deployment with the Pulumi Service

In addition to the pulumi CLI for deploying and updating GCP infrastructure, Pulumi is always connected to the Pulumi Service, which provides a rich interface for viewing the history of your stacks and deployments, management of access controls for teams, and details of all resources under management.

With deep links into the Google Cloud Console, Pulumi provides quick access to all resources in your project.

And because Pulumi works with both Google Cloud and Kubernetes resources at the same time, it’s just as easy to deep link into resources in the Kubernetes console as well!

You can check out the Pulumi Service at app.pulumi.com and create your first Pulumi GCP project with just a few clicks, with an array of New Project options to get you up and running quickly.

Next Steps

Here at Pulumi, we love working with GCP and Kubernetes, and we are constantly working on ways to make Pulumi the most productive (and most fun!) way to program the Google Cloud Platform.

Pulumi is free and open source. You can get started with Pulumi today.

Here are a few resources to learn more about working with Pulumi and GCP:

Programming the Cloud with Python

Sean Gilespie — Thu, 04 Apr 2019 00:00:00 +0000

Across the industry, the popularity of Python is exploding. Amongst our own customers at Pulumi, who automate their infrastructure using Python, we’ve seen the same. Stack Overflow wrote about the astounding growth of Python:

The term “fastest-growing” can be hard to define precisely, but we make the case that Python has a solid claim to being the fastest-growing major programming language.

– David Robinson, Stack Overflow

Since Python is not a new language, what could be driving this incredible adoption curve?

Python is, and always has been, particularly amenable to the automation of previously-manual tasks. So much so, in fact, that books have been written on how to automate common tasks with Python. In recent years, Python’s penchant for task automation have led to its use at the foundation of tools now ubiquitous in the DevOps space: tools like Ansible, SaltStack, and OpenStack are all authored primarily in Python. It would not be a stretch to claim that Python is the language of automation.

At Pulumi we are passionate about automating cloud infrastructure. However, we see a major open opportunity to leverage our most powerful, fundamental tool - the programming language - to automate our infrastructure tasks. We love the idea of “infrastructure of code”, but we struggle with the industry’s definition of “code”; it’s hardly code at all, often consisting of YAML documents, DSLs, or other, domain-specific solutions to the task at hand. Since Python is such an incredible tool for task automation and since its use is rapidly pervading the industry, wouldn’t it make sense to use code, to use Python, to automate tasks as critical as the provisioning of our cloud infrastructure and delivery of our applications?

Delivery is not a detail, it is our job. Now is the time to apply our core skills to our own work. Now is the time to engineer our delivery. We divide our work between ourselves and computers: humans for decisions, and automation for tasks.

– The Software Defined Delivery Manifesto

That sounds great, but what is this “Pulumi” thing?

Pulumi is an infrastructure as code tool for provisioning cloud infrastructure and deploying applications. The key difference between Pulumi and other tools in this space (such as Ansible, Terraform, and AWS CloudFormation) is that Pulumi aggressively reclaims the word “code”: users of Pulumi write programs in general purpose programming languages that, when run, deploy and run applications in cloud environments, while still retaining all the known benefits of infrastructure as code that our industry has grown accustomed to (repeatability, resilience to failure, history and auditing, and so on).

Pulumi is a rebellion against YAML and domain-specific non-Turing-complete languages. We firmly believe that your infrastructure isn’t just code: your infrastructure is software. By applying the same engineering discipline to your infrastructure code that you apply to your application code, we can iterate quickly and with confidence, leaning on things such as automated tests, code analysis tools, and a common language. Pulumi does not generate YAML; it instead brings the full API surface area of your cloud environment into Python, powered by an infrastructure as code engine.

It is hard to overstate how liberating it is to write infrastructure as code with a full general-purpose programming language at your disposal. Below is a code snippet of a Pulumi program, written in Python, that deploys a static website to S3:

import mimetypes
import os

from pulumi import export, FileAsset
from pulumi_aws import s3

web_bucket = s3.Bucket('s3-website-bucket', website={
    "index_document": "index.html"
})

content_dir = "www"
for file in os.listdir(content_dir):
    filepath = os.path.join(content_dir, file)
    mime_type, _ = mimetypes.guess_type(filepath)
    obj = s3.BucketObject(file,
        bucket=web_bucket.id,
        source=FileAsset(filepath),
        content_type=mime_type)

There are only 18 lines of Python code here, but this program does a ton!

First, we create an AWS S3 Bucket that we’ll use to host our website from, setting the index document to be “index.html”.
Next, we iterate over all files in the “www” directory (using the standard Python for loop!
Finally, for every file in the “www” directory, we guess its MIME type using the Python standard library and then create an S3 BucketObject inside the website bucket, which uploads the file to S3.

This is a simple example, but it’s hard to overstate how awesome it is to be able to reach into Python’s deep standard library whenever we need to. In this example we’re using both the “os” and “mimetypes” packages in the Python standard library: the first to list the files in a particular directory and the second to guess a particular file’s MIME type based on its extension and contents, which we eventually pass directly to S3.

Remember, this code isn’t performing imperative commands — like, say, your cloud provider’s Python SDK or Boto would — and instead creates a declarative resource graph that the Pulumi engine understands how to act upon. This works for the first deployment, but more importantly, for all subsequent incremental updates too. If we were to change the contents of one of the files on disk, Pulumi would recognize that one of the files has changed and re-upload it, without changing the bucket or any of the other already existing bucket objects!

Where can I learn more?

Don’t take our word for it, though! We’d love for you to check it out and see for yourself how great it is to reclaim the “code” in “infrastructure as code”.

If you want to know more about Pulumi and the things it can do for you, check out the Getting Started page and Documentation for more information.

If you’d like to try the above example out for yourself, we’ve put the full code on GitHub. There are also dozens of other Python examples across AWS, Azure, GCP, Kubernetes and more you can explore.

Python is the language of automation today and for the future. Pulumi is the infrastructure as code automation tool of the future. Using them both together is an incredible way to automate a crucial job while staying entirely within a language you already know and love!

Easy Serverless Apps and Infrastructure

Joe Duffy — Mon, 11 Mar 2019 00:00:00 +0000

With Pulumi, you can create, deploy, and manage any cloud resource using your favorite language. This includes application and infrastructure related resources, often in the same program.

One area this gets really fun is serverless computing. Because we’re using general purpose languages, we can create resources, and then wire up event handlers, just like normal event-driven programming. This is the way serverless architecture should be!

In this article, we’ll see how. There’s a broad range of options depending on what you want to do, and how your team likes to operate. We’ll be using Amazon Web Services (AWS) and TypeScript, but other clouds and languages are available.

A Simple Serverless App

In this article, we’ll use a very simple serverless app. We have two S3 buckets: one for TPS reports, and the other to store archived zipfiles of those reports. Anytime a new report comes in, our Lambda will zip and archive it.

Simple enough. This can be visualized as follows:

All source code for this example is available on GitHub.

Event Sources and Event Handlers

Serverless app models today make you think of the event sources – the S3 buckets – and event handlers – the Lambdas and associated code – as very different things – “infrastructure” versus “app code” – managed with distinct tools and workflows. Pulumi, in contrast, gives you a single CLI, pulumi, to manage everything consistently.

You can create event sources and handlers in the same program, but Pulumi’s approach is flexible, allowing you to ease into serverless without boiling the ocean. You can choose to wire functions up to existing resources (say, if your team already created the buckets), or even use functions your team has already made.

Also, because Pulumi provisions and manages resources, updating your functions after creating them is easy. Just edit your code, run pulumi up, and Pulumi will diff and compute the minimal set of changes it can make to upgrade your code – no downtime required. We’ll see this in action shortly. But first let’s create our initial app.

Creating an Event Handler

There are multiple approaches to creating a function. Let’s start with the easiest.

Approach 1: Magic Functions

Our favorite way to create functions is to write them inline. This blurs the line between infrastructure and application logic in a way that makes Pulumi programs feel like real, first class distributed apps.

First, we have our TPS reports and zips buckets (for now we’ll new them up, more on that later):

import * as aws from "@pulumi/aws";
const tpsReports = new aws.s3.Bucket("tpsReports");
const tpsZips = new aws.s3.Bucket("tpsZips");

The bucket objects have many properties, including obvious ones like the ARN, domain name, CORS rules, and so on. But if we look closely, we’ll see there are some onX methods. These register event handlers:

Let’s use onObjectCreated to create a Lambda that will zip up any new reports:

tpsReports.onObjectCreated("zipTpsReports", (e) => {
   const AdmZip = require("adm-zip");
   const s3 = new aws.sdk.S3();
   for (const rec of e.Records || []) {
      const zip = new AdmZip();
      const [ buck, key ] = [ rec.s3.bucket.name, rec.s3.object.key ];
      console.log(`Zipping ${buck}/${key} into ${tpsZips.bucket.get()}/${key}.zip`);
      const data = await s3.getObject({ Bucket: buck, Key: key }).promise);
      zip.addFile(key, data.Body);
      await s3.putObject({
         Bucket: tpsZips.bucket.get(),
         Key: `${key}.zip`,
         Body: zip.toBuffer(),
      }).promise();
   }
});

Most of the code here is the application logic – precisely what we want to be focusing on! In fact, this code looks just like any ordinary JavaScript event-driven program.

A neat thing that Pulumi can do is capture references to other resources. So, notice here, we actually capture a reference to tpsZips from within the event handler, and Pulumi figures out how to serialize the resulting closure. For the most part, any code you write there will work, unlocking all sorts of powerful composition scenarios: multiple functions can be composed together, you can use modules as usual, and so on. Just program like you normally would, without the awkward environment variables and configuration. That’s why we call these magic functions.

Now, let’s deploy the resources to AWS:

After confirming, yes, everything is up and running after just a few seconds:

Note the resource graph. Pulumi uses familiar languages, which allows encapsulation and hiding of unnecessary complexity. So underneath that simple-looking onObjectCreated are many AWS resources, like IAM attachments. This is a lot like how we can ignore all the messy OS API calls when using Node.js abstractions.

Pulumi has many convenience functions built-in, and one of them is pulumi logs, which will let us tail the logs associated with all compute in our stack (including serverless functions):

$ pulumi logs -f Collecting logs for stack dev since 2019-03-10T10:09:56.000-07:00…

Now that we’re tailing the logs, let’s copy a file over using the AWS CLI:

$ aws s3 cp ./tps001.txt s3://$(pulumi stack output tpsReportsBucket) upload: ../tps001.txt to s3://tpsreports-96458ef/tps001.txt

We have used the pulumi stack output command to conveniently fetch the bucket name. After the upload completes, we’ll see that our function comes alive in the pulumi logs command:

2019-03-10T11:10:48.617-07:00[zipTpsReports] Zipping tpsreports-96458ef/tps001.txt into tpszips-edfde11/tps001.txt.zip

And, sure enough, if we list the contents of the zip bucket, we will see our newly added file:

$ aws s3 ls s3://$(pulumi stack output tpsZipsBucket) 2019-03-10 11:10:50 6206 tps001.txt.zip

Voila! A fully functioning serverless application.

Note also that the CallbackFunction class – the powerful abstraction behind all of this – is exported, and offers some knobs, in case you want to do things like reuse existing IAM roles rather than creating new ones. See the pulumi/aws/lambda documentation for details. For instance, say we want to increase the RAM available to our function from 128MB to 256MB:

tpsReports.onObjectCreated(
   "zipTpsReports",
   new aws.lambda.CallbackFunction("zipTpsReportsFunc", {
      memorySize: 256 /*MB*/,
      callback: (e) => { /* same code as before */ },
   },
);

Soon we’ll see how to manage and update our functions, but first let’s see some alternative coding styles.

Approach 2: Manual Function Resources

We just saw that we can write our serverless application logic inside of our Pulumi program, alongside our infrastructure resource definitions. This can feel magical and super productive. But not everybody wants to mix these together. Sometimes you want to stick to the classic way of doing this: keeping your function’s application logic in one place, and your function’s infrastructure definition in another. This aligns more closely with many team workflows.

There is another practical reason to want this: if your functions are written in one language (say, C# or Java) while your Pulumi infrastructure is authored in a different one (say, JavaScript or Python).

Pulumi supports this, by simply defining your Lambda function like any other infrastructure resource:

// First, create some IAM machinery:
const zipFuncRole = new aws.iam.Role("zipTpsReportsFuncRole", {
   assumeRolePolicy: {
      Version: "2012-10-17",
      Statement: [{
         Action: "sts:AssumeRole",
         Principal: {
            Service: "lambda.amazonaws.com",
         },
         Effect: "Allow",
         Sid: "",
      }],
   },
});
new aws.iam.RolePolicyAttachment("zipTpsReportsFuncRoleAttach", {
   role: zipFuncRole,
   policyArn: aws.iam.ManagedPolicy.AWSLambdaExecute,
});

// Next, create the Lambda function itself:
const zipFunc = new aws.lambda.Function("zipTpsReportsFunc", {
   environment: {
      variables: {
         "TPS_ZIP_BUCKET": tpsZips.bucket,
      },
   },
   code: new pulumi.asset.AssetArchive({
      ".": new pulumi.asset.FileArchive("./app"),
   }),
   runtime: "nodejs12.x",
   role: zipFuncRole.arn,
});

// Finally, register the Lambda to fire when a new TPS report arrives:
tpsReports.onObjectCreated("zipTpsReports", zipFunc);

Because we’re taking matters into our own hands, we have to create the IAM machinery ourselves. We also need a way to communicate the zip bucket name, so we use an environment variable. This highlights both the benefits and drawbacks to programming at this level – we need to know how to configure all of these ancillary resources, but as a result, the entire power of Lambda is at our fingertips.

Notice that we’ve pointed to our application logic inside of ./app. Pulumi will create the zipfile for you. If we instead wanted to use a zipfile we’ve already packaged, just change code as follows:

// ...
   code: new pulumi.asset.FileArchive("./app.zip"),
// ...

Using Pulumi’s Asset and Archive classes, we can fetch code from anywhere – even the network.

Although managing your functions manually isn’t quite as magical, it is practically useful, and still delivers all the usual infrastructure as code benefits, enabling robust infrastructure management and versioning.

Approach 3: Wire Up an Existing Function

This post is mainly about using serverless functions with Pulumi. But what if you’re using Pulumi mainly for infrastructure, and another solution for the functions themselves? No problem – Pulumi can still glue everything together, even if it’s not provisioning all of the resources.

Every resource type in Pulumi has a static get that looks up an existing resource. These resources aren’t managed by Pulumi, but you can still access their properties and use them. In this case, we’ll just look up an existing Lambda, zipTpsReportsFunc, and register it as an event handler:

const zipFunc = aws.lambda.Function.get("zipTpsReportsFunc", "zipTpsReports-19d51dc");
tpsReports.onObjectCreated("zipTpsReports", zipFunc);

We’ve given the function’s ID, zipTpsReports-19d51dc, which allows Pulumi to locate it in your account and reuse it. This can make it easy to incrementally adopt Pulumi one piece at a time, collaborate between teams, or stitch together resources managed by different stacks.

More About Functions

Now that we’ve seen the basics, let’s look at a few associated function management topics.

Updating a Function

Updating the function is as easy as can be. Imagine we’re running into a problem and want to do some good old fashioned printf debugging inside of our lambda. Maybe we want to log the size of the report and its resulting zipfile. Just add the relevant line:

// ...
zip.addFile(key, data.Body);
console.log(`Report: ${data.ContentLength}, Zip: ${zip.toBuffer().length}`);
s3.putObject(
// ...

And run pulumi up – it will show us what changed:

We can view the full diff, by choosing details, including a Git-like diff of the code changes themselves!

After selecting yes, the function is updated in just a couple of seconds. Similarly, we can update any of the resource definitions, and Pulumi will figure out how to incrementally update them.

Splitting Up your Codebase

A nice middle ground between magic and manual functions is to use your language’s module system to structure code differently. This is similar to how you might structure a typical application: route definitions over here, business logic over there, markup over here, etc. Pulumi can figure out the diffs regardless of how you’ve structured your code, so updates are always based only on what’s changed.

For example, maybe we’ve defined our callback function in ./app:

export async function zipReport(e: aws.s3.BucketEvent): Promise<void> {
   // app code, as shown above, goes here
}

And now we can go back to our infrastructure code, and eliminate the application logic entirely:

import { zipReport } from "./app";
...
tpsReports.onObjectCreated("zipTpsReports", zipReport);

We can take this further and use dynamic package management to split up the code, possibly even spreading pieces of infrastructure and application code across multiple repos and/or packages. This works well for larger teams with independent components versioning at their own pace.

Lastly, it’s possible to use Pulumi stacks to actually break apart your cloud resources and functions into independently deployable pieces. This allows teams to leverage features like RBAC. For instance, it’s common for the DevOps team to manage the physical cloud resources like queues, topics, and buckets, while the development team authors and manages the serverless functions attached to them. Read more about this in the organizaing projects & stacks documentation.

More About Event Sources

We glossed over one of the more interesting points: what resources have event handlers associated with them, and how to get our hands on them.

Provision a New Resource

The simplest answer here is to create a new resource in your Pulumi program using new, as we saw above. Because Pulumi is an infrastructure as code platform, any resources in any cloud are available – AWS, Azure, GCP, Kubernetes, etc. When you new one up, Pulumi understands how to provision and manage it.

We saw simple examples of this earlier:

const tpsReports = new aws.s3.Bucket("tpsReports");

Note that the full set of configuration options are available on each resource. Let’s say we want to turn on server-side encryption of our TPS archive bucket:

const tpsReports = new aws.s3.Bucket("tpsReports", {
   serverSideEncryptionConfiguration: {
      sseAlgorithm: "AES256",
   },
});

Just as with functions, subsequent updates will be diffed and updated in the minimally impactful way.

Connect to an Existing Resource

Although it’s nice we can provision the event sources and targets in a single program, sometimes different members of the team manage different parts of the infrastructure. For example, maybe your DevOps engineers provision resources like buckets, topics, and so on, and your developers wire up the functions.

Not to worry, you can import an existing resource and program against it in the same way:

const tpsReports = aws.s3.Bucket.get("tpsReports", "arn:aws:s3:::tpsReports-4f64efc");

This time, when you run your Pulumi program, you’ll still see this resource but notice it says read instead of create or update. This just means that the resource is read from your account as-is. No matter what you do, the bucket itself will not be modified by Pulumi, other than to subscribe an event.

This, of course, can be combined with the earlier similar functionality for functions, to glue together a bucket and a Lambda, where neither was actually provisioned by Pulumi!

const tpsReports = aws.s3.Bucket.get("tpsReports", "arn:aws:s3:::tpsReports-4f64efc");
const zipFunc = aws.lambda.Function.get("zipTpsReportsFunc", "zipTpsReports-19d51dc");
tpsReports.onObjectCreated("zipTpsReports", zipFunc);

Other Notable Event Sources

Finally, it’s worth noting there are many event handler functions exposed in the AWS package:

apigateway.x.API: create serverless APIs using an Express.js style
cloudwatch.onSchedule: fire a CloudWatch event on a particular schedule, e.g. a cron expression
cloudwatch.Event.onEvent: fire an event when a particular CloudWatch event occurs
cloudwatch.LogGroup.onEvent: fire an event when a CloudWatch logs event occurs
dynamodb.Table.onEvent: fire events for DynamoDB insert, modify, or remove operations
kinesis.Stream.onEvent: fire Kinesis Stream events at particular times or batch sizes
s3.Bucket.onObjectCreated: trigger a function anytime an object is created in an S3 Bucket
s3.Bucket.onObjectRemoved: trigger a function anytime an object is removed from an S3 Bucket
s3.Bucket.onEvent: trigger a function for a wide range of S3 Bucket events
sns.Topic.onEvent: fire SNS Topic events when new messages arrive
sqs.Queue.onEvent: fire SQS Queue events when new messages are enqueued (or on DLQ events, etc)

All of these handlers can be programmed using the spectrum of techniques outlined above.

In Conclusion

In this post, we saw many different options for serverless programming with Pulumi, from magic functions that make serverless apps feel like true event-driven programming, to provisioning and managing infrastructure as code.

We also saw that all of these options can be combined in any way, depending on you and your team’s needs. This is the magic of programming languages in action!

If you’d like to try it out, check out the code for this post on GitHub. Happy serverless hacking!

Serverless on AWS with Pulumi: Simple, Event-based Functions

Cyrus Najmabadi — Mon, 14 Jan 2019 00:00:00 +0000

One of Pulumi’s goals is to provide the simplest way possible to do serverless programming on AWS by enabling you to create cloud infrastructure with familiar programming languages that you are already using today. We believe that the existing constructs already present in these languages, like flow control, inheritance, composition, and so on, provide the right abstractions to effectively build up infrastructure in a simple and familiar way.

In a previous post we focused on how Pulumi could allow you to simply create an AWS Lambda out of your own JavaScript function. While this was much easier than having to manually create a Lambda Deployment Package yourself, it could still be overly complex to integrate these Lambdas into complete serverless application.

To get a sense of that complexity, let’s look at how one would normally have to work with AWS’s resource system to create a simple Serverless application:

// Simplified for brevity
import * as aws from "@pulumi/aws";
import * as slack from "@slack/client";

// Create a simple bucket.
const bucket = new aws.s3.Bucket("testbucket", {
 serverSideEncryptionConfiguration: ...,
 forceDestroy: true,
});

// Create a lambda that will post a message to slack when the bucket changes.
// We can pass a simple JavaScript/TypeScript lambda here thanks to the magic of "Lambdas as Lambdas"
// See: www.pulumi.com/blog/lambdas-as-lambdas-the-magic-of-simple-serverless-functions/
const lambda = new aws.lambda.CallbackFunction("postToSlack", {
 callback: async (e) => {
 const client = new slack.WebClient(...);
 for (const rec of e.Records) {
 await client.chat.postMessage({ ... });
 }
 },
 ...
});

// Give the bucket permission to invoke the lambda.
const permission = new aws.lambda.Permission("invokelambda", {
 function: lambda,
 action: "lambda:InvokeFunction",
 principal: "s3.amazonaws.com", sourceArn: bucket.id.apply(bucketName => `arn:aws:s3:::${bucketName}`), }));

// Now hookup a notification that will trigger the lambda when any object is created in the bucket.
const notification = new aws.s3.BucketNotification("onAnyObjectCreated", {
 bucket: bucket.id,
 lambdaFunctions: [{
 events: ["s3:ObjectCreated:*"],
 lambdaFunctionArn: lambda.arn,
 }],
})

Phew… that’s a lot of code :-/ So what happened above? Well, in the AWS resource-oriented view of the world, most things are nouns (i.e objects). Your bucket is an object. The lambda is an object. The permission is an explicit object. And even the connection between the bucket and lambda (the BucketNotification) is itself an object. That’s one way to decompose the world, and it makes sense in some domains where everything is just ‘data’. But that isn’t a natural fit for a programming language, and it just doesn’t feel very idiomatic or appropriate for how people would expect things to work in their programming language of choice. Programming languages have lots of other facilities at their disposal to work with data. You can have methods/functions/messages you can send to an object. You can have events and callbacks on objects. You can encapsulate objects behind other objects. You can ‘run code’ to do things based on those objects, etc. etc.

Now, while data should be the start of how you interact with system, it shouldn’t be the end. Toward that goal, Pulumi makes it possible to simply compose and connect these resources in a more natural fashion. By just adding our own components, and leveraging what programming languages can already do, we can make it possible to simplify the above down to:

import * as aws from "@pulumi/aws";
import * as slack from "@slack/client";

// Create a simple bucket.
const bucket = new aws.s3.Bucket("testbucket", {
 serverSideEncryptionConfiguration: ...,
 forceDestroy: true,
});

// Create a lambda that will post a message to slack when the bucket changes.
bucket.onObjectCreated("postToSlack", async (e) => {
 const client = new slack.WebClient(...);
 for (const rec of e.Records) {
 await client.chat.postMessage({ ... });
 }
});

This now feels far more like how one might expect to express this concept with a normal application. A simple conceptual idea now maps to a simple code pattern. Instead of manually creating objects to represent the connection between the Bucket and the Lambda, we simply directly ask the Bucket to call the Lambda when the ObjectCreated event fires. Because this is all ‘code’, we can take care of all the boring cruft (like creating permissions) on your behalf. Of course, if you need to tweak this, that’s still possible. We believe you should always be in control of what’s going on. The actual low level details of how that is done are interesting as well, but a followup post will explain how we did it (and how you can do it too!). If you want to create your own composable modules for making cloud deployments easier then definitely look for that post!

Now, while it’s how the above examples worked, it’s also worth noting that the use of a JavaScript function for the AWS Lambda is not required. You can hook up a serverless event to call an AWS Lambda you create just by using new aws.lambda.Function. Or, you can get a reference to an existing AWS Lambda created outside of Pulumi and have that be the receiver of your serverless event. Here’s how that would look:

import * as aws from "@pulumi/aws";
import * as slack from "@slack/client";

// Create a simple bucket.
const bucket = new aws.s3.Bucket("testbucket", {
 serverSideEncryptionConfiguration: ...,
 forceDestroy: true,
});

// Retrieve an existing Lambda Function already created in your AWS infrastructure.
const func = aws.lambda.Function.get("postToSlack", "... existing lambda arn ...");

// Call that Lambda Function when an object is created in the bucket.
bucket.onObjectCreated("postToSlack", func);

We’ve tried to make it this simple to hook up many interesting AWS serverless events. For example, you can register to hear about events on S3 Buckets, SNS Topics, SQS Queues, Kinesis Streams, DynamoDB Tables, Cloudwatch Events, and more. We’ve tried to provide these prebuilt components for the most common and interesting cases. If there’s some serverless event we haven’t added support for that you want to use, definitely let us know!

Finally, because we think it’s great to be able to use a JavaScript or TypeScript function as the code for your Lambda, we’ve introduced a simple quality-of-life improvement for writing those functions. Specifically, because it’s so common that one will use the real AWS SDK for Node.js within a Lambda, we’ve provided a quick and simple way to get to a fully initialized instance of that API without needing any additional requires or imports in your code. Here’s how that would look:

// The only module you need to import.
import * as aws from "@pulumi/aws";

// ...

bucket.onObjectCreated("postToSlack", async (e) => {
 // direct access to the aws-sdk through `aws.sdk`.
 const sqs = new aws.sdk.SQS();
 sqs.sendMessage(/*...*/);
});

This allows you to use @pulumi/aws as both the deployment-time API to define your AWS infrastructure, as well as being the run-time API that you can use when your Lambda executes to access the full set of AWS functionality.

To get a sense of a more complete example of how this all ties together you can see many of our examples. One example that helps demonstrate many of the above concept is Twitter+Athena. In it, we use serverless events to define a schedule, a callback to invoke when the even t fires, use of a third-party twitter API to retrieve a set of relevant Tweets, use of the aws-sdk to store the Tweet information into an S3 Bucket, and finally the creation of an Athena Query to extract information from the information stored into the bucket. You could augment this example with any number of resources and events for all sorts of interesting use cases.

As we’ve seen, with a tiny amount of code, you can easily define the cloud resources you want to create (or reference your existing cloud resources). It’s then simple to add code to respond to events and call new or existing Lambdas. And, if you use a JavaScript function to create your Lambda, you can reference those resources along with the AWS SDK easily, all within the same application code!

Epsagon: Define, Deploy and Monitor Serverless Applications

Luke Hoban — Thu, 29 Nov 2018 00:00:00 +0000

Pulumi makes it incredibly easy to use serverless functions within your cloud infrastructure and applications - an AWS Lambda is as simple as writing a JavaScript lambda!

const bucket = new aws.s3.Bucket("my-bucket");
bucket.onObjectCreated("onNewObject", async (ev) => console.log(ev));

By making it so easy to introduce serverless functions into cloud infrastructure, Pulumi programs often incorporate many Lambdas, all wired together as part of a larger set of infrastructure and application code.

Epsagon is a serverless monitoring solution that lets users observe entire serverless applications composed of many functions and other AWS infrastructure, instead of just looking at individual functions one at a time. This is a great fit for Pulumi applications!

Pulumi users can use the @pulumi/epsagon package to automatically get the benefits of Epsagon monitoring for all of the serverless functions defined in their Pulumi applications. By just adding two lines of code - every AWS Lambda created from a JavaScript callback (like in the example above) will automatically get instrumented with Epsagon’s monitoring library.

const epsagon = require("@pulumi/epsagon");
epsagon.install(pulumi, { appName: "my-example" });

Monitoring a Serverless Pulumi Application

I have a small Pulumi application I used to compare the latency of various Lambda event sources. In about 200 lines of code, it does the following:

Provisions required infrastructure for various AWS event sources
Hooks up Lambda event handlers to this infrastructure
Describes the code to run in those event handlers (using JavaScript callbacks)
Creates shared infrastructure for storing test results, and uses it inside the event handlers (via a generic test harness)
Creates and exposes a REST API for kicking off tests and viewing graphs of test results

Each test case looks like this (you can see more details in the README):

// Test for SNS Topic
const topic = new aws.sns.Topic("my-topic");
function sendTopicValue(val: number) {
return (new aws.sdk.SNS()).publish({
TopicArn: topic.arn.get(),
Message: val.toString(),
}).promise();
}
const topicSubscription = topic.onEvent("ontopicevent", (ev, ctx) => {
return harness("sns", Number(ev.Records[0].Sns.Message), sendTopicValue);
});

All up, it creates 6 Lambda functions connected to a variety of event sources.

I add the two lines of code above to add Epsagon, and set a config secret for the Epsagon API token:

pulumi config set –secret epsagon:token <my-epsagon-token>

Then, after kicking off a test using the REST API exposed by my Pulumi application, I go over to the Epsagon console for my project to see what happened. First, I can get a birds-eye view of my overall architecture. This provides a great visualization of how my resources are related at runtime. Each event source triggers a lambda attached to it, which both triggers that same event source again, and also logs to a central datastore that keeps track of statistics.

I can also drill in and see details about each piece of this, and what performance looked like across all of these.

And finally, I can see what chain of events in my infrastructure led to an operation occurring in DynamoDB - in this case, a REST API was handled by a Lambda, which invoked a Table scan.

Conclusion

Together, Pulumi and Epsagon make it easy to build truly serverless applications – not just a single Lambda, but a whole application made up of many small event driven callbacks that run on-demand. Pulumi makes it easy to author and deploy these kinds of architectures, and Epsagon makes it easy to monitor them. Users get an application-centric view across their full serverless application infrastructure.

Checkout the quickstart guides for Pulumi and Epsagon now!

Lambdas as Lambdas: The magic of simple serverless Functions

Cyrus Najmabadi — Wed, 10 Oct 2018 00:00:00 +0000

Pulumi’s approach to infrastructure as code uses familiar languages instead of YAML or DSLs. One major advantage of this approach is that AWS Lambdas, Azure Functions, Google Cloud Functions, et al. can just be real lambdas in your favorite language, offering a flexible and simple path to serverless. Such functions behave as normal functions, allowing you to treat serverless code as part of your application instead of separate “infrastructure” that needs to be configured, managed, and versioned manually. In this post, we’ll examine this capability in JavaScript, which is already very function- and callback-oriented, making serverless feel like a natural extension of the language we already know and love.

While Functions as a Service (FaaS) systems have become more popular, getting up and running can still feel overly complex compared to normal application development. FaaS offerings today divide the development experience between “infrastructure” – doing all the work to configure the Lambda runtime itself (i.e. how much memory to use, what environment variables should be present, etc.) – and writing and maintaining the code that will execute in the function itself when triggered. Most developers just want to focus on the latter, write some code, and have it work.

For AWS Lambda, for example, the code itself needs to written and packaged up with all its dependencies into a single zip-file, stored as a blob that is then placed in AWS S3. This blob is then referenced by Lambda so that it can be loaded on-demand when necessary. Updating just a single character of this code, like fixing a bug, then requires fully repackaging and re-uploading everything. Compounding all of that, this code is necessary independent of any other pieces of your application. If the code for the Lambda needs to read or write to an S3 bucket, publish to an AWS SNS topic, etc., it will need to be manually configured to have that bucket’s information passed to it, often using out of band in environment variables. These same issues also apply when creating FunctionApps using Node.js on Azure.

At Pulumi, we thought there could be a better way to do this. Why have to jump out of your application code just to create another bit of configuration (in YAML or JSON) just to run your application? Put simply, we wanted developers to write event-driven code, in their favorite language and in their favorite IDE, and simply hit “deploy” when done.

Magic Functions

From a desire to do better here, the “closure serialized function” was born. However, “closure serialized function” is a geeky mouthful, so I like just calling them “magic functions.” Our team is comprised of cloud, language, and runtime geeks alike, so we figured out a way to hook into the language runtimes to just make lambdas as lambdas work.

Before diving more deeply into how magic functions work, let’s take a look at how they look in action:

// Example simplified to not get too bogged down :)
const sharp = require("sharp");
const videoBucket = new aws.s3.Bucket("videos");
const videoTopic = new cloud.Topic("videoTopic");
videoBucket.onObjectCreated("onNewVideo", async (event) => {
for (const record of event.Records) {
const dimensions = computeDimensions(record.s3.object.key);
videoTopic.publish({ s3ObjectKey: record.s3.object.key, time: record.eventTime });
}
});
function computeDimensions(objectKey: string): { height: number, width: number } {
/* simplified */
const metadata = await sharp(...);
return { height: metadata.height, width: metadata.width };
}
videoTopic.subscribe("compressVideo", async (event) => {
const { s3ObjectKey, time } = event;
// actually compress video.
});
videoTopic.subscribe("createThumbnail", async (event) => {
const { s3ObjectKey, time } = event;
// actually create thumbnail.
});

Let’s still walk through it to see what’s happening. It can be a little subtle because we are mixing deployment-time code – code that declares infrastructure as code resources – and *runtime *code – application code that runs as a function.

First, we’re just defining two simple resources: 1) an s3.Bucket, where we expect new video files to be uploaded to, and 2) a cloud.Topic that will inform interested parties when new videos are uploaded. (The full surface area of AWS is available in the aws package, and the cloud package offers multi-cloud abstractions that work at a higher level of abstraction.) Right after defining the resources, we start creating our first FaaS resources. videoBucket has an onObjectCreated event subscription function that will fire when new objects appear, and we invoke it with a real JavaScript (or TypeScript) arrow-function defining the code that will execute in an AWS Lambda at runtime.

Not So Fast…

This code is pretty magical in a few ways. To see how it’s magic, let’s consider a fairly naive way to create a Lambda for this onObjectCreated event handler function – something we need to do by hand with FaaS solutions today. Imagine we took the raw source text of this arrow-function by toStringing it and using that as the text of the Lambda code:

// index.js
module.exports = async (event) => {
foreach (var record in event.Records) {
const dimensions = computeDimensions(record.s3.object.key);
videoTopic.publish({ s3ObjectKey: record.s3.object.key, time: record.eventTime });
}
};

Now, the above translation could have worked for an extremely basic function, but totally breaks down here for several reasons. First, computeDimensions isn’t included, which will make this immediately fail if ever called. Second, if computeDimensions was included, it wouldn’t work unless the sharp module was properly imported for it. Third, even if the module was imported in the code, the actual NPM package would have to be available along with the code in the cloud. Fourth, if that module was properly required, and the package was properly included, the code is both referencing videoTopic (a resource defined when the application originally ran), and it is invoking a helper method on it. Remember, this code is going to be executing at cloud runtime, not at the time when Pulumi is actually executing the application on your developer machine, and actually doing things like running the const videoTopic = new cloud.Topic("videoTopic") code.

To summarize, we have quickly realized, we can’t just analyze everything locally to figure out what has to be included, instead the entire transitive closure must be considered – a task usually left up to FaaS developers to perform manually! This coupled with the manual configuration using YAML can lead to plenty a headache.

Compilers to the Rescue

Despite all those issues, Pulumi is able to make that original application code work and it is able to magically effectively translate the above into a working Lambda. The real secret sauce here is that, by leveraging familiar languages, we can stand on the shoulders of all the great compiler work underpinning them.

So let’s dig in – how does this all really work?

(Note that, if you want some of the more nitty gritty details, we have a doc going in depth. For now, we’ll try to stick to providing a high-level overview of what’s going on, so that we can grok the basic approach.)

Pulumi executes your application as a normal Node.js application. As such, when it hits the const videoBucket = new aws.s3.Bucket("videos") lines, it actually goes and creates those resources. When it then hits the videoBucket.onObjectCreated("onNewVideo", async (event) => { line this executes a normal line of JavaScript. Namely, onObjectCreated is a normal JavaScript function that will be passed the name parameter as the first arg, and the JavaScript function as the second arg. onObjectCreated eventually bottoms out with a call to a call to pulumi.runtime.serializeFunction. This function is the root of where our deep analysis of your JavaScript function analysis and translation will happen.

It’s worth pausing here. We can use all the standard approaches to function composition to build bigger functions out of smaller functions, even before we hit that serialization routine. The closures are built up in the V8 runtime as usual, and are eagerly awaiting the transformation into magic functions as soon as they are needed. Pretty cool!

Once we get to that point, using the awesome analysis APIs provided by TypeScript, we start introspecting the function instance, seeing what code it contains, and what external objects it references. The legal objects that can be included in the captured function’s environment include (but aren’t limited) to things like simple JavaScript objects, other functions (like computeDimensions), modules (like sharp), other Pulumi resources (like videoTopic), and even complex beasts like Promises. Basically, anything that Pulumi knows how to serialize and rehydrate inside the Lambda.

When Pulumi determines that these objects are referenced, it automatically includes enough information about them into the final translated Lambda code file so that they are available and can be effectively used at runtime. Thanks to JavaScript’s deep, dynamic introspection capabilities, as well as its rich APIs for creating nearly anything at runtime, it’s very possible to effectively hydrate out all the information present when Pulumi runs, and put it into a form that can be cheaply and accurately rehydrated into runtime objects again when the Lambda runs.

This analysis also understands the NPM package structure of an application and its dependencies, ensuring that necessary imported packages are properly included with the final Lambda. This analysis is also very smart about not including things that aren’t necessary at all. For example, it might drop information from the serialized form if it can prove that would never be observable, in addition to relying on built-in runtime functions being there without needing any sort of serialization. There are definite subtleties and interesting cases that can arise here, and for those very curious, reading the documentation linked above may be very enlightening.

Pulumi also understands diffs and dependencies, so updating is a cinch. Any changes to the code in the JavaScript function (including anything it depends on) is properly tracked and understood by Pulumi, allowing an easy development model where FaaS code can be easily modified and republished without jumping through extra hoops.

The Result? Simple serverless Functions

With this approach, serverless programming becomes dramatically simpler, whether it’s an AWS Lambda, Azure Function, or GCP Cloud Function. (Stay tuned, we’re hoping to bring this to Kubernetes FaaS offerings soon.)

In short, you just write event-driven JavaScript code that’s familiar and easy, you run pulumi up, and everything is taken care of from there. No YAML configuration files, no point and clicking in a cloud console UI, no fuss.

On top of all of that, because this is just code itself, and because these are normal APIs, other tooling (like your editor) knows exactly what is going on and can guide you through this. For example:

While there’s a lot of really cool tech going on under the covers here, we really hope that this ends up just feeling super-awesome to use for you. Now that we have this functionality, we wouldn’t want to ever write an AWS Lambda any other way!

Get started building serverless functions on Pulumi today!

Running a Serverless Node.js HTTP Server on AWS and Azure

Cyrus Najmabadi — Tue, 02 Oct 2018 00:00:00 +0000

The newly introduced cloud.HttpServer in Pulumi makes it easy to serve a standard Node.js HTTP server as a serverless API on any cloud platform. This new API brings together the flexibility and rich ecosystem of Node.js HTTP servers, the cost and operational simplicity of serverless APIs, and the multi-cloud authoring and deployment of Pulumi. In this post, we walk through some of the background on why we introduced this new API and how it fits into the Node.js HTTP ecosystem.

Almost 10 years ago, Node.js was introduced, helping to usher in an era of server-side JavaScript development. From the beginning, Node.js made it simple and easy to stand up an HTTP server and to start listening and responding to requests in an environment that felt natural to people already comfortable working with JavaScript. From the earliest commits this was a core focus, and from early on Node.js exposed the initial JavaScript API for creating an HTTP Server. That API evolved over time to become the well known HTTP module that is the backbone of so many projects.

The approach taken by Node.js was to provide an API, http.createServer, to easily stand up a reliable and performant HTTP server, but it left the processing and control of those actual HTTP conversations up to a simple callback function that a user of the API would provide. This callback function itself was very simple, with the entire API having the form:

http.createServer((req, res) => ...);

With this a developer could now easily plug in a function that would be called with the information about an incoming request in the callback’s first parameter, and which could supply the appropriate response through the callback’s second parameter. This approach allowed Node.js to focus on being simple and easy to get the server stood up, while being relatively unopinionated giving the developer a lot of flexibility in terms of how they would actually process requests and responses.

Thanks to this split, the Node.js ecosystem then came forward to provide more structured design patterns to handle those requests and responses to make things more manageable for developers. These communities provided ‘middleware’ APIs that could bridge between this extremely basic entry-point into one more structured and opinionated, but simpler to understand and work with. Middleware libraries like Express.js allowed one to write simple route-handlers like:

const app = express();
// GET method route
app.get('/', function (req, res) {
res.send('GET request to the homepage')
})
// POST method route
app.post('/', function (req, res) {
res.send('POST request to the homepage')
})
http.createServer(app).listen();

These route-handlers also have extensible middleware points to plug in other libraries (like Body Parser, Passport, and many others). This large effort across the entire ecosystem has made the task of serving websites using JavaScript dramatically simpler, and has helped propel a huge amount of popularity in this space.

Interestingly enough though, the Cloud space has gone a slightly different route here (pun intended). While cloud providers like AWS and Azure provide ways to both create HTTP servers and to use Node.js, they offer a very different shape altogether for processing HTTP messages. For example, in AWS, one needs to first set up an API gateway. That gateway will then have ‘methods’ defined on it which point at ‘Lambda Proxies’ to finally process the request. The connected AWS Lambda then will manage the request and response in a decidedly AWS-specific manner. For example, instead of working with IncomingMessage and ServerResponse like one normally would with Node.js, AWS specific messages like so are the required. Responding to messages with your own data uses a very different form as well. Code has to be written like so:

exports.handler = function(event, context, callback) {
 // process AWS 'event' using information from AWS 'context'
 // ...
 // build response
 var response = {
 statusCode: responseCode,
 headers: {
 "x-custom-header" : "my custom header value"
 },
 body: JSON.stringify(responseBody)
 };

 callback(null, response);
}

Azure follows its own distinct style as well for processing HTTP messages. These approaches end up working, but often feel decidedly un-Node.js-like. They’re full of cloud-provider-specific values and behaviors, and they end up making it more difficult to work with the existing ecosystem of middleware components out there. (Note that Google Cloud HTTP Functions actually do natively support the Node.js request/response pattern!).

We thought this was a place we could definitely help Pulumi applications with a simpler approach for these cloud ecosystems. To further this goal, we’ve created a new API called cloud.HttpServer. HttpServer is a Pulumi Resource, but is designed to work well with the existing large middleware ecosystem out there. And critically, the same HttpServer API can be implemented consistently on AWS, Azure and GCP - so you can write once and deploy to any cloud. The core API shape that accomplishes this is:

// Factory function for creating a requestListener function. The returned function is the same
// callback function that would be passed to http.createServer. See:
// https://nodejs.org/api/http.html#http_http_createserver_options_requestlistener for more details.
export type RequestListenerFactory = () => (req: http.IncomingMessage, res: http.ServerResponse) => void;

export interface HttpServerConstructor {
 /** * @param createRequestListener Function that, when called, will produce the [[requestListener]] * function that will be called for each http request to the server. The function will be * called once when the module is loaded. As such, it is a suitable place for expensive * computation (like setting up a set of routes). The function returned can then utilize the * results of that computation. */
 new (name: string, createRequestListener: RequestListenerFactory, opts?: pulumi.ResourceOptions): HttpServer;
}

The idea here is that a Pulumi app can now simply take the previous Express.js example and rewrite it as:

// The `() => { ... }` callback will actually become the code inside an AWS Lambda!**
const server = new cloud.HttpServer("myserver", () => {
 const app = express();

 // GET method route
 app.get('/', function (req, res) {
 res.send('GET request to the homepage')
 })

 // POST method route
 app.post('/', function (req, res) {
 res.send('POST request to the homepage')
 })

 return app;
});

The majority of this code is identical to the original Node.js form. The only small difference is the creation of the cloud.HttpServer resource (which is needed for Pulumi to track work and dependencies) and the returning of the ‘app’ variable instead of directly calling http.createServer.

Pulumi will take that code and convert the JavaScript callback^**^ into all the cloud-provider-specific resources necessary to make this work. For example, on AWS, an API Gateway and Lambda. The API Gateway will be properly setup to communicate with the Lambda function, and the Lambda function will properly handle AWS’s specific input/output forms and will map them along so they can properly work with Node.js’ http library expectations.

Effectively, the above callback will get translated into the following code that is uploaded to AWS Lambda.

// index.js
module.exports = WrapAwsEventAndContextAndCallBackAndForwardTo((() => {
 const app = express();

 // GET method route
 app.get('/', function (req, res) {
 res.send('GET request to the homepage')
 })

 // POST method route
 app.post('/', function (req, res) {
 res.send('POST request to the homepage')
 })

 return app;
})())

In other words, that factory function will get called, returning the expected requestListener function. Pulumi will then inject the call to the helper that will wrap that function with the right translation code to map from the AWS specific inputs to the expected Node.js form, and from the Node.js outputs to the AWS expected forms.

Right now, this functionality has been enabled for AWS and Azure in our @pulumi/cloud-aws and @pulumi/cloud-azure packages (support for GCP is on the roadmap!). This means that the above code will run properly on either platform without consumers having to write platform specific code or figure out how they would need to translate between the two. And, because of the translation from AWS/Azure specific APIs to the common Node.js API, you can now use virtually any middleware components effectively on either platform. Currently, this only works if you are writing a Pulumi application. However, we are tracking the work to extract the specialized code we’ve written into its own library. Once that work is done, you’ll then be able to easily do something like create an Azure FunctionApp manually, setup the appropriate bindings, and then upload your code which will setup the appropriate Azure endpoint, but then call into this library to translate to the Node.js http form that will allow you to then code things up as simply as if this was just an Express.js app!

^**^ One thing glossed over was how Pulumi knows how to convert a JavaScript callback into an AWS Lambda or an Azure FunctionApp. Stay tuned for more on this subject! It’s an incredibly cool part of Pulumi, and we can’t wait to dive deeper into that functionality and how it can make it simple and clear to create an entire Cloud App in one single package, even including the code in-line that will end up executing in the cloud at runtime! That magic, along with powerful components like the new HttpServer API can help make cloud applications dramatically simpler to write and maintain. Happy coding!

You can dig in to serverless coding with Pulumi here, and join us on Wednesday 3rd October at 11am PDT to hear more about serverless programming with Pulumi on our YouTube live stream.

Program the Cloud with 12 Pulumi Pearls

Joe Duffy — Wed, 25 Jul 2018 00:00:00 +0000

In this post, we’ll look at 12 “pearls” – bite-sized code snippets – that demonstrate some fun ways you can program the cloud using Pulumi. In my introductory post, I mentioned a few of my “favorite things”. Now let’s dive into a few specifics, from multi-cloud to cloud-specific, spanning containers, serverless, and infrastructure, and generally highlighting why using familiar languages is so empowering for cloud scenarios. Since Pulumi lets you do infrastructure-as-code from the lowest-level to the highest, we will cover a lot of interesting ground in short order.

If you want to follow along and try some of this out, Pulumi is open source on GitHub, free to download and use, and the quickstart will acquaint you with the CLI. Most of the examples are directly runnable and available in our examples repo, and are just a pulumi up away, unlike other approaches that require you to point-and-click around in your cloud’s console, and/or author reams of yucky YAML. And you get to use familiar languages!

Here is an index of the pearls in case you want to dive straight into one in particular:

Infrastructure:

Declare cloud infra using a familiar language (with loops!)
Make a reusable component out of your cloud infra

Serverless:

Go serverless without the YAML
Capture state in your serverless funcs, like real lambdas
Simple serverless cron jobs
Run Express-like serverless SPAs and REST APIs at near zero cost

Containers:

Deploy production containers without the fuss
Use containers without Dockerfiles
Invoke a long-running container as a task

General Tips and Tricks:

Use code to avoid hard-coding config
Use config to enable multi-instantiation and code reuse
Give your components runtime APIs

Even if you’re uninterested in low-level infrastructure, it can be fun to work through these examples; it’s “turtles all the way down” with Pulumi and doing so can help understand how the system works. And similarly, it can be fun to see the high-level scenarios these building blocks facilitate, even if you just want to stand up containers and functions.

And with that, let’s dive in.

Infrastructure

Familiar Languages

[ Runnable Example ]

Pulumi gives you a way to express infrastructure configuration using your favorite programming language. In this article, we’ll use TypeScript on Node.js, as it delivers great productivity by blending dynamic and static typing with the NPM ecosystem of reusable packages.

At the lowest layer, there are packages for all major cloud providers – AWS, Azure, Google Cloud – in addition to Kubernetes and OpenStack. So, if you want an EC2 instance, Azure CosmosDB, or Kubernetes Pod, you just new up an object. From there, Pulumi uses an infrastructure-as-code approach similar to using AWS CloudFormation, Azure Resource Manager, Google Resource Manager, Kubernetes or Helm YAML, or Terraform – just without the YAML DSLs.

The strength of using a language truly begins to shine when you go beyond new, however. Using a language gives you control structures, like for loops and if branches. This is easy to take for granted, but compared to existing YAML DSLs, it is very powerful.

For example, let’s create three EC2 VMs and export their IP addresses. Normally you’d need to copy-and-paste, however for loops to the rescue!

index.ts:

import * as aws from "@pulumi/aws";

let webSg = new aws.ec2.SecurityGroup("web-secgrp", {
 ingress: [
 { protocol: "tcp", fromPort: 80, toPort: 80, cidrBlocks: ["0.0.0.0/0"] },
 ],
});

let webServers = [];
for (let i = 0; i < 3; i++) {
 webServers.push(new aws.ec2.Instance(`web-server-${i}`, {
 instanceType: "t2.micro",
 ami: "ami-6869aa05", // us-east-1-only
 securityGroups: [ webSg.name ],
 userData: `#!/bin/bashn` +
 `echo 'Hello, World #${i}!' > index.htmln` +
 `nohup python -m SimpleHTTPServer 80 &n`,
 }));
}
export const publicHostnames = webServers.map(s => s.publicDns);

This provisions three web servers, each of which runs a simple Python webserver.

Of course, this can unlock far more sophisticated scenarios. For example, our own AWS Infrastructure package offers core AWS infrastructure components using AWS’s own best practices, and a community member built an AWS Virtual Private Cloud (VPC) component that automatically distributes subnets across all availability zones in a region, including smartly calculating the CIDR addresses, hiding lots of the messy details of AWS networking.

Reusable Components

[ Runnable Example ]

Speaking of components, this is another benefit of using familiar languages: abstraction. This includes the ability to hide uninteresting details through encapsulation, and create bigger things out of smaller things, a key to how we’ve become so productive with programming languages generally over the years. Applied to infrastructure, this enables some powerful scenarios.

For instance, perhaps you want a standard base class for all servers in your organization, enforcing security policy or even just naming conventions. Or maybe you want to ensure there is an Envoy proxy sidecar for all of your Kubernetes pods. No matter the scenario, we call these things “components” and Pulumi has a rich model that supports parents and children.

To stick with the running example, let’s make a WebServer class that internally creates the EC2 VM so that callers don’t need to know messy details like how to create and configure a SecurityGroup and choosing the right image:

web.ts:

import * as aws from "@pulumi/aws";

let webSg = new aws.ec2.SecurityGroup("web-secgrp", {
 ingress: [
 { protocol: "tcp", fromPort: 80, toPort: 80, cidrBlocks: ["0.0.0.0/0"] },
 ],
});

export class WebServer {
 public readonly vm: aws.ec2.Instance;

 constructor(name: string) {
 this.vm = new aws.ec2.Instance(`${name}`, {
 instanceType: "t2.micro",
 ami: "ami-6869aa05", // us-east-1-only
 securityGroups: [ webSg.name ],
 userData: `#!/bin/bashn` +
 `echo 'Hello from ${name}!' > index.htmln` +
 `nohup python -m SimpleHTTPServer 80 &n`,
 });
 }
}

Now we can go back to our original program and tidy it up a bit:

index.ts:

import { WebServer } from "./web";

let webServers = [];
for (let i = 0; i < 3; i++) {
 webServers.push(new WebServer(`web-server-${i}`));
}
export const publicHostnames = webServers.map(s => s.vm.publicDns);

Not only is the result simpler, but we now have a way to divvy up responsibilities within a team, helping Dev and DevOps to work better together.

This example used purely local components, but you can of course publish and consume any old packages from your favorite package manager – including private registries for your own team.

We will return to these examples later on and improve them to be more reusable through configuration, however let’s first tour some fun serverless and container pearls.

Serverless

No YAML

[ Runnable Example ]

Let’s shift gears and jump into serverless. Pulumi is unique in that its goal is to be a platform for all aspects of modern cloud programming, so that includes serverless, in addition to the abovementioned infrastructure examples and the soon-to-be-mentioned container ones. Most real-world examples we see in fact mix all of these things together in the same program.

Because of the way it uses familiar languages, there are great benefits to serverless programming in Pulumi. No YAML required – just write lambdas in your favorite language, and our compiler/runtime magic will deal with them. But first, let’s start simple.

The AWS Serverless package provides event sources for all of the possible AWS Lambda triggers. This leverages the aws.serverless.Function abstraction that takes a lambda in your language and translates it into a package suitable for uploading to AWS. No need to embed code in YAML files, manually upload to S3, etc. Pulumi handles all of this including versioning.

Let’s look at a simple example that simply posts a Slack message for every new message that arrives in an SQS queue (a new capability recently added to AWS). The way you express serverless functions in Pulumi is by writing ordinary lambdas and the way you hook them up to event sources is by using an event-driven style on your existing cloud resources:

index.ts:

import * as aws from "@pulumi/aws";
import * as serverless from "@pulumi/aws-serverless";

let queue = new aws.sqs.Queue("mySlackQueue");
serverless.queue.subscribe("mySlackPoster", queue, async (e) => {

let slack = require("@slack/client");
let client = new slack.WebClient(config.slackToken);
 for (let rec of e.Records) {
 await client.chat.postMessage({
 channel: config.slackChannel,
 text: `*SQS message ${rec.messageId}*:n${rec.body}n`+
 `(with :love_letter: from Pulumi)`,
 as_user: true,
 });
 console.log(`Posted SQS message ${rec.messageId} to ${config.slackChannel}`);
 }
});
export const queueURL = queue.id;

The code and instructions for running this example are available in our examples repo.

Serverless programs written in this style feel just like your favorite event-driven frameworks, and – because these are just ordinary lambdas in your language of choice – they compose in all the usual ways, unlocking some fascinating future possibilities (like serverless Spark and/or RxJS). Behind the scenes, Pulumi is doing the hard work to extract out the body of the function, rewrite it minimally but as needed to wire it up to AWS Lambda, and package it up. Subsequent deployments are as easy as the first one, as Pulumi does diffing to compute the minimal deltas.

Not everybody will want to mix infrastructure definitions and runtime code like this. Because these are just ordinary programs, you are free to split code into submodules and structure your workspace as you see fit. In fact, you can create distinct stacks for the different pieces.

This example is AWS-specific, however the Pulumi Cloud Framework offers several multi-cloud serverless (and container) capabilities. We will see some examples of that package in action below, however being able to target your cloud of choice directly means you can use all of its underlying features.

Real Lambdas

[ Runnable Example ]

Pulumi also lets you capture state in your serverless functions. This is one of the major advantages to this approach; normally you’d need to awkwardly arrange for such state to be passed into your function through some sideband mechanism, such as environment variables.

In Pulumi, we just capture state. Just like in your favorite language.

Let’s say we want the Slack information above to be configurable – a good idea, since the Slack token is a secret that should be configured using the –secret flag. To do so, first let’s create a config module:

config.ts:

import * as pulumi from "@pulumi/pulumi";
let config = new pulumi.Config(pulumi.getProject());

export const slackChannel = config.get("slackChannel") || "#general",
export const slackToken: config.require("slackToken");

And now let’s go back to our above program, and use the config module for these properties. Notice that we simply capture a reference to the config module – that’s it!

index.ts:

import * as config from "./config";
import * as aws from "@pulumi/aws";
import * as serverless from "@pulumi/aws-serverless";

let queue = new aws.sqs.Queue("mySlackQueue");
serverless.queue.subscribe("mySlackPoster", queue, async (e) => {
 let slack = require("@slack/client");
 let client = new slack.WebClient(config.slackToken);

 for (let rec of e.Records) {
 await client.chat.postMessage({
 channel: config.slackChannel,
 text: `*SQS message ${rec.messageId}*:n${rec.body}n`+
 `(with :love_letter: from Pulumi)`,
 as_user: true,
 });
 console.log(`Posted SQS message ${rec.messageId} to ${config.slackChannel}`);
 }
});
export const queueURL = queue.id;

We often call this “capture by serialization” because, of course, the captured state is marshaled into the serverless function’s package, and then unmarshaled at runtime at the time of use. The semantics of capture are specific to each language, but for JavaScript, we do a deep serialization of the object, including its prototype chain, so that the full structure is preserved.

Because of this, functions are even available. We use this capability to let you capture resources by-value that internally have references to a live cloud resource. We will see that next in our serverless cron job example.

Cron Jobs

[ Runnable Example ]

Pulumi’s Cloud Framework has an ultra-straightforward timer module that lets you schedule cron jobs that run serverless functions. This is the easiest way to get up and running with serverless functions, because you don’t even need any other resources to trigger events from.

There are several ways to schedule a timer, depending on your stylistic preferences. These examples simply print the current time to the console on a given interval:

index.ts:

import * as cloud from "@pulumi/cloud";

// Run a timer every minute:
cloud.timer.interval("interval-timer", { minutes: 0 }, () => {
 console.log(`interval-timer: ${Date.now()}`);
});

// Run a timer every minute (cron-style expression):
cloud.timer.cron("cron-timer", "0 * * * * *", () => {
 console.log(`cron-timer: ${Date.now()}`);
});

// Run a timer every day at 7:30 UTC:
cloud.timer.daily("daily-timer", { hourUTC: 7, minuteUTC: 30 }, () => {
 console.log(`daily-timer: ${Date.now()}`);
});

// Run a timer at the 45th minute UTC of every hour:
cloud.timer.hourly("hourly-timer", { minuteUTC: 45 }, () => {
 console.log(`hourly-timer: ${Date.now()}`);
});

To make things slightly more interesting, let’s write a serverless timer that fetches the Hacker News homepage every hour and stashes it into a document database:

index.ts:

import * as cloud from "@pulumi/cloud";

let snapshots = new cloud.Table("snapshots");
cloud.timer.daily("daily-yc-snapshot", { hourUTC: 0, minuteUTC: 0 }, () => {
 let req = require("https").get("https://news.ycombinator.com", (res) => {
 let content = "";

 res.setEncoding("utf8");
 res.on("data", (chunk) => { content += chunk });
 res.on("end", () => {
 snapshots.insert({ date: Date.now(), content: content });
 });
 });
 req.end();
});

We are using the ability to capture a reference to our cloud.Table and call functions on it, like insert. Of course, the body of that function can do absolutely anything that you’d like.

REST APIs

[ Runnable Example ]

Us Node.js programmers love web frameworks, whether that be serving static content using a single page application (SPA) built using React.js, Vue.js, or Angular.js, a program built using dynamic server-side logic using Express.js, or some combination thereof. Serverless offers a way to run these programs at incredible density and low cost, often reducing a VM- or container-based solution to 1/10th its cost (or better). The problem is, this usually means abandoning productive frameworks, and resorting to YAML, hand-authoring Swagger files, and configuring complex API Gateway machinery – all foreign concepts when you just want to run a website.

Pulumi lets you just write code to host both static and dynamic content simply, using its cross-cloud framework’s cloud.API abstraction. It has been designed to have a familiar Node.js feel. The following example serves all static content underneath my www directory at the root, and provides a simple GET endpoint at /hello that returns a { hello: "World!" } JSON object:

index.ts:

import * as cloud from "@pulumi/cloud";
// Create a new API endpoint:
let app = new cloud.API("my-app");
// Serve static files from the `www` folder (using AWS S3):
app.static("/", "www");
// Serve a simple REST API on `GET /hello` (using AWS Lambda):
app.get("/hello", (req, res) => res.json({ hello: "World!" }));
// Export the public URL for the HTTP service:
export const url = app.publish().url;

All of the usual verbs are available – post, patch, etc. – as is support for middleware.

The structure of my local filesystem will demonstrate how some of this ties together:

.
├── Pulumi.yaml
├── index.ts
├── package.json
└── www
├── favicon.png
└── index.html
1 directory, 5 files

Again, all we need to do is pulumi up, and Pulumi will figure out what has changed. So, if we are building an SPA using React.js, etc., then we would simply have a build step that updates the content, and the next time we run pulumi up, it will figure out what has changed and make the minimal set of updates. Similarly, if we edit a dynamic function, it will update just the function.

As you can imagine, things get fun when you start mixing scenarios. Say we wanted a web frontend to our Hacker News timer fetcher above. Thanks to the way we can simply capture a reference to the snapshots database and use it in the body of our API, this is extremely trivial.

Per the earlier conversation around mixing infrastructure and application code, some folks prefer to split the routes from the application code. This is a fine convention, and easy to do:

index.ts:

import * as cloud from "@pulumi/cloud";
import * as routes from "./app/routes";

// Create an API like before, but use a routes module instead of defining inline:
let app = new cloud.API("my-app");
app.get("/hello", routes.hello);

export const url = app.publish().url;

For now, Pulumi only offers an AWS implementation of these APIs, but more clouds are on their way!

Containers

Easy Deployment

[ Runnable Example ]

Let’s shift gears again and take a look at containers. Pulumi supports programming against raw container orchestrators – Amazon’s ECS, Azure’s ACS, Google Cloud’s GKE, Kubernetes directly, and so on – using the cloud resources directly. In this mode, Pulumi is entirely unopinionated about how containers are built, published, and deployed to your orchestrator.

It is important that Pulumi supports that low-level approach, and is what distinguishes it compared to PaaS-like offerings. This entails significant spelunking at a level that is only loosely connected to the task of standing up a container, however, which can be time-consuming and frustrating for many developers. To make this common case easier, Pulumi also offers higher-level abstractions to make these common cases easier, similar in spirit to Docker Compose, except that you can mix these abstractions with all your other cloud resources.

For example, to create a load balanced Nginx container using an image from the Docker Hub:

index.ts:

import * as cloud from "@pulumi/cloud";

let nginx = new cloud.Service("nginx", {
 image: "nginx",
 ports: [{ port: 80 }],
 replicas: 2,
});

export const url = nginx.defaultEndpoint;

This is all you need to run Nginx with 2 load balanced replicas listening on port 80. By default, this will run in Fargate when targeting AWS, which means you can skip the complications of configuring an orchestrator. After running pulumi up, the auto-assigned URL will be printed.

The image directive supports any normal Docker image URL, so private registries work just as well as the default public Docker Hub. And there are a number of other supported properties for memory and CPU constraints, advanced port mapping, and volumes.

Now things get a bit more interesting. Let’s say we wanted to build our own custom container instead. Normally that would mean defining a Dockerfile, provisioning a private registry, and manually building and publishing your image to the registry, and then consuming that container image from the registry. That’s a lot of boilerplate scripting!

Pulumi unifies all container building, publishing, and consumption underneath a simple pulumi up:

index.ts:

import * as cloud from "@pulumi/cloud";

let nginx = new cloud.Service("nginx", {
 build: ".",
 ports: [{ port: 80 }],
 replicas: 2,
});

export const url = nginx.defaultEndpoint;

Dockerfile:

FROM nginx
COPY ./www /usr/share/nginx/html

This is trivial Dockerfile that derives from the nginx base image and copies the ./www directory into the Nginx HTML target so that it will be served up. This is our directory structure:

.
├── Dockerfile
├── Pulumi.yaml
├── index.ts
├── package.json
└── www
├── favicon.png
└── index.html
1 directory, 5 files

Of course, this Dockerfile could be arbitrarily complex, using any Docker image.

No Dockerfiles

[ Runnable Example ]

The above example showed you how to create services that use Docker containers, either via a pre-built image stored in a registry like the Docker Hub, or by building one from a Dockerfile. The cloud.Service class supports a third possibility, which is really fun, as it takes advantage of Pulumi’s serverless programming model earlier to define a container and its service.

Imagine we want to define a simple Node.js webserver rather than using Nginx like this. Perhaps it uses Express.js. For this blog post, however, let’s stick with the most basic http-module-based webserver:

index.ts:

import * as cloud from "@pulumi/cloud";

let www = new cloud.Service("www", {
 function: () => {
 let rand = Math.random();
 require("http").createServer((req, res) => {
 res.end(`Hello, world! (from ${rand})`);
 }).listen(80);
 },
 ports: [{ port: 80 }]
 memory: 128,
 replicas: 2,
});

export const url = www.defaultEndpoint;

This code defines a load balanced service, backed by a container that Pulumi builds on your behalf out of the body of the function provided in the program, all done with a simple pulumi up gesture. This function, of course, can close over other state just like the earlier serverless examples, including cloud resources, making it as easy as can be to connect your microservices.

Long Running Containers as Tasks

[ Runnable Example ]

The above examples are great if what you want is a load balanced, long-running service. Sometimes a container is useful simply for executing a long-running task, or one that has very specific requirements about its execution environment (such as a specific base image).

For those cases, a cloud.Task is just what the doctor ordered. Task supports all the same construction techniques as Service, so you can give it a pre-built image, a build directive, or an inline function. For purposes of illustration, let’s build a custom Dockerfile that performs an FFMPEG transformation, to extract a thumbnail anytime someone adds a video to a bucket:

index.ts:

let cloud = require("@pulumi/cloud-aws");
// A bucket to store videos and thumbnails.
let videos = new cloud.Bucket("bucket");
// A task which runs an FFMPEG transform to extract a thumbnail image.
let ffmpegThumbnailTask = new cloud.Task("ffmpegThumbTask", {
 build: ".",
 memoryReservation: 512,
});

// When a new video is uploaded, run the FFMPEG task on the video file.
videos.onPut(
 "onNewVideo",
 args => {
 let file = args.key;
 ffmpegThumbnailTask.run(
 {
 environment: {
 "S3_BUCKET": videos.bucket.name.get(),
 "INPUT_VIDEO": file,
 "TIME_OFFSET": file.substring(file.indexOf('_')+1, file.indexOf('.')).replace('-',':'),
 "OUTPUT_FILE": file.substring(0, file.indexOf('_')) + '.jpg',
 },
 });
 },
 { keySuffix: ".mp4" });

exports.bucketName = videos.bucket.name;

Dockerfile:

FROM jrottenberg/ffmpeg
RUN apt-get update && apt-get install python-dev python-pip -y && apt-get clean
RUN pip install awscli
WORKDIR /tmp/workdir
ENTRYPOINT
echo "Starting ffmpeg task... " &&
echo "Copying video from s3://${S3_BUCKET}/${INPUT_VIDEO} to ${INPUT_VIDEO}..." &&
aws s3 cp s3://${S3_BUCKET}/${INPUT_VIDEO} ./${INPUT_VIDEO} &&
ffmpeg -v error -i ./${INPUT_VIDEO} -ss ${TIME_OFFSET} -vframes 1 -f image2 -an -y ${OUTPUT_FILE} &&
echo "Copying thumbnail to S3://${S3_BUCKET}/${OUTPUT_FILE} ..." &&
aws s3 cp ./${OUTPUT_FILE} s3://${S3_BUCKET}/${OUTPUT_FILE}

Notice that, just like before, this is backed by an ordinary Dockerfile. This example actually showcases serverless and containers living alongside one another in harmony. Notice how we kick off the task – simply by capturing a reference to the Task from the serverless event handler, and invoking its run method. This leverages Pulumi’s ability to give resources like Tasks custom APIs that can be invoked at runtime, something we will return to again later on.

General Tips and Tricks

Avoid Hard-Coding Config

Let’s return to our webserver example from earlier. Unfortunately, this code will only work in one region, because we hard-coded the "ami-6869aa05" image name (AMI). (Image names in AWS are region-specific.) In general, we prefer to build Pulumi programs that can be multi-instantiated across different regions. Pulumi’s stacks model streamlines this workflow, making it easy to spin up fresh copies of your entire application and/or infrastructure when you need to scale, while also reducing friction during development and testing. Such hard-coding is a definite code smell and impediment, however.

Because we are using a familiar language, we can do anything that general-purpose languages can do. That includes dynamically querying sources of information which, in this case, can be used to avoid hard-coding. We can even use the AWS SDK, which gives us full access to all AWS functionality:

ami.ts:

import * as aws from "@pulumi/aws";

export function getLinuxAmi(): Promise<string> {
 return new Promise<string>((resolve, reject) => {
 let ec2 = new (require("aws-sdk")).EC2({ region: aws.config.requireRegion() });
 ec2.describeImages(
 {
 Owners: [ "amazon" ],
 Filters: [{
 Name: "name",
 Values: [ "amzn-ami-hvm-????.??.?.x86_64-gp2" ],
 }],
 },
 (err: any, data: any) => {
 if (err) {
 reject(err);
 } else {
 let newestImage: string | undefined;
 let newestImageDate: number | undefined;
 for (let image of data.Images) {
 let imageDate = Date.parse(image.CreationDate);
 if (!newestImage || !newestImageDate ||
 imageDate > newestImageDate) {
 newestImage = image.ImageId;
 newestImageDate = imageDate;
 }
 }
 if (newestImage) {
 console.log(`AMI: ${newestImage} (${newestImageDate})`);
 resolve(newestImage);
 } else {
 reject("No Linux AMI found for the current region");
 }
 }
 },
 );
 });
}

I’ll be the first to admit that this is code that only a mother could love. Even better would be to wrap it up in a reusable NPM package that others can use without the boilerplate! Either way, it’s very powerful to have the full capabilities of a language at our fingertips, as this sort of thing simply isn’t possible in other infra-as-code solutions.

Now that we have this function, we can revisit the way we set the AMI property on our VMs:

web.ts:

import * as aws from "@pulumi/aws";
import { getLinuxAmi } from "./ami";

let webSg = new aws.ec2.SecurityGroup("web-secgrp", {
 ingress: [
 { protocol: "tcp", fromPort: 80, toPort: 80, cidrBlocks: ["0.0.0.0/0"] },
 ],
});

export class WebServer {
 public readonly vm: aws.ec2.Instance;

 constructor(name: string) {
 this.vm = new aws.ec2.Instance(`${name}`, {
 instanceType: "t2.micro",
 ami: getLinuxAmi(),
 securityGroups: [ webSg.name ],
 userData: `#!/bin/bashn` +
 `echo 'Hello from ${name}!' > index.htmln` +
 `nohup python -m SimpleHTTPServer 80 &n`,
 });
 }
}

Given this change, we can safely deploy our WebServer to any AWS region.

Multi-Instantiation and Code Reuse

In the webserver example, we used the aws.config.requireRegion() function to obtain the configured AWS region. Using configuration in this manner lets us multi-instantiate stacks; in other words, to easily stand up multiple copies of a service, application, or infrastructure.

We can even make the quantity configurable using Pulumi’s config system:

index.ts:

import { WebServer } from "./web";
import { Config, getProject } from "@pulumi/pulumi";

let config = new Config(getProject());
let webServers = [];
let webServersCount = config.getNumber("count") || 3;
for (let i = 0; i < webServersCount; i++) {
 webServers.push(new WebServer(`web-server-${i}`));
}

export const publicHostnames = webServers.map(s => s.vm.publicDns);

Given this specific program, we can easily change the number of servers created from the default of three, to something larger:

$ pulumi config set count 7

There are plenty of opportunities for configuration like this, like the machine size. By making such settings configurable, different environments can easily have different settings, while still sharing the same code beneath them, eliminating copy-and-paste and its associated pitfalls.

Give Components Runtime APIs

[ Runnable Example ]

As our final example, let’s see how to give your own components APIs that can be invoked at runtime. We’ve already seen this a few times now, such as managing data in a cloud.Table that has been captured in a lambda, and cloud.Task’s run method. Pulumi is built to be extensible, so you can define your own abstractions just like these with APIs too.

For this example, let’s take a simple Redis cache service and turn it into a component. After doing so, we can then add simple get/set APIs on it that let you interact with the cache at runtime in simple ways. By providing this sort of abstraction, we open up interesting possibilities. For instance, our default implementation can use a custom container-backed service, via cloud.Service, but opt to use a simpler hosted solution when running in a public cloud – AWS ElastiCache, Azure Redis Cache, or Memorystore in Google Cloud.

To make our component, we will simply allocate a cloud.Service in the constructor, taking an optional memory parameter to control its memory reservation. We are using configuration to read the Redis password using Pulumi’s built-in secrets mechanism. To give that component APIs, we just define methods on the class that access Redis through its own NPM package, and that’s it! The way lambda capture works will ensure that these functions are accessible:

cache.ts:

import * as cloud from "@pulumi/cloud";
import * as config from "./config";

// A simple cache abstraction that wraps Redis.
export class Cache {
 public readonly get: (key: string) => Promise<string>;
 public readonly set: (key: string, value: string) => Promise<void>;
 private readonly redis: cloud.Service;

 constructor(name: string, memory: number = 128) {
 this.redis = new cloud.Service(name, {
 image: "redis:alpine",
 memory: memory,
 ports: [{ port: 6379 }],
 command: ["redis-server", "--requirepass", config.redisPassword],
 });
 }

 public async get(key: string): Promise<string> {
 let client = await this.createRedisClient();
 return new Promise<string>((resolve, reject) => {
 client.get(key, (err: any, v: any) => {
 if (err) reject(err);
 else resolve(v);
 });
 });
 }

 public async set(key: string, value: string): Promise<void> {
 let client = await this.createRedisClient();
 return new Promise<void>((resolve, reject) => {
 client.set(key, value, (err: any, v: any) => {
 if (err) reject(err);
 else resolve();
 });
 });
 }

 private async createRedisClient(): Promise<any> {
 let ep = (await this.redis.defaultEndpoint).get();
 return require("redis").createClient(
 ep.port, ep.hostname, { password: config.redisPassword });
 }
 }
}

Because this is just a class, we have all the standard programming facilities available to us. If we wanted to specialize it to use our cloud’s hosted Redis services, we could do that using the standard techniques, including conditional code or, slightly more elegantly, using concrete subclasses for the different options.

Now that we’ve defined our cache, we can create an instance, and then use it from within a callback, such as a serverless API implementation. This is a fairly complex application that acts as a URL shortener, with both GET and POST endpoints, but really shows off the power of combining many of the techniques we’ve seen throughout this post:

index.ts:

import * as cloud from "@pulumi/cloud";
import * as cache from "./cache";

// Create a URL shortener app and a table to hold shortened URLs.
let app = new cloud.API("shortener");
let urlTable = new cloud.Table("urls", "name");

// Use our cache component to cache frequently accessed URLs.
let urlCache = new cache.Cache("urlcache");

// GET /url lists all URLs currently registered.
endpoint.get("/url", async (req, res) => {
 try {
 res.status(200).json(await urlTable.scan());
 } catch (err) {
 res.status(500).json(err.stack);
 }
});

// GET /url/{name} redirects to the target URL based on a short-name.
endpoint.get("/url/{name}", async (req, res) => {
 let name = req.params["name"];
 try {
 // First try the Redis cache. If we miss, consult the table.
 let url = await urlCache.get(name);
 if (url) {
 res.setHeader("X-Powered-By", "redis");
 } else {
 let value = await urlTable.get({ name });
 if (value && value.url) {
 url = value.url;
 urlCache.set(name, url); // cache for next time.
 }
 }

 // If we found an entry, 301 redirect to it; else, 404.
 if (url) {
 res.setHeader("Location", url);
 res.status(302);
 res.end("");
 } else {
 res.status(404);
 res.end("");
 }
 } catch (err) {
 res.status(500).json(err.stack);
 }
});

// POST /url registers a new URL with a given short-name.
endpoint.post("/url", async (req, res) => {
 let url = req.query["url"];
 let name = req.query["name"];

 try {
 // Insert into the table and our URL cache.
 await urlTable.insert({ name, url });
 await urlCache.set(name, url);
 res.json({ shortenedURLName: name });
 } catch (err) {
 res.status(500).json(err.stack);
 }
});

export const url = endpoint.publish().url;

Notice that we can capture and use our Cache component just like we can the cloud.Table!

We’ve only scratched the surface of what’s possible with components and custom APIs, however hopefully you’re starting to get a sense of why this is such a powerful capability.

Winding Down

In this post, we’ve seen how Pulumi can define infrastructure, containers, and serverless cloud software, often combining many of the techniques together to build powerful distributed applications. This includes programming to specific resources that your cloud of choice provides, in addition to writing higher level cloud programs that are able to target many clouds (expect more multi-cloud progress here in the weeks to come – including Kubernetes). Each pearl is a topic unto itself; we’d love your feedback on what topics to dig into next.

Happy hacking – and have fun programming the cloud!

Managing GitHub Webhooks with Pulumi

Matt Ellis — Fri, 13 Jul 2018 00:00:00 +0000

At Pulumi, we do all of our development on GitHub, with a workflow built around topic branches. When a developer wants to make a change, they push a branch to GitHub, open a pull request and (in theory) once it’s merged, delete the branch. In practice, we’ll often forget to delete the topic branch (I’m probably the worst offender), which means we end up having topic branches linger on our main repository until they are explicitly cleaned up. While it’s a lot of fun to go a click through the GitHub UI from time to time, deleting merged branches, it’s even more fun to build automation to do this for us. Since GitHub has a rich set of webhooks and Pulumi makes it easy to write serverless functions, it felt like it would be natural to use Pulumi to write a hook that would clean up branches after a pull request got merged. In addition, Pulumi lets us leverage real programming languages to build abstractions, which means we can build a simple framework that hides much of the ceremony behind defining a hook and lets us focus on the core logic of our hook, without worrying about how it is deployed and managed.

A simple hook

When you register a webhook with GitHub, you provide an HTTP endpoint and a set of events you’d like to listen for. When one of the events happens on GitHub, it makes a POST request to your HTTP endpoint with a payload that describes the event. We’ll start by creating a simple endpoint that just logs when an event happens. We’ll start by using the aws-serverless package to define an API that just calls console.log to write the X-GitHub-Event header that is sent with the request:

import * as serverless from "@pulumi/aws-serverless";

const api = new serverless.apigateway.API("hook", {
 routes: [{
 path: "/",
 method: "POST",
 handler: async (req, ctx) => {
 console.log(`Got event ${req.headers['X-GitHub-Event']}`);
 return {
 statusCode: 200,
 body: ""
 }
 }
 },],
});

export const url = api.url;

After running pulumi update, we’ll have a HTTP endpoint we can give to GitHub:

Updating stack 'github-hook-blog-dev'
Performing changes:
Type Name Status Info
+ pulumi:pulumi:Stack github-hook-blog-github-hook-blog-dev created
+ ├─ aws-serverless:apigateway:API hook created
+ │ ├─ aws:apigateway:RestApi hook created
+ │ ├─ aws:apigateway:Deployment hook created
+ │ ├─ aws:lambda:Permission hook-980655da created
+ │ └─ aws:apigateway:Stage hook created
+ └─ aws:serverless:Function hook980655da created
+ ├─ aws:iam:Role hook980655da created
+ ├─ aws:iam:RolePolicyAttachment hook980655da-32be53a2 created
+ └─ aws:lambda:Function hook980655da created
---outputs:---
url: "https://t1vyz1x203.execute-api.us-west-1.amazonaws.com/stage/"

Now that we have our Webhook deployed, we’ll add it to GitHub. In this case, I have a little throwaway GitHub repository I use for testing this sort of stuff. In that repository, I go to Settings -> Webhooks -> Add webhook and fill in my information:

Payload URL: The value of the url output of my Pulumi program (in this case it is https://t1vyz1x203.execute-api.us-west-1.amazonaws.com/stage/).
Content Type: application-json. I know we’ll be inspecting this content as we develop the hook and since we’re writing the implementation in TypeScript, it will be easier to interact with JSON encoded data.
Secret: Generate a random string and put it here. I used random.org to do so, but any random string will suffice. We’ll use this string to validate that the request came from GitHub.
Events: I only care about pull_request events, so I picked “Let me select individual events” and then checked “Pull requests” and unchecked the other event types.
Active: Since we want GitHub to deliver events, we’ll keep this checked.

Once that’s done, we can test our hook by opening a pull request, waiting a few moments and then use pulumi logs to ensure our hook was triggered:

$ pulumi logs
Collecting logs for stack github-hook-blog-dev since 2018-07-09T09:06:51.000-07:00.
2018-07-09T10:06:00.610-07:00[ hook980655da-d0462d4] START RequestId: 5aab2d71-839a-11e8-a7ae-93a25d68b5a9 Version: $LATEST
2018-07-09T10:06:00.612-07:00[ hook980655da-d0462d4] 2018-07-09T17:06:00.611Z 5aab2d71-839a-11e8-a7ae-93a25d68b5a9 Got event ping
2018-07-09T10:06:00.630-07:00[ hook980655da-d0462d4] END RequestId: 5aab2d71-839a-11e8-a7ae-93a25d68b5a9
2018-07-09T10:06:00.630-07:00[ hook980655da-d0462d4] REPORT RequestId: 5aab2d71-839a-11e8-a7ae-93a25d68b5a9 Duration: 10.99 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 20 MB
2018-07-09T10:06:23.241-07:00[ hook980655da-d0462d4] START RequestId: 68445127-839a-11e8-80a6-759a29eb6005 Version: $LATEST
2018-07-09T10:06:23.243-07:00[ hook980655da-d0462d4] 2018-07-09T17:06:23.243Z 68445127-839a-11e8-80a6-759a29eb6005 Got event pull_request
2018-07-09T10:06:23.250-07:00[ hook980655da-d0462d4] END RequestId: 68445127-839a-11e8-80a6-759a29eb6005
2018-07-09T10:06:23.250-07:00[ hook980655da-d0462d4] REPORT RequestId: 68445127-839a-11e8-80a6-759a29eb6005 Duration: 1.32 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 20 MB

Here we can see our hook got two events, the first is the ping event, which GitHub sends to verify the hook. The second event is from when I opened a pull request in the repository.

Before we go any further, let’s start using the secret we set to validate the Webhook payload. When GitHub sends an event to a Webhook, it sets a special header X-Hub-Signature which is an HMAC digest of the request body. The value we placed in the Secret field when configuring the Webhook is the key. So I’ll take that secret value, set it as a configuration value in my project:

$ pulumi config set --secret hookSecret [READACTED]

Once that’s done, I can use this value to validate the signature. We’ll use some functions from node’s crypto library at runtime to handle this. If the signature we computed doesn’t match what was provided with the request, we’ll return error code 400. Our program now looks like this:

import * as pulumi from "@pulumi/pulumi";
import * as serverless from "@pulumi/aws-serverless";

const cfg = new pulumi.Config(pulumi.getProject());
const hookSecret = cfg.require("hookSecret");

const api = new serverless.apigateway.API("hook", {
 routes: [{
 path: "/",
 method: "POST",
 handler: async (req, ctx) => {
 // Compute the HMAC of the body, using `hookSecret` as the key and `sha1` as the algorithm

 // First, grab the body. It may be base64 encoded, depending on if `req.isBase64Encoded` is set. If so, we
 // should decode it.
 let body = req.body;
 if (req.isBase64Encoded) {
 body = Buffer.from(body, 'base64').toString();
 }

 // Now compute the HMAC.
 const crypto = await import("crypto");
 const hmac = crypto.createHmac("sha1", hookSecret);
 hmac.update(body);

 const computedSignature = `sha1=${hmac.digest("hex")}`;

 // Compare the signature that came in with the request to what we computed.
 if (req.headers['X-Hub-Signature'] === undefined ||
 !crypto.timingSafeEqual(Buffer.from(req.headers['X-Hub-Signature']), Buffer.from(computedSignature)))
 {
 console.log(`error: bad signature ${req.headers['X-Hub-Signature']} !== ${computedSignature}`);
 return {
 statusCode: 400,
 body: "bad signature"
 }
 }

 console.log(`Got event ${req.headers['X-GitHub-Event']} and signature ${computedSignature} matched`);
 return {
 statusCode: 200,
 body: ""
 }
 }
 },],
});

export const url = api.url;

After deploying, we can trigger another event by closing an existing pull request or opening a new one, and then use pulumi logs to confirm that our validation is working as intended:

$ pulumi logs
Collecting logs for stack github-hook-blog-dev since 2018-07-09T12:43:05.000-07:00.
2018-07-09T13:43:00.630-07:00[ hook980655da-d0462d4] START RequestId: ab363ec7-83b8-11e8-a7ae-93a25d68b5a9 Version: $LATEST
2018-07-09T13:43:00.693-07:00[ hook980655da-d0462d4] 2018-07-09T20:43:00.693Z ab363ec7-83b8-11e8-a7ae-93a25d68b5a9 Got event pull_request and signature sha1=b83320205edc7cb9aeea1a845b85de0c83092e1d matched
2018-07-09T13:43:00.731-07:00[ hook980655da-d0462d4] END RequestId: ab363ec7-83b8-11e8-a7ae-93a25d68b5a9
2018-07-09T13:43:00.731-07:00[ hook980655da-d0462d4] REPORT RequestId: ab363ec7-83b8-11e8-a7ae-93a25d68b5a9 Duration: 128.75 ms Billed Duration: 200 ms Memory Size: 128 MB Max Memory Used: 20 MB

Building an abstraction

We now have a nice little skeleton that we can use when writing a webhook on GitHub. Let’s start to leverage some other Pulumi features to make things simpler and provide a nicer interface for writing hooks. Let’s start by abstracting away the code that handles validating the request and returning a response behind a helper method. We’ll pass a function to this helper which will be called after the validation has completed. We’ll also parse the body into a JSON object that our handler can use.

import * as pulumi from "@pulumi/pulumi";
import * as serverless from "@pulumi/aws-serverless";

const cfg = new pulumi.Config(pulumi.getProject());
const hookSecret = cfg.require("hookSecret");

function createWebhook(name: string, handler: ((req: serverless.apigateway.Request, body: any) => Promise<void>)) {
 return new serverless.apigateway.API(name, {
 routes: [{
 path: "/",
 method: "POST",
 handler: async (req, ctx) => {
 // Compute the HMAC of the body, using `hookSecret` as the key and `sha1` as the algorithm

 // First, grab the body. It may be base64 encoded, depending on if `req.isBase64Encoded` is set. If so, we
 // should decode it.
 let body = req.body;
 if (req.isBase64Encoded) {
 body = Buffer.from(body, 'base64').toString();
 }

 // Now compute the HMAC.
 const crypto = await import("crypto");
 const hmac = crypto.createHmac("sha1", hookSecret);
 hmac.update(body);

 const computedSignature = `sha1=${hmac.digest("hex")}`;

 // Compare the signature that came in with the request to what we computed.
 if (req.headers['X-Hub-Signature'] === undefined ||
 !crypto.timingSafeEqual(Buffer.from(req.headers['X-Hub-Signature']), Buffer.from(computedSignature)))
 {
 console.log(`error: bad signature ${req.headers['X-Hub-Signature']} !== ${computedSignature}`);
 return {
 statusCode: 400,
 body: "bad signature"
 }
 }

 // Call the handler after parsing the body as JSON
 await handler(req, JSON.parse(body));

 return {
 statusCode: 200,
 body: ""
 }
 }
 },],
 });
}

const api = createWebhook("hook", async (req, body) => {
 console.log(`Got event ${req.headers['X-GitHub-Event']} with action ${body.action}`);
});

export const url = api.url;

The shape of the JSON object for this event is documented on GitHub. Here we’re pulling out the action field, which tells us something about the state of the pull request. For our bot, we only care about PRs where the action is closed and the merged key is set to true. When this holds, we’ll want to delete the branch the pull request came from, if it was a topic branch (instead of a PR opened from a fork). We’ll leverage the existing @octokit/rest NPM package to do this. We can generate a personal access token, and give it repo level scope, so it can act as us and delete branches. We’ll then add this key to our configuration:

$ pulumi config set --secret githubToken [REDACTED]

And we can now update our handler to use this API:

import * as pulumi from "@pulumi/pulumi";
import * as serverless from "@pulumi/aws-serverless";

const cfg = new pulumi.Config(pulumi.getProject());
const hookSecret = cfg.require("hookSecret");

function createWebhook(name: string, handler: ((req: serverless.apigateway.Request, body: any) => Promise<void>)) {
 /* Same as the previous version */
}

const githubToken = cfg.require("githubToken");

const api = createWebhook("hook", async (req, body) => {
 if (body.action === "closed" && body.pull_request.merged &&
 body.pull_request.base.user.login === body.pull_request.head.user.login &&
 body.pull_request.base.repo.name === body.pull_request.head.repo.name)
 {
 const octokit = require('@octokit/rest')()
 octokit.authenticate({
 type: 'token',
 token: githubToken
 });

 console.log(`Deleting reference heads/${body.pull_request.head.ref}`);

 await octokit.gitdata.deleteReference({
 owner: body.pull_request.head.user.login,
 repo: body.pull_request.head.repo.name,
 ref: `heads/${body.pull_request.head.ref}`
 });
 }
});

export const url = api.url;

After pulumi updating, try merging a pull request! Shortly after you click the merge button, the source branch should be deleted automatically. You can also see our log message in the output of pulumi logs:

$ pulumi logs
[matell@matell throwaway]$ pulumi logs
Collecting logs for stack github-hook-blog-dev since 2018-07-09T13:30:26.000-07:00.
2018-07-09T14:28:52.254-07:00[ hook980655da-d0462d4] START RequestId: 1368afec-83bf-11e8-bf0d-cf623578c325 Version: $LATEST
2018-07-09T14:28:52.431-07:00[ hook980655da-d0462d4] 2018-07-09T21:28:52.371Z 1368afec-83bf-11e8-bf0d-cf623578c325 Deleting reference heads/test-branch
2018-07-09T14:28:53.478-07:00[ hook980655da-d0462d4] END RequestId: 1368afec-83bf-11e8-bf0d-cf623578c325
2018-07-09T14:28:53.478-07:00[ hook980655da-d0462d4] REPORT RequestId: 1368afec-83bf-11e8-bf0d-cf623578c325 Duration: 1221.97 ms Billed Duration: 1300 ms Memory Size: 128 MB Max Memory Used: 33 MB

So, in about 75 lines of code, we’ve defined a small abstraction for authoring a GitHub Webhook and we’ve used it to write a little bot that deletes topic branches after pull requests from them have been merged. In itself, that’s pretty cool, but we can do some more cool stuff with Pulumi. As is, while we can create the Webhook using Pulumi, we have to register it using the GitHub console, which is a little tedious. What if we could manage the registration of the hook with Pulumi itself? We’d really like to model the hook’s association with GitHub as a resource that can be created, updated and deleted. Is there a way we can do that? Yes, there is!

Managing all the things with Pulumi

Pulumi uses gRPC to define a contract between resource providers and the rest of Pulumi. So, we could go implement that contract in a language like go, like we do in our Kubernetes provider. Another option would be to take the existing Terraform GitHub Provider and use Pulumi’s terraform bridge wrap it into a Pulumi resource provider, like we do with our pulumi-aws provider. There’s one final option, which is to use the “dynamic provider”, which allows us to implement a resource provider in JavaScript itself. We wrote this provider to help us with testing, so we could mock out resources and control their lifecycle, but we can also use it to create a resource that manages registration of a webhook with GitHub. To do so, we need to implement dynamic.ResourceProvider interface. Once we have the provider, we can also write a resource that uses it.

class GithubWebhookProvider implements dynamic.ResourceProvider {

 // Check ensures that all required properties are set. In this case we have three required parameters.
 check = async (olds: any, news: any) => {
 const failedChecks = [];

 for (const prop of ["url", "owner", "repo"]) {
 if (news[prop] === undefined) {
 failedChecks.push({property: prop, reason: `required property '${prop}' missing`});
 }
 }

 return { inputs: news, failedChecks: failedChecks };
 }

 // Today the engine does the diff between properties to detect if there is a change but this method does
 // tell the engine if the changes between the old and new values require the resource to be "replaced"
 // (that is a new one is created and the old one is deleted) vs being edited in place. For us, if the owner
 // or repo the hook is installed on changes, we'll trigger a replacement.
 diff = async (id: pulumi.ID, olds: any, news: any) => {
 const replaces = [];

 for (const prop of ["owner", "repo"]) {
 if (olds[prop] !== news[prop]) {
 replaces.push(prop);
 }
 }

 return { replaces: replaces };
 }

 // Create actually creates the hook. We use octokit under the hood and return the ID of the hook that was created.
 // Pulumi retains this ID and gives it to us when we need to update or delete the hook.
 create = async (inputs: any) => {
 const octokit = require("@octokit/rest")();
 octokit.authenticate({
 type: "token",
 token: githubToken,
 });

 const res = await octokit.repos.createHook({
 owner: inputs["owner"],
 repo: inputs["repo"],
 name: "web",
 events: ["pull_request"],
 config: {
 content_type: "json",
 url: inputs["url"],
 secret: hookSecret,
 },
 });

 if (res.status !== 201) {
 throw new Error(`bad response: ${JSON.stringify(res)}`);
 }

 // The engine expects that the ID property is a string.
 return {
 id: `${res.data.id}`,
 };
 }

 update = async (id: pulumi.ID, olds: any, news: any) => {
 const octokit = require("@octokit/rest")();
 octokit.authenticate({
 type: "token",
 token: githubToken,
 });

 const res = await octokit.repos.editHook({
 hook_id: id,
 owner: news["owner"],
 repo: news["repo"],
 events: ["pull_request"],
 config: {
 content_type: "json",
 url: news["url"],
 },
 });

 if (res.status !== 200) {
 throw new Error(`bad response: ${JSON.stringify(res)}`);
 }

 return {}
 }

 delete = async (id: pulumi.ID, props: any) => {
 const octokit = require("@octokit/rest")();

 octokit.authenticate({
 type: "token",
 token: githubToken,
 });

 const res = await octokit.repos.deleteHook({
 hook_id: id,
 owner: props["owner"],
 repo: props["repo"],
 });

 if (res.status !== 204) {
 throw new Error(`bad response: ${JSON.stringify(res)}`);
 }
 }
}

interface GitHubWebhookResourceArgs {
 url: pulumi.Input<string>;
 owner: pulumi.Input<string>;
 repo: pulumi.Input<string>;
}

class GitHubWebhookResource extends dynamic.Resource {
 constructor(name: string, args: GitHubWebhookResourceArgs, opts?: pulumi.ResourceOptions) {
 super(new GithubWebhookProvider(), name, args, opts);
 }
}

With this new dynamic resource, registering the hook itself becomes easy:

new GitHubWebhookResource("hook-registration", {
 url: api.url,
 owner: "ellismg",
 repo: "testing",
});

Before running pulumi update go and manually delete the hook registration on GitHub. Now, when you pulumi update you’ll see the new resource created:

Updating stack 'github-hook-blog-dev'
Performing changes:
Type Name Status Info
* pulumi:pulumi:Stack github-hook-blog-github-hook-blog-dev done
+ └─ pulumi-nodejs:dynamic:Resource hook-registration created
---outputs:---
url: "https://t1vyz1x203.execute-api.us-west-1.amazonaws.com/stage/"
info: 1 change performed:
+ 1 resource created
10 resources unchanged
Update duration: 4.649249758s

And if you look on GitHub, the hook should once again be registered! Now, if the URL for our handler ends up changing or we destroy our stack, the hook registration will be updated accordingly. Since our little program is getting large (~200 lines at this point), let’s do a little more work on our abstraction.

Building a Component

Pulumi has the concept of a ComponentResource which is a resource that aggregates other resources. Many resources that we interact with day to day in Pulumi are actually ComponentResources (In fact, the API resource we’re using here is itself a component). In another file, we can create the component resource and move much of our logic into there:

The interesting new code looks like this:

export interface GitHubWebhookArgs {
 owner: pulumi.Input<string>;
 repo: pulumi.Input<string>;
 handler: (req: serverless.apigateway.Request, body: any) => Promise<void>;
}

export class GitHubWebhook extends pulumi.ComponentResource {
 public readonly url: pulumi.Output<string>;

 constructor(name: string, args: GitHubWebhookArgs, opts?: pulumi.ResourceOptions) {
 super("github:webhooks:Hook", name, {}, opts);

 const api = new serverless.apigateway.API("hook", {
 routes: [
 {
 path: "/",
 method: "POST",
 handler: async (req, ctx) => {
 // Compute the HMAC of the body, using `hookSecret` as the key and `sha1` as the algorithm

 // First, grab the body. It may be base64 encoded, depending on if `req.isBase64Encoded` is set. If so, we
 // should decode it.
 let body = req.body;
 if (req.isBase64Encoded) {
 body = Buffer.from(body, 'base64').toString();
 }

 // Now compute the HMAC.
 const crypto = await import("crypto");
 const hmac = crypto.createHmac("sha1", hookSecret);
 hmac.update(body);

 const computedSignature = `sha1=${hmac.digest("hex")}`;

 // Compare the signature that came in with the request to what we computed.
 if (req.headers['X-Hub-Signature'] === undefined ||
 !crypto.timingSafeEqual(Buffer.from(req.headers['X-Hub-Signature']), Buffer.from(computedSignature)))
 {
 console.log(`error: bad signature ${req.headers['X-Hub-Signature']} !== ${computedSignature}`);
 return {
 statusCode: 400,
 body: "bad signature"
 }
 }

 // Call the handler after parsing the body as JSON:
 await args.handler(req, JSON.parse(body));

 return {
 statusCode: 200,
 body: ""
 }
 },
 },
 ],
 }, {
 parent: this,
 });

 new GitHubWebhookResource(`${name}-registration-${args.owner}-${args.repo}`, {
 owner: args.owner,
 repo: args.repo,
 url: api.url,
 }, {
 parent: this,
 });

 this.url = api.url;
 }
}

Here, we’ve transformed our createWebhook method into an actual GitHubWebhook component that manages both the API of the hook as the hook’s registration with GitHub. With this abstraction (and all of this complexity hidden off in a separate github.ts file), the code we focus on when actually writing our bot is quite small:

import * as pulumi from "@pulumi/pulumi";
import { GitHubWebhook } from "./github";

const cfg = new pulumi.Config(pulumi.getProject());
const githubToken = cfg.require("githubToken");

const hook = new GitHubWebhook("hook", {
 owner: "ellismg",
 repo: "testing",
 handler: async (req, body) => {
 if (body.action === "closed" && body.pull_request.merged &&
 body.pull_request.base.user.login === body.pull_request.head.user.login &&
 body.pull_request.base.repo.name === body.pull_request.head.repo.name)
 {
 const octokit = require('@octokit/rest')()
 octokit.authenticate({
 type: 'token',
 token: githubToken
 });

 console.log(`Deleting reference heads/${body.pull_request.head.ref}`);

 await octokit.gitdata.deleteReference({
 owner: body.pull_request.head.user.login,
 repo: body.pull_request.head.repo.name,
 ref: `heads/${body.pull_request.head.ref}`
 });
 }
 },
});

export const url = hook.url;

Going Further

From here, there’s a lot of ways we could extend this code. For example, while we require the configuration value hookSecret to be set before using this abstraction, we could manage this on behalf of the user. To do so, we could use the dynamic provider to create a special “random” resource, this resource would generate a new random string when created, but update calls would not impact the resource. Since Pulumi stores the state of every resource across invocations in the checkpoint, our random resource would not need to be backed by any cloud infrastructure. We could change the shape of the GitHubWebhook component to allow multiple owner/repository pairs to be provided and it would generate a single AWS API and then multiple GitHubWebhookResource’s to register the same hook across multiple repositories. We could even extend this to allow registration of both repository and organization level webhooks. Or instead of hard-coding in the pull_request event as the only event we care about take it as another input property to GitHubWebhook.

One of my favorite things about Pulumi is while it is easy to build these abstractions, it’s also easy to share them. For example, I’ve implemented a few of these ideas to build a framework for our use at Pulumi. You can find the source on GitHub and I’ve published it as an NPM package at @ellismg/pulumi-github-webhooks.

AWS SQS to Lambda with Pulumi: Serverless Queue Example

Cyrus Najmabadi — Tue, 10 Jul 2018 00:00:00 +0000

Two weeks ago Amazon added Simple Queue Service (SQS) as a supported event source for Lambda. SQS is one of AWS’s oldest services, providing access to a powerful message queue that can do things like guarantee messages will be delivered at least once, or messages that will be processed in the same order they were received in. Adding SQS as a supported event source for Lambda means that now it’s possible to use SQS in a serverless computing infrastructure, where Lambdas are triggered in response to messages added to your SQS queue. Now, instead of needing some sort of Service dedicated to polling your SQS queue, or creating Simple Notification Service (SNS) notifications from your messages, you can instead just directly trigger whatever Lambda you want.

Example: Using AWS SQS and Lambda to post messages to Slack

Here’s a simple example of using SQS as an event source with Pulumi: upon receipt of a message via SQS we post a new message into Slack. The full project is available on Github in our examples repo, and includes the instructions to get this up and running.

let aws = require("@pulumi/aws");
let serverless = require("@pulumi/aws-serverless");
let config = require("./config");

let queue = new aws.sqs.Queue("mySlackQueue", { visibilityTimeoutSeconds: 180 });

serverless.queue.subscribe("mySlackPoster", queue, async (e) => {
 let slack = require("@slack/client");
 let client = new slack.WebClient(config.slackToken);
 for (let rec of e.Records) {
 await client.chat.postMessage({
 channel: config.slackChannel,
 text: `*SQS message ${rec.messageId}*: ${rec.body}`+
 `(with :love_letter: from Pulumi)`,
 as_user: true,
 });
 console.log(`Posted SQS message ${rec.messageId} to Slack channel ${config.slackChannel}`);
 }
}, { batchSize: 1 });

module.exports = {
 queueURL: queue.id,
};

Implementing SQS in Pulumi

When we heard about this new functionality, we immediately knew we wanted to let people access this new power from their Pulumi applications. Based on Amazon’s documentation on this new capability we felt it would likely be very easy to provide. Amazon introduced this functionality by reusing existing systems and APIs. Specifically, to create a Lambda trigger off an SQS message you only need to create a new Event Source Mapping mapping between the two. This seemed like it would be very simple, and we set off to expose the right Pulumi API to make this possible.

@pulumi/aws-serverless is Pulumi’s convenience library for creating Serverless computing components. And we added the necessary code to expose the functionality in our simple Pulumi programming model here. With this new API you would now be able to write code in your Pulumi application like so:

import * as aws from "@pulumi/aws";
import * as serverless from "@pulumi/aws-serverless";

// Create the queue. Provide whatever specific configuration you need here.
const sqsQueue = new aws.sqs.Queue("queue", {
 visibilityTimeoutSeconds: 300,
});

// Set up a subscription that will fire whenever the queue receives a message. Here we ask
// for 'batchSize = 1' so we will only process a single message at a time.
serverless.queue.subscribe("subscription", sqsQueue, async (event) => {
 // Add whatever code you want here to run in the AWS lambda. 'event' will contain the
 // all the necessary data about the message added to the queue.
}, { batchSize: 1 });

This example also demonstrates several of the great capabilities of a Pulumi application. First, the ability to define your infrastructure directly in code (i.e. the AWS Queue, Lambda, and Event Subscriptions), instead of needing to do that separately. Second, the ability to avoid needing to specify every single AWS resource necessary to get up and running. For example, I did not have to supply explicit Roles, Role-Policy-Attachments, or Permissions here as Pulumi took care of that for me by default. Third, the ability to create an AWS Lambda directly from a JavaScript/TypeScript arrow-function. That async (event) => ... is written in code as a normal async-arrow-function, but it will be converted by Pulumi into all the machinery necessary to have a true AWS Lambda running in your infrastructure.

After adding the API and writing up a simple app to test things, we then ran a pulumi update to deploy our application to AWS. Unfortunately, we immediately ran into a problem:

Plan apply failed:
Error creating Lambda event source mapping: ValidationException: 1 validation error detected: Value '' at
'startingPosition' failed to satisfy constraint: Member must satisfy enum value set:
[LATEST, AT_TIMESTAMP, TRIM_HORIZON]

So what happened? Well, as it turns out, Pulumi leverages and integrates with many other amazing Open Source projects under the covers. In this case, it was our integration with Terraform that was causing this small snag. It turns out that while Amazon relaxed their API to no longer have that constraint on ‘startingPosition’, Terraform was still referencing an older awssdk API that still contained that constraint.

At Pulumi we think that Terraform is fantastic, and we love using it. So we thought the best thing we could do in this position was to help out that project to support this new functionality as well. We did this around two weeks ago by contributing this PR to their project. We worked with Terraform over a couple over a few days to get the PR up to snuff, and eventually got it in. With this, now both Terraform and Pulumi customers will be able to benefit from these changes in the upcoming releases for both projects!

After getting the change in we went back and tested our example and saw that now everything worked as expected. We also expanded things out in an example to show how you might use this in practice. In this example, we receive the event, and then just write the data of it into an S3 Bucket. Running this example we now successfully see:

pulumi update
Updating stack 'cysqstest'
Performing changes:
Type Name Status
+ pulumi:pulumi:Stack queue-cysqstest created
+ ├─ aws:serverless:Function subscription-queue-subscription created
+ │ ├─ aws:iam:Role subscription-queue-subscription created
+ │ ├─ aws:iam:RolePolicyAttachment subscription-queue-subscription-32be53a2 created
+ │ ├─ aws:iam:RolePolicyAttachment subscription-queue-subscription-7cd09230 created
+ │ └─ aws:lambda:Function subscription-queue-subscription created
+ ├─ aws:sqs:Queue queue created
+ ├─ aws:s3:Bucket testbucket created
+ └─ aws-serverless:queue:QueueEventSubscription subscription created
+ ├─ aws:lambda:Permission subscription created
+ └─ aws:lambda:EventSourceMapping subscription created
info: 11 changes performed:
+ 11 resources created
Update duration: 28s

Once created, we could then see this Stack at app.pulumi.com, and we could easily use the functionality there to even navigate to those resources over at our AWS Console. Over there were were able to use the console to both send a message to the Queue, verify our Lambda got triggered, and even check our Bucket to see the data written into it. In only around a dozen lines of code, we were able to provision 11 AWS resources, and create a real end-to-end Serverless message-receiving and processing pipeline. Now, we can expand on this functionality with more resources or more functionality, run pulumi update and just simply and safely update our cloud infrastructure!

You’ll see this functionality lighting up in our next release of @pulumi/pulumi and @pulumi/aws-serverless. We hope it helps you out and makes your cloud life that much easier. Happy coding!

Build your first serverless app using only JavaScript

Marc Holmes — Thu, 05 Jul 2018 00:00:00 +0000

This post is outdated. The hello-aws-javascript template and @pulumi/cloud-aws package used in this post are no longer maintained. We recommend using TypeScript for new Pulumi projects. See the Get Started guide to build your first serverless app with TypeScript.

In this tutorial, we’ll use Pulumi to build a complete serverless application using only JavaScript. When we say ‘using only JavaScript’, we’re not kidding:

write code just like an Express app… but end up with a fully deployable serverless app
lambdas are… just lambdas
no YAML required… freedom from indentation
all the features of the V8 runtime… async await ahoy
all the behaviors of immutable infrastructure as code tools… but we really mean ‘as code’

Pulumi also supports containers (including Kubernetes), managed services, infrastructure and everything else in between that you might need for building cloud applications. Better than that, you can even combine them all in the same program.

Our first serverless app in 5 lines of JavaScript

After installing the Pulumi CLI, a small index.js file (the main Pulumi code) represents a simple but complete serverless application:

// Add the required package
const cloud = require("@pulumi/cloud-aws");

// Declare an HTTP endpoint
const endpoint = new cloud.API("hello");

// Serve up some static content on that endpoint
endpoint.static("/", "www");

// Create a simple serverless function responding to GET
endpoint.get("/source", (req, res) => res.json({name: "AWS"}));

// Publish the URL so we can easily access our app
exports.url = endpoint.publish().url;

So what’s happening here? This code creates a cloud.API which exposes an HTTP endpoint to the internet. It serves static content at the root of the API from what’s in the www folder on your local machine. And it serves a REST API on GET /source which runs the JavaScript callback to return the JSON object {name: “AWS”} (our serverless function). Finally, it exports the URL where the HTTP API is exposed.

Now just run pulumi update to deploy the application. Before the app deploys, you’ll get a preview of what will be deployed and you can choose to go ahead with the full deployment. Pulumi figures out all of the cloud resources needed to run the code above and prepares that for deployment.

Once deployed, your application will run in the cloud. Use pulumi stack output url to get the URL where your app is running, and open it up in your browser:

With Pulumi, we’ve built a serverless JavaScript app in just a few lines of code. This approach avoided significant amounts of configuration (YAML, or point-and-click). Pulumi also supports containers (including Kubernetes), managed services, infrastructure and everything else in between that you might need for building cloud applications. Get started with Pulumi.

Why JavaScript for serverless programming?

There are many great reasons to use JavaScript for serverless programming (and cloud programming generally). Pulumi also supports TypeScript, Python, and Go with more languages on the way.

Pulumi uses NPM, Express, and supports modern JavaScript features like async await, module etc. and so you can bring your JavaScript skills directly to cloud programming.

Code is the best Config

Serverless platforms such as AWS Lambda offer an easy and cost-effective way to run your code in the cloud. But using these serverless platforms often requires learning lots of finicky details of the cloud platform, and using esoteric YAML configuration files to tell the cloud platform how to run your app. Often, the work needed to configure the app can outweigh the work needed to code the app.

Pulumi changes that, and lets you create serverless cloud applications using only JavaScript (or TypeScript). These applications can use all the features of cloud platforms like Amazon Web Services (AWS), Azure or Google Cloud Platform (GCP) - but exposed as simple JavaScript APIs on NPM.

This means that you can write code that contains logic alongside code that defines the services and infrastructure needed to run it and Pulumi can then deployed the app in seconds and run as a managed, scalable and - in the case of serverless apps such as this one - extremely low cost service on your chosen cloud.

Code is more productive, expressive, and… fun

One of the biggest advantages of a pure code approach is the automatic gains from better tooling support. In particular:

Code completion. There are a lot of services in each cloud. Wouldn’t it be great to be able to autocomplete?
Error checking. Less worry about correctly indented YAML, more real error checking.
Versioning and Packaging. Much improved versioning and packaging scenarios through Github and NPM workflows.
Reusable components. Build on top of base classes, and share for easy reuse, standard configurations and so on.

Want more?

To learn more take a look at more tutorials and example code:

Our origin story: Hello, Pulumi!
Tutorial: Deploying Containers with Pulumi
Tutorial: Build a video thumbnailer using AWS Lambda, Fargate, and S3 in JavaScript
Pulumi Quickstart
Pulumi Community Slack
Pulumi Examples on GitHub
Node.js and Pulumi
Node.js Examples for AWS and Azure

How we use Pulumi to build Pulumi

Luke Hoban — Tue, 26 Jun 2018 00:00:00 +0000

Here at Pulumi we are (perhaps unsurprisingly!) huge fans of using Pulumi to manage our cloud infrastructure and services. We author our infrastructure in strongly-typed programming languages, which allows us to to benefit from rich tooling - documenting and factoring our infrastructure using the same software engineering practices we apply to our application code. This also allows us to create reusable abstractions which accelerate our ability to deliver new features and services, and our ability to standardize and refactor infrastructure patterns across our services with relative ease.

Like other users, we use Pulumi at a variety of levels of abstraction. We use Pulumi for raw infrastructure provisioning, defining the core networking layer for our AWS-based backend infrastructure. And we use Pulumi to define how our application services are deployed into ECS using just a few lines of code. Pulumi hosts and manages static content for www.pulumi.com and get.pulumi.com. We use Pulumi to define the CloudWatch dashboards connected to our infrastructure. And for monitoring, Pulumi defines metrics and notifications/alarms in PagerDuty and Slack.

Best of all, we’ve been able to take things we’ve learned from these use cases, and others we’ve worked with beta users on over the last few months (thank you!), and factor common patterns out into reusable libraries like @pulumi/aws-infra and @pulumi/cloud for ourselves and others to build upon.

In this post, we’ll do a deeper dive into each of these use cases, highlighting unique aspects of how we use Pulumi itself, and some of our engineering processes around how we integrate Pulumi into the rest of our toolchain.

Infrastructure

Our backend infrastructure that supports app.pulumi.com and the Pulumi CLI is fairly typical of a cloud service/application building on AWS. Those who have done this before will be familiar with the combination of AWS VPC, EC2, ASG, ALB, ECS, S3, Aurora, CloudFront and Route53 that powers the core of many AWS-based web applications.

We started off building this infrastructure last year - before we even had completed support for managing AWS resources in Pulumi! As a result, we brought this up initially using CloudFormation. Moving this to Pulumi was an amazing experience. We immediately brought more engineering rigour to the codebase, factoring and decomposing functionality in clearer ways, extracting out useful patterns into named functions, using loops and conditionals instead of copy/paste, and getting completion lists and on-the-fly error checking from within our development tools.

Equally important, authoring our infrastructure in Pulumi gave us the ability to factor out reusable abstractions. We found that our own VPC definition was something that many early customers we worked with also needed - so we factored it out into new awsinfra.Network. We found that standing up an ECS cluster was another big pile of AWS resources that are commonly needed, but can be configured with a much smaller space of configuration values in the common case, so we factored out new awsinfra.Cluster.

Defining infrastructure in Pulumi made it easy to stand up many instances of our service, including per-developer stacks, with just pulumi stack init my-developer-stack and pulumi update. The ability to build reusable components which can be multi-instantiated into the same target cloud is critical here, and something that Pulumi takes care of by default. Within minutes, a developer can have an isolated copy of our full infrastructure stood up, which can be used to test out and quickly iterate on changes. This includes not just the basic infrastructure, but also provisioning of DNS entries in Route53 and creation of custom CloudWatch dashboards (see more on this below).

We have three shared environments - development, staging and production - each an isolated environment with separate configuration. For these shared environments we drive deployments of the Pulumi programs that define the infrastructure above via our CI/CD pipeline. Pulumi can be integrated into any existing CI/CD pipeline or toolchain, via direct use of pulumi preview and pulumi update as part of build and deployment stages. This allows us to reliably deliver a known desired state for each target environment, and to promote changes between environments with just a git merge.

API and App Services

Most of our backend application services are developed in Go, and we deploy them onto our production infrastructure as container images. However, instead of building separate CI pipelines for our container images and the infrastructure they need to function correctly (ECS Service, Application Load Balancer, ECR Repository, etc.), we drive the deployment of both infrastructure and application code via a single Pulumi program. This allows our application code to version along with its supporting infrastructure - when we need to expose a new port for example, we can make the changes to both application code and load balancer as part of a single commit, and promote it through CI as an atomic action. We have found this to significantly simplify the delivery of containerized services on AWS, as well as making it easy to stand up new services into our existing shared networking and cluster infrastructure.

Again, we found this to be a common pattern as we worked with other early customers, and factored out a library to simplify deployment of containerized services into a production deployment on AWS as part of the cloud.Service API in @pulumi/cloud. Here’s an example of using an application service defined in the ./airflow-container folder (with a Dockerfile) into an ECS cluster. This one declaration will provision a scaled-out ECS Service for running the container images, a ECR repository to store the image, optionally a load balancer if ports were exposed, and each pulumi update will build the container image, push it to the private repository, and turn-over each service task to use the new image. All of that workflow complexity is abstracted away behind a simple API and a single CLI command.

Metrics, Alarms and Dashboards

We also use Pulumi to define a few less traditional parts of our infrastructure - including our metrics, dashboards and alarming infrastructure on top of AWS CloudWatch.

Defining and versioning this along with our infrastructure has significant benefits: new features are built with metrics, dashboards and alarms lit up from the get-go, developers get a full set of dashboards as part of their personal isolated development environments, and promotions from development to staging to production can verify not just physical infrastructure, but also that alarms and dashboards are behaving as expected prior to pushing code to production. Here’s a sample of our CloudWatch dashboards:

We build up AWS resources representing CloudWatch dashboards using a compositional framework shown below. As this example shows, bringing software engineering approaches to infrastructure as code can lead to new approaches to how code is authored and factored - and ultimately to the creation of higher level frameworks for building domain-specific cloud infrastructure - in this case, CloudWatch dashboards.

The same metrics that power these dashboards are also used to power notifications and alarms. In these cases, we want to define both the infrastructure for the alarm - the SNS topics and registration of those topics with CloudWatch - as well as the small piece of code that fires off Pagerduty and Slack notifications on notifications being pushed to that SNS topic - all in one codebase. This ensures that we can version these together, and that we can deploy both consistently as part of updates. To do this, we include the event handlers for these infrastructure events directly inline in our Pulumi programs. Note in the code snippet below how we define the infrastructure for a Topic, and then conditionally wire up event handlers on that topic, including the code to execute at runtime on these events directly in our Pulumi program. This code can even reach out and reference configuration used to provision our Pulumi deployment - key to enabling this to be deployed seamlessly into a variety of different development environments.

Static Content

We serve three properties as static content from S3, each described using Pulumi. One of these is https://get.pulumi.com which hosts downloads of our pulumi CLI installer. It’s a very simple program, and even better - it’s open source on GitHub. It combines AWS S3, CloudFront and Route53.

This infrastructure is deployed and managed using the Pulumi Service at app.pulumi.com, allowing us to track the history of deployments and changes to the infrastructure and quickly jump in to the resources defined by this stack in the AWS console.

Conclusion

Treating cloud infrastructure as software has changed how we think about infrastructure engineering. Using Pulumi for both traditional cloud infrastructure as well as to get application code and supporting service infrastructure into production provides a simple workflow and the ability to create simple high-level APIs to make new services easy to define and deploy.

Try out Pulumi yourself today!

Code, Deploy, and Manage a Serverless REST API on AWS

Donna Malayeri — Fri, 22 Jun 2018 00:00:00 +0000

Pulumi makes it easy to build serverless applications and connect to other cloud resources. In this blog post, we’ll create a simple REST API that counts the number of times a route has been hit, using JavaScript to define both the infrastructure and application code.

In Pulumi, you define your application infrastructure in regular code, using JavaScript, Python or Go, and you can target AWS, Azure, Google Cloud, or Kubernetes. The Pulumi command line tool transforms your into a declarative plan, following the best practices of immutable infrastructure. You can write your app code in any language supported by your serverless platform.

Example app: serverless route counter

In this tutorial, we’ll build a simple REST API that counts the number of times a route has been hit. For example:

$ curl $URL/wow
{"route":"wow","count":1}
$ curl $URL/pulumi
{"route":"pulumi","count":1}
$ curl $URL/wow
{"route":"wow","count":2}

We’ll implement this using API Gateway, Lambda, and Dynamo DB:

Setup

If this is your first time using Pulumi, go to app.pulumi.com and sign in with GitHub.

Then, run the following command to install the Pulumi CLI:

$ curl -fsSL https://get.pulumi.com/ | sh

If you’re on Windows, run this:

@"%SystemRoot%System32WindowsPowerShell1.0powershell.exe" -NoProfile -InputFormat None -ExecutionPolicy Bypass -Command "iex ((New-Object System.Net.WebClient).DownloadString('https://get.pulumi.com/install.ps1'))"
SET "PATH=%PATH%;%USERPROFILE%.pulumiin"

You’ll deploy this app to your own AWS account, so follow the steps to configure your AWS account.

Make sure you have Node.js installed, with a version of 6.10.x or later.

Create the App

First we’ll create a Pulumi project and add code for both the infrastructure definitions and application code.

To create a new Pulumi project, run the following commands:
```
mkdir hello-http && cd hello-http
pulumi new aws-javascript
```
This creates a new project in the directory hello-http.

Replace the contents of index.js with the following:

const cloud = require("@pulumi/cloud-aws");
/* Create a mapping from 'route' to a count */
let counterTable = new cloud.Table("counterTable", "route");
/* Create an REST API endpoint */
let endpoint = new cloud.API("hello-world");
endpoint.get("/{route+}", (req, res) => {
let route = req.params["route"];
console.log(`Getting count for '${route}'`);
/* get previous value and increment */
/* reference outer counterTable object */
counterTable.get({ route }).then(value => {
let count = (value && value.count) || 0;
counterTable.insert({ route, count: ++count }).then(() => {
res.status(200).json({ route, count });
console.log(`Got count ${count} for '${route}'`);
});
});
});
exports.endpoint = endpoint.publish().url;

The definition for counterTable stores a counter for each route, using cloud.Table. On AWS, this provisions a DynamoDB instance. To create a new API Gateway instance, we create an instance of cloud.API. New routes can be added to this endpoint using functions such as get, post, put etc.

The function passed to get is the interesting part: this becomes the body of a new AWS Lambda function that is called on a GET request to the API Gateway. The body of this function can use variables defined in the main program, such as counterTable. This is translated to a lookup on the provisioned DynamoDB instance; there is no need to store the table name in an environment variable.

Finally, add @pulumi/cloud-aws NPM package:

$ npm install --save @pulumi/cloud @pulumi/cloud-aws

Deploy the App

To deploy both the infrastructure and app code, we’ll run pulumi update. This command first shows a preview of all the resources that will be created and prompts for confirmation.

Choose the “yes” option to deploy to AWS. At the end of the update, you’ll see a link to the Pulumi Service that shows the details of the deployment.

Go to this link and click the Resources tab. You’ll see all the resources you’ve created, including links to the AWS Console.

Test the App

Now that the app is deployed, let’s try it out! With the Pulumi CLI, you can easily view output properties for your stack. This line in the JavaScript code creates a stack output.

exports.endpoint = endpoint.publish().url;

You can view stack outputs in the Pulumi Service, or via pulumi stack output:

$ pulumi stack output
Current stack outputs (1):
OUTPUT VALUE
endpoint https://5e8xrktey3.execute-api.us-west-2.amazonaws.com/stage/

Now, let’s curl some routes!

$ curl $(pulumi stack output endpoint)/wow
{"route":"wow","count":1}
$ curl $(pulumi stack output endpoint)/pulumi
{"route":"pulumi","count":1}
$ curl $(pulumi stack output endpoint)/wow
{"route":"wow","count":2}

You can also view logs for the Lambda via pulumi logs. To get a log tail, use the --follow or -f flag:

$ pulumi logs
Collecting logs for stack hello-http-dev since 2018-06-15T12:53:35.000-07:00.
2018-06-15T13:52:30.123-07:00[ hello-world4fcc7b60] Getting count for 'wow'
2018-06-15T13:52:34.361-07:00[ hello-world4fcc7b60] Got count 1 for 'wow'
2018-06-15T13:52:39.621-07:00[ hello-world4fcc7b60] Getting count for 'pulumi'
2018-06-15T13:52:39.757-07:00[ hello-world4fcc7b60] Got count 1 for 'pulumi'
2018-06-15T13:52:42.189-07:00[ hello-world4fcc7b60] Getting count for 'wow'
2018-06-15T13:52:42.325-07:00[ hello-world4fcc7b60] Got count 2 for 'wow'

Clean up

To clean up the resources, run pulumi destroy.

Next steps

The sample code for this application is available in the Pulumi examples repo on GitHub. For an end-to-end TypeScript application with a frontend, see the URL shortener sample.

Build a Video Thumbnailer on AWS

Donna Malayeri — Thu, 21 Jun 2018 00:00:00 +0000

Pulumi makes it easy to build cloud applications that use a combination of containers, lambdas, and connected data services and infrastructure: Colada apps.

An example of a Colada app is extracting a thumbnail from a video. A serverless function can only run for 5 minutes, so we’ll run a container in AWS Fargate to do the video processing.

In this app, a Lambda function is triggered whenever a new video is uploaded to S3. This function launches a task in Fargate that uses FFmpeg to extract a video thumbnail. A second Lambda function is triggered when a new thumbnail has been created.

This post was inspired by a blog post from Serverless, Inc that shows how to combine AWS Fargate and Lambda for a long-running process. Let’s see how a similar app would be implemented in Pulumi entirely in code.

Setup

If this is your first time using Pulumi, go to https://app.pulumi.com and sign in with GitHub.

Then, run the following command to install the Pulumi CLI:

$ curl -fsSL https://get.pulumi.com/ | sh

If you’re on Windows, run this:

@"%SystemRoot%System32WindowsPowerShell1.0powershell.exe" -NoProfile -InputFormat None -ExecutionPolicy Bypass -Command "iex ((New-Object System.Net.WebClient).DownloadString('https://get.pulumi.com/install.ps1'))"
SET "PATH=%PATH%;%USERPROFILE%.pulumiin"

You’ll deploy this app to your own AWS account, so follow the steps to configure your AWS account.

Make sure you have Node.js installed, with a version of 6.10.x or later.

Finally, make sure Docker is installed and running.

Create the App

We’ll create a Pulumi project, define the infrastructure and app code in JavaScript, and create a Dockerfile for the Fargate task.

First, to create a new Pulumi project, run the following commands:

mkdir hello-colada && cd hello-colada
pulumi new aws-javascript

This creates a new project in the hello-colada directory.

Second, replace the contents of index.js with the following:

const cloud = require("@pulumi/cloud-aws");

// A bucket to store videos and thumbnails.
const bucket = new cloud.Bucket("bucket");
const bucketName = bucket.bucket.id;

// A task which runs a containerized FFMPEG job to extract a thumbnail image.
const ffmpegThumbnailTask = new cloud.Task("ffmpegThumbTask", {
 build: "./", // folder containing the Dockerfile
 memoryReservation: 128,
});

// When a new video is uploaded, run the FFMPEG task on the video file.
// Use the time index specified in the filename (e.g. cat_00-01.mp4 uses timestamp 00:01)
bucket.onPut("onNewVideo", async (bucketArgs) => {
 console.log(`*** New video: file ${bucketArgs.key} was uploaded at ${bucketArgs.eventTime}.`);
 const file = bucketArgs.key;

 const thumbnailFile = file.substring(0, file.indexOf('_')) + '.jpg';
 const framePos = file.substring(file.indexOf('_')+1, file.indexOf('.')).replace('-',':');

 await ffmpegThumbnailTask.run({
 environment: {
 "S3_BUCKET": bucketName.get(),
 "INPUT_VIDEO": file,
 "TIME_OFFSET": framePos,
 "OUTPUT_FILE": thumbnailFile,
 },
 });
 console.log(`Running thumbnailer task.`);
}, { keySuffix: ".mp4" });

// When a new thumbnail is created, log a message.
bucket.onPut("onNewThumbnail", async (bucketArgs) => {
 console.log(`*** New thumbnail: file ${bucketArgs.key} was saved at ${bucketArgs.eventTime}.`);
}, { keySuffix: ".jpg" });

// Export the bucket name.
exports.bucketName = bucketName;

This code uses cloud.Task, a high-level, convenient component for working with containers. The component automatically provisions a container registry instance in ECR, runs a Docker build, and saves the Docker image to the provisioned ECR instance. It also defines an ECS task and configures it to use the built image. All this in just 4 lines of code!

Next, in the same directory, create a Dockerfile with the following contents. We’ll use an existing FFmpeg container and install the AWS CLI. When the container is started, it copies the video file from S3, runs ffmpeg, and copies the output back to S3.

 FROM jrottenberg/ffmpeg
RUN apt-get update &&
apt-get install python-dev python-pip -y &&
apt-get clean
RUN pip install awscli
WORKDIR /tmp/workdir
ENTRYPOINT
echo "Starting ffmpeg task..." &&
echo "Copying video from S3" &&
aws s3 cp s3://${S3_BUCKET}/${INPUT_VIDEO} ./${INPUT_VIDEO} &&
ffmpeg -v error -i ./${INPUT_VIDEO} -ss ${TIME_OFFSET} -vframes 1 -f image2 -an -y ${OUTPUT_FILE} &&
echo "Copying thumbnail to S3" &&
aws s3 cp ./${OUTPUT_FILE} s3://${S3_BUCKET}/${OUTPUT_FILE}

Install the @pulumi/cloud-aws NPM package:

$ npm install --save @pulumi/cloud-aws @pulumi/cloud

Finally, configure Pulumi to use AWS Fargate. Note that, currently, Fargate is available only in us-east-1, us-east-2, us-west-2, and eu-west-1.

$ pulumi config set cloud-aws:useFargate true

That’s it! Now, let’s deploy the app.

Deploy the App

To deploy both the infrastructure and app code, we’ll run pulumi update. This command first shows a preview of all the resources that will be created and prompts for confirmation. During the preview phase, Pulumi invokes docker build.

Choose the “yes” option to deploy to AWS. This will take about 5 minutes. Pulumi automatically builds and provisions an AWS container repository in ECR, builds the Docker container, and places the image in the repository. This all happens automatically and does not require manual configuration on your part.

At the end of the update, you’ll see a link to the Pulumi Service that shows the details of the deployment.

Go to the Resources tab and filter to S3 resources. Then, on the S3 bucket, click the link to go to the AWS Console.

Upload a Video

Upload an .mp4 video to your S3 bucket, making sure to encode the desired time index in the filename. For instance, use myvideo_00-02.mp4 to extract the frame at 0 minutes and 2 seconds.

You can use this video of my cat, which you should save as cat_00-01.mp4.

Upload this video in the S3 console. Or, use the AWS CLI along with the pulumi stack output command:

$ aws s3 cp cat_00-01.mp4 s3://$(pulumi stack output bucketName)
upload: cat_00-01.mp4 to s3://bucket-3a4f226/cat_00-01.mp4

To see what the app is doing, use the pulumi logs command, with the --follow or -f parameter. This command aggregates all the logs for your compute, so the logs for the Lambda function and the Fargate task are all in one place!

Once the thumbnail has been generated, either view it in the S3 console, or download it with the AWS CLI:

$ aws s3 cp s3://$(pulumi stack output bucketName)/cat.jpg .
download: s3://bucket-0c91106/cat.jpg to ./cat.jpg

Clean Up

To clean up the resources we’ve provisioned, run pulumi destroy.

Next Steps

In this post, we saw how easy it is to use containers and serverless functions in one application. With Pulumi, you get the best of both worlds, and don’t have to choose one or the other.

The sample code for this post is available in the Pulumi examples repo on GitHub.

For a version of this sample that includes AWS Rekognition, see the Video Thumbnailer with Machine Learning JavaScript example.

Pulumi Blog: Serverless

Self-Verifying AI Agents: Vercel's Agent-Browser in the Ralph Wiggum Loop

Why browser automation matters for AI coding

The context problem with browser automation

Vercel’s “less is more” philosophy

Applying “less is more” to browser automation

Installing agent-browser

The experiment: adding analytics

Agent-browser workflow in practice

The numbers: 82.5% context reduction

What I noticed

When to use each

How Ralph Wiggum Built a Serverless SaaS with Pulumi

The problem with AI coding assistants

What if we just do not let it stop?

How the Ralph loop works

Why Pulumi is perfect for Ralph

The experiment

The results

Home screen

Dashboard

Documentation

Best practices for Ralph with Pulumi

The honest assessment

Try it yourself

Or try Pulumi Neo: Ralph built-in

Resources

Host your Python app for $1.12 a month

How does the pricing break down? (2026)

Cheap Python hosting compared (2026)

Why use AWS Lambda for low-traffic Python apps?

Why containers?

How do I build a Flask app for Lambda?

How do I containerize a Flask app for Lambda?

Do I need to learn Lambda-specific patterns?

How do I deploy a Lambda container with Pulumi?

Why is this approach worth it?

Alternative approaches

Conclusion

Related reading

Changelog

Pinecone Provider Now Available for Pulumi

Hello, Pulumi Pinecone Provider! 👋

What is Pinecone? 🧐

Getting Started 🤖

Deploying a Pinecone Serverless Index

Deleting a Pinecone Index

Programmatically Creating a Pulumi Pinecone Provider

Conclusion 💡

Serverless Arch Templates

Serverless on any Cloud, in any Language

AWS

GCP

Azure

Pulumi to the Cloud

Extending Templates

Accessing Secrets Safely in Lambda Functions

Getting started

Structured configuration

Writing our Pulumi program

A warning about accessing the configuration

Accessing the configuration

Storing the secret values

Identity and Access Management

Finally, the Function

The Resource

The Code

Deployment time

Tidying up

Complete code

Summary

Secure AWS Lambda with IAM ABAC Policies

What are Attributes?

Using ABAC for AWS Lambda Functions

Try It Out!

Deploying a URL Shortener with Cloudflare Workers

Setting Up

Our URL Shortener

Deploying Our Worker

Where’s the Types?!

API Project Application - Different `dev` and `prod` Stacks