Pulumi Blog: Josh Kodroff

Day 2 Operations: Drift Detection and Remediation

Mitch Gerdisch — Fri, 24 Oct 2025 00:00:00 +0000

Welcome to the fourth post in our IDP Best Practices series. Today we’re diving into the world of drift detection and remediation, those critical day 2 operations that keep your infrastructure aligned with its intended configuration long after the initial deployment.

You’ve built a beautiful platform with robust guardrails, comprehensive templates, and well-defined golden paths. Your developers are productive, deployments are smooth, and everything seems perfect. Then reality hits. An on-call engineer makes an emergency change through the AWS console during a 3 AM incident. A team member tweaks a security group rule to debug a connection issue and forgets to revert it. Auto-scaling adjusts capacity based on load patterns. Before you know it, your actual infrastructure has quietly diverged from what your code describes.

This post continues our journey through the IDP Best Practices series. We’ve covered strategy and self-service infrastructure, built golden paths with components and templates, and established deployment guardrails through policy as code. Now we’re tackling what happens after deployment when the real world starts making changes to your carefully crafted infrastructure.

The Reality of Infrastructure Drift

Let me paint you a picture that might feel uncomfortably familiar. It’s Saturday afternoon, and you’re enjoying a barbecue with friends when your phone buzzes with that dreaded alert: production is down. You rush to your laptop, but you’re not on the corporate VPN. The AWS console is accessible, but you can’t reach the application itself to properly diagnose the issue.

Faced with a growing incident and impatient stakeholders, you make a quick decision. You modify the security group directly in the console, temporarily allowing access from anywhere (0.0.0.0/0) just so you can get in and fix the problem. Twenty minutes later, the issue is resolved, everyone’s relieved, and you return to your barbecue. That temporary security group change? It stays there, a silent time bomb waiting to be discovered in your next security audit.

This scenario plays out in organizations every day. Your infrastructure has now drifted from its intended state, and it will continue drifting, accumulating more changes, until someone or something notices and fixes it. Each untracked change increases the gap between what you think you have and what actually exists in production.

Understanding the Prevention vs. Detection Paradigm

When building a robust platform, you need to think in layers of defense. Your first instinct might be to prevent all drift from happening in the first place, and that’s absolutely the right starting point. But prevention alone isn’t enough in the messy reality of production operations.

Prevention: The First Line of Defense

Prevention is what we’ve been building throughout this series. You establish code reviews where teammates scrutinize infrastructure changes before they go live. You implement automated testing that validates configurations before deployment. You deploy CrossGuard policies that act as guardrails, blocking misconfigurations before they can cause damage. You create standardized components that encode best practices, making it easier to do the right thing than the wrong thing.

These preventative measures work brilliantly for changes that flow through your normal IaC pipeline. They catch typos, enforce standards, and maintain consistency across your infrastructure. But they have a critical blind spot: they can’t see changes that bypass your pipeline entirely.

Detection and Remediation: The Safety Net

This is where drift detection enters the picture. Think of it as continuous verification, constantly comparing what should exist with what actually exists. While prevention blocks most issues at the gate, detection catches everything that finds another way in.

What kinds of changes slip past prevention? Emergency modifications made during incidents when there’s no time for a proper code review. Console changes made by well-meaning engineers debugging production issues. External factors you can’t control, like AWS changing service limits or deprecating API endpoints. Even accidental modifications by team members who thought they were in the staging environment.

The truth is, no matter how robust your preventative controls, you need detective controls to maintain infrastructure integrity. Prevention and detection work together, creating a comprehensive defense against infrastructure chaos.

What Is Infrastructure Drift?

Infrastructure drift sounds technical, but the concept is surprisingly simple. Drift occurs when your actual infrastructure state doesn’t match your declared state in code. It’s the gap between intention and reality, between what you coded and what’s actually running.

To understand drift properly, you need to grasp three distinct states that Pulumi manages. First, there’s the desired state, which is what your Pulumi program declares in code. This is your intention, your blueprint for how things should be. Then there’s the current state, stored in Pulumi’s state file, which tracks what Pulumi believes exists based on its last interaction with your cloud provider. Finally, there’s the actual state, the ground truth of what really exists in your cloud provider right now.

The journey to drift typically follows a predictable pattern. You start by running pulumi up to deploy your resources, and everything is in perfect harmony. Pulumi creates the resources, records their configuration in its state file, and all three states align perfectly. Time passes, and your resources continue running in the cloud, serving traffic and doing their job.

Then someone makes a manual change. Maybe they modify a resource directly in the console, or an external system alters a configuration. Suddenly, your actual state has diverged from both your current state (what Pulumi thinks exists) and your desired state (what your code says should exist). You now have drift, and it will persist until you detect and address it.

Pulumi’s Drift Detection and Remediation Workflow

Pulumi provides a complete workflow for handling drift that’s both powerful and surprisingly straightforward. Rather than treating drift as an exceptional case, Pulumi makes it a routine part of infrastructure management.

1. Drift Detection with `pulumi refresh`

The foundation of drift detection is the pulumi refresh command. When you run this command, Pulumi springs into action, querying your cloud providers to discover the actual state of your resources. It then compares this reality against what it has recorded in its state file:

# Run drift detection
pulumi refresh --preview-only

# This command:
# 1. Reads your current Pulumi state
# 2. Queries cloud providers for actual resource configurations
# 3. Identifies any differences
# 4. Reports what has drifted

The --preview-only flag is particularly useful here. It shows you exactly what’s drifted without making any changes to your state file. You get a clear report of the differences, allowing you to review and understand the drift before deciding how to handle it.

2. State Reconciliation with `pulumi refresh`

After reviewing the drift, you face a decision. If the changes are legitimate and should be preserved, you can update your state file to match reality:

# Update state file to match cloud reality
pulumi refresh

# This command:
# 1. Detects drift (as above)
# 2. Updates state file to match actual cloud state
# 3. Prepares for remediation via pulumi up

This reconciliation step is crucial. It acknowledges the current reality and updates Pulumi’s understanding of your infrastructure without making any actual changes to your resources. You’re essentially telling Pulumi, “This is what actually exists right now.”

3. Drift Remediation with `pulumi up`

With your state file now reflecting reality, you can use pulumi up to remediate the drift. This command compares the updated state against your code and determines what changes are needed to restore your infrastructure to its intended configuration:

# Remediate drift - restore to desired state
pulumi up

# This command:
# 1. Compares updated state with your Pulumi program
# 2. Identifies required changes
# 3. Applies changes to restore infrastructure

This three-step dance of detect, reconcile, and remediate gives you complete control over how you handle drift. You can review changes, decide whether to keep or revert them, and ensure your infrastructure returns to its intended state.

Automating Drift Detection with Pulumi Deployments

Running pulumi refresh manually works great for small teams and simple infrastructures, but it doesn’t scale. As your platform grows to dozens or hundreds of stacks across multiple environments, manual drift detection becomes impossible. This is where Pulumi Deployments transforms drift detection from a manual chore into an automated safeguard.

Setting Up Automated Drift Detection

The journey to automated drift detection starts with connecting your infrastructure code to Pulumi’s deployment system. Let’s walk through the setup process.

Step 1: Configure Deployment Settings

Your first step is connecting Pulumi to your source control system using Deployment Settings. This integration allows Pulumi to access your infrastructure code and run automated operations:

Pulumi offers native integrations with all major version control systems. If you’re using GitHub, you get full app integration with PR previews, making it easy to see the impact of changes before merging. GitLab users enjoy similar functionality with merge request automation. For other Git providers or self-hosted solutions, you can use raw Git integration with direct repository access using credentials.

Step 2: Create Drift Detection Schedules

With your source control connected, you can now configure automated drift detection to run on whatever schedule makes sense for your organization. Some teams check hourly for production environments, while others might run daily checks for development stacks:

For environments where you’re confident in your automation, you can take it a step further and enable automatic remediation:

With auto-remediation enabled, Pulumi not only detects drift but automatically runs pulumi up to restore your infrastructure to its intended state. This is powerful but should be used carefully, especially in production environments.

Step 3: Configure Webhooks for Notifications

Detecting drift is only valuable if the right people know about it. Pulumi’s webhook system ensures your team stays informed when drift occurs:

You can route notifications wherever your team actually pays attention. Send alerts directly to Slack channels where your ops team congregates. Push notifications to Microsoft Teams if that’s your collaboration platform. Use custom webhooks to integrate with PagerDuty, Datadog, or any other monitoring system. You can even use deployment triggers to automatically run dependent stacks when drift is detected and fixed.

Automate drift detection with Pulumi

Connect your stacks to Pulumi Deployments, schedule drift checks across every environment, and route alerts to Slack or Teams the moment your infrastructure diverges from code.

Get started

Handling Common Drift Scenarios

Every organization faces similar drift scenarios. Understanding how to handle these common cases will prepare you for real-world operations.

Scenario 1: Emergency Hotfix

The most common drift scenario occurs during incidents. Someone makes an emergency change to fix production, and now you need to decide whether to keep or revert that change:

# 1. Acknowledge the drift
pulumi refresh --stack production

# 2. Review the changes
pulumi preview --stack production

# 3. Either incorporate or revert
# Option A: Accept the change
pulumi refresh --stack production --yes

# Option B: Revert to code
pulumi up --stack production --yes

The key here is having a clear process. First, acknowledge the drift by running refresh to update your state. Then review the changes carefully. Finally, make an explicit decision: either accept the change by updating your code to match, or revert it by running pulumi up to restore the original configuration.

Scenario 2: Legitimate External Changes

Not all drift is bad. Auto-scaling groups adjust capacity based on load. Managed services update configurations automatically. Some changes are legitimate and should be preserved:

// Mark resources that can change externally
const autoScalingGroup = new aws.autoscaling.Group("app-asg", {
 minSize: 2,
 maxSize: 10,
 // ... other config
}, {
 ignoreChanges: ["desiredCapacity"] // Ignore scaling changes
});

The ignoreChanges option tells Pulumi to ignore specific properties when detecting drift. Use it for fields that legitimately change outside your control, like auto-scaling capacity or last-modified timestamps.

Scenario 3: Configuration Drift

Sometimes drift occurs in configuration values rather than resource structure. IP allowlists, feature flags, and environment-specific settings often drift. Handle these with Pulumi’s configuration system:

# Use Pulumi's configuration system
config = pulumi.Config()
allowed_ips = config.get_object("allowed_ips") or ["10.0.0.0/8"]

security_group = aws.ec2.SecurityGroup("app-sg",
 ingress=[{
 "from_port": 443,
 "to_port": 443,
 "protocol": "tcp",
 "cidr_blocks": allowed_ips # Managed via configuration
 }]
)

By externalizing configuration values, you make them easier to update without code changes. This pattern works particularly well for values that change frequently or differ between environments.

The Value of Automated Drift Detection

After implementing drift detection for hundreds of stacks, patterns emerge that tell a compelling story about its value. The transformation begins subtly but grows profound as teams discover what they’ve been missing.

The first change most organizations notice is in their security posture. Where once unauthorized changes might lurk undetected for weeks, only surfacing during quarterly audits or worse, during incident investigations, now they’re caught within minutes. Security teams who previously dreaded audit season find themselves with continuous compliance data at their fingertips. Every change, whether legitimate or not, creates an audit trail that transforms compliance reporting from a scrambling fire drill into a routine export of already-collected data.

As teams settle into this new reality, operational improvements become apparent. Incidents that used to take hours to diagnose now resolve in minutes because engineers can immediately check whether drift contributed to the problem. One team reported that their mean time to recovery dropped by 40% simply because they could eliminate drift as a cause within seconds rather than spending hours manually comparing configurations. Change management, once a bureaucratic nightmare of spreadsheets and approval chains, transforms into an automated system where every modification is tracked, timestamped, and traceable to its source.

Then come the unexpected discoveries. Drift detection becomes a cost optimization tool that nobody anticipated. It uncovers orphaned resources spinning away forgotten in some corner of your cloud account, created during a debugging session six months ago. It catches that instance someone manually upgraded to a larger size during a performance investigation and forgot to downsize. One organization discovered they were spending $15,000 monthly on resources that weren’t even supposed to exist, all caught by their drift detection system in its first week of operation.

Perhaps most importantly, drift detection changes how teams work. Engineers stop wasting time on manual infrastructure audits. Debugging accelerates when you have clear reports showing exactly what changed and when. The constant anxiety about whether production matches your code disappears, replaced by confidence that any discrepancies will be caught and reported immediately. Teams report feeling more empowered to move quickly, knowing they have a safety net that will catch configuration drift before it causes problems.

The cumulative effect is transformative. Organizations running drift detection report not just fewer incidents and lower costs, but a fundamental shift in how they think about infrastructure management. It becomes proactive rather than reactive, confident rather than anxious, automated rather than manual. What starts as a simple scheduled check evolves into a cornerstone of operational excellence.

Getting Started with Drift Detection

Ready to implement drift detection for your IDP? Here’s a practical action plan that you can execute today.

Quick Start Checklist

Begin with the simplest possible drift detection: a manual check on your production stack. Run pulumi refresh --preview-only --stack production to see if you have any existing drift. This takes just 15 minutes and often reveals surprising discrepancies.

Next, invest 30 minutes in setting up automated detection. Configure Pulumi Deployments, create an hourly detection schedule for your most critical stack, and set up Slack notifications so you’ll know immediately when drift occurs.

Spend the next week monitoring and learning. Review the drift patterns that emerge. Identify common causes. Are certain resources drifting repeatedly? Are changes happening at predictable times? Document which changes are legitimate and which represent problems.

After a week of observation, you’re ready to implement remediation. Start with non-production environments where mistakes have lower impact. Enable auto-remediation gradually, monitoring success rates and adjusting your configuration based on what you learn.

Finally, optimize and scale your drift detection. Adjust schedules based on the patterns you’ve observed. Implement custom workflows for complex scenarios. Expand coverage to all your environments and stacks. What starts as a simple check becomes a comprehensive system protecting your entire infrastructure.

Conclusion: Day 2 Operations as a Competitive Advantage

Drift detection and remediation represent more than just good hygiene; they’re competitive advantages in a world where infrastructure complexity keeps growing.

Teams with robust day 2 operations move faster because they have confidence in their infrastructure state. They recover quickly from incidents because they can immediately identify what changed. They maintain security without slowing development by automatically detecting and reverting unauthorized modifications. Most importantly, they scale operations without scaling headcount, automating work that would otherwise require armies of engineers manually checking infrastructure.

The math is compelling. A single undetected security group change could lead to a breach costing millions. One overlooked configuration drift might cause hours of downtime. Yet implementing comprehensive drift detection takes just hours of setup and minutes of ongoing maintenance. The ROI is immediate and substantial.

Our IDP journey continues with “Extend Your IDP for AI Applications: GPUs, Models, and Cost Controls”. As AI workloads become central to modern applications, we’ll explore how to adapt your platform for machine learning workflows. You’ll learn about GPU orchestration, model deployment pipelines, and the unique cost management challenges that AI infrastructure presents.

But don’t wait for the next post to get started with drift detection. Even a simple hourly check can prevent major incidents. Set up basic detection today, and your future self will thank you the next time you’re debugging a production issue at 3 AM. More importantly, your on-call team will appreciate not having to debug issues caused by accumulated drift that could have been prevented.

Ready to implement drift detection? Start your free Pulumi Cloud trial and set up your first drift detection schedule in minutes. No credit card required.

Amazon EKS Hybrid Nodes: Bridging Cloud and On-Premises

Josh Kodroff — Sun, 01 Dec 2024 15:06:25 -0500

AWS has introduced a new feature for Amazon Elastic Kubernetes Service (EKS): Amazon EKS Hybrid Nodes. This addition to the Amazon EKS Hybrid/Edge portfolio allows organizations to maintain their Kubernetes control plane in AWS while running workloads on-premises or at the edge. This hybrid approach offers the best of both worlds - AWS’s reliable and scalable control plane management combined with the flexibility to run workloads wherever they make the most sense for your business.

Amazon EKS Hybrid Nodes are particularly valuable for organizations that need to keep certain workloads on-premises due to data sovereignty requirements, latency concerns, or existing infrastructure investments. By extending EKS to support on-premises workloads, AWS is addressing a critical need in the enterprise market, especially for industries like manufacturing, telecommunications, and healthcare that often require local data processing.

Pulumi’s strategic partnership with AWS has always focused on making cloud infrastructure easier to manage through modern infrastructure as code. As an EKS Service Ready Partner, Pulumi provides first-class support for AWS services, including EKS. This partnership ensures that as AWS releases new features like EKS Hybrid Nodes, Pulumi users can quickly adopt and integrate them into their infrastructure management workflows. Managing Hybrid Environments with Pulumi

Pulumi’s infrastructure as code platform is perfectly positioned to help organizations manage their hybrid EKS environments. Using Pulumi, teams can define and manage both their EKS clusters and Kubernetes resources using their favorite programming languages, bringing software engineering practices to infrastructure management. This includes managing Hybrid Node configurations, networking setup, and security policies across both cloud and on-premises environments.

Pulumi’s Environments Secrets and Config (ESC) adds another layer of capability to Hybrid Node deployments. Operators can securely manage OIDC IAM credentials, kubeconfig files and other sensitive credentials through ESC, ensuring secure access to EKS clusters regardless of where the workloads run. Furthermore, pods running on Hybrid Nodes can consume ESC secrets through Pulumi ESC’s integration with the Kubernetes External Secrets Operator, maintaining a consistent secrets management approach across your entire hybrid infrastructure.

Pulumi’s solutions for EKS Anywhere are automatically validated across a variety of EKS deployment models (including cloud, VMWare, Nutanix, and bare metal) via AWS’ Conformitron framework. The framework makes it simple for AWS partners to test their offerings across common customer environments without having to provision these environments themselves.

Automation and Operations

Organizations can leverage Pulumi’s CI/CD capabilities within their Hybrid Node environment. Pulumi Deployments or the Pulumi Kubernetes Operator can run directly on Hybrid Nodes, enabling automated infrastructure management from within your on-premises environment. This approach allows for consistent infrastructure automation while respecting network boundaries and security requirements.

Summary

The combination of EKS Hybrid Nodes and Pulumi’s infrastructure as code platform provides organizations with a powerful solution for managing hybrid Kubernetes environments. As this feature continues to evolve, Pulumi will provide comprehensive support for deploying and managing Hybrid Node configurations, making it easier for teams to adopt and maintain hybrid architectures while modernizing their application workloads.

Stay tuned for upcoming Pulumi examples for managing EKS Hybrid Nodes, which will enable you to manage your hybrid Kubernetes infrastructure using your preferred programming language, complete with type safety, testing capabilities, and modern development practices.

If you’d like to learn more about how you can harness the power of EKS with Pulumi, check out the latest version of our EKS package or join us for our AWS Immersion Day workshop following re:Invent 2024.

The Guide to Platform Engineering: 7 Steps to Get It Right

Sara Huddleston — Tue, 22 Oct 2024 07:47:03 +0000

In today’s fast-paced digital landscape, organizations are increasingly adopting platform engineering to optimize their software delivery and operations. Gartner predicts that by 2026, 80% of large software engineering organizations will have platform engineering teams to provide reusable services, components, and tools for application delivery. Additionally, by 2027, 80% of large enterprises will leverage platform engineering to scale DevOps initiatives in hybrid cloud environments effectively.

This shift is driven by the rise of cloud adoption, where many enterprises face the challenge of uncoordinated application teams deploying workloads in different ways across various cloud platforms. This siloed approach often results in a lack of standardization, security risks, and operational inefficiencies.

Platform engineering offers a strategic solution to these issues. This guide provides the essential steps to successfully implement platform engineering, from laying the foundation to scaling internal developer platforms (IDPs) for future growth.

On this article:

Step 1: Securing Executive Buy-in

The first step in your platform engineering journey is securing executive buy-in. This high-level support is essential, as platform engineering teams must create an organization-wide strategy that integrates internal developer portals (IDPs) and self-service capabilities. Leadership needs to understand how platform engineering can address inefficiencies caused by siloed application teams and uncoordinated cloud deployments.

To do this, present a clear roadmap with measurable outcomes, such as improved delivery speed and security. Use metrics like DORA (Deployment Frequency, Lead Time, Mean Time to Recovery) to demonstrate how platform engineering enhances security, standardization, and efficiency. Outline the required resources (headcount, budget, tooling) and align the project with business objectives.

Step 2: Staffing the Platform Engineering Team

Once you have executive backing, the next step is to assemble your platform engineering team. This team will develop and maintain your internal developer platform (IDP), ensuring it offers the organization secure, scalable, and self-service solutions.

Successful platform engineering teams are characterized by a strong customer focus, empathy for the needs of application teams, and the ability to act as the “glue” that binds the organization together. Key roles and responsibilities within the platform team include:

Customer-facing roles: Serve as the primary point of contact for application teams, understanding their needs and advocating for their requirements within the platform.
Infrastructure expertise: Possess deep knowledge of the underlying infrastructure, whether on-premises or in the cloud, to ensure the platform is reliable and scalable.
DevOps and automation skills: Leverage infrastructure as code (IaC) tools and techniques to automate the provisioning and management of the platform.
FinOps and cost optimization expertise: Understand the organization’s financial systems and processes to ensure the platform is cost-effective and aligned with budgetary constraints.
Software development capabilities: Develop the platform’s core components, including self-service interfaces and reusable infrastructure modules, using best practices in software engineering.

By assembling a team with this diverse set of skills and a customer-centric mindset, you’ll be well-positioned to build a platform that truly meets the needs of your application teams.

Step 3: Defining the Platform Mandate

Next, clearly define the platform team’s mandate to set expectations. This document should outline the platform’s purpose, the services it provides, and how it supports internal developer portals (IDPs) and self-service capabilities.

The platform mandate should serve as a central reference point for the platform team and its customers, ensuring everyone is aligned on the team’s mission and its value. Key elements to include in the platform mandate include:

Introduction: A high-level overview of the platform team’s purpose and the organizational outcomes it aims to achieve.
Supported systems: A list of the infrastructure, services, and tools that the platform team is responsible for managing and supporting.
Support model: Details on how application teams can access platform services, including any self-service capabilities and the team’s response times for different types of requests.
Team structure: An overview of the platform team’s members and their areas of expertise, making it easy for application teams to identify the right point of contact.

A well-defined mandate aligns the team and its customers with the platform’s value and purpose.

Step 4: Implementing the Pre-Production Pipeline

With a mandate in place, begin building the core capabilities, starting with a pre-production pipeline. This pipeline standardizes how application code moves from development to deployment, emphasizing security and consistency.

Key components of the pre-production pipeline include:

Version control: Implement a centralized version control system to manage application code and infrastructure as code (IaC) configurations.
Continuous integration (CI): Automate the build and testing of application artifacts, such as container images or deployment packages.
Artifact management: Provide a secure and versioned repository for storing and distributing the application artifacts produced by the CI process.
Static analysis: Integrate tools that can scan application code and dependencies for security vulnerabilities and other issues.
Application delivery: Automate the deployment of application artifacts to pre-production environments, such as staging or testing.

By establishing a standardized pre-production pipeline, you’ll ensure that all application teams follow a consistent, secure, and efficient process for getting their code into a production-ready state.

Step 5: Embracing Infrastructure as Code

Leveraging infrastructure as code (IaC) is crucial for managing the platform’s infrastructure. It enables consistent, scalable, and secure operations by automating the provisioning of resources, monitoring, and enforcing security policies.

Key areas where IaC can be applied within the platform include:

Foundational infrastructure: Provision and manage the underlying cloud resources, such as virtual networks, storage, and compute, that form the platform’s foundation.
Runtime platforms: Deploy and configure the runtime environments where application workloads will be executed, such as Kubernetes clusters or serverless functions.
Observability and monitoring: Set up the logging, metrics, and alerting systems that provide visibility into the platform’s health and performance.
Security and compliance: Implement security controls, such as secrets management and access policies, to ensure the platform meets regulatory and organizational requirements.
Pipelines and automation: Use IaC to define and version-control the platform’s own deployment and management pipelines, ensuring consistency and repeatability.

With infrastructure as code, the platform engineering team can ensure reliable, scalable, and secure infrastructure across the organization.

Step 6: Implementing Policy as Code

Alongside your IaC efforts, it’s important to set up a robust policy-as-code framework within your platform. Policy-as-code allows you to enforce security, compliance, and operational policies programmatically. This provides governance at scale, ensuring all teams follow the same rules.

Policy as code can be applied in two key ways:

Preventative controls: Implement policies that proactively validate and reject non-compliant infrastructure changes before they are provisioned, providing fast feedback to application teams.
Detective controls: Establish policies that continuously monitor the deployed infrastructure, triggering alerts or remediation actions when deviations from the desired state are detected.

By combining IaC and policy as code with self-service provisioning, you maintain security and compliance while giving teams autonomy.

Build your internal developer platform

Give developers self-service infrastructure with reusable components, golden paths, and governance built in using Pulumi IDP.

Explore Pulumi IDP

Step 7: Evolving Towards Self-Service

As your platform matures, the ultimate goal is to empower your application teams with self-service capabilities, allowing them to provision the infrastructure and resources they need quickly and autonomously. This self-service model can take various forms, from pre-defined infrastructure modules to fully automated deployment pipelines.

When implementing self-service capabilities, keep the following platform engineering best practices in mind:

Start small and iterate: Don’t try to boil the ocean. Begin with a few well-defined use cases, such as CI/CD pipelines or database provisioning, and gradually expand the self-service offerings as your platform team gains experience and confidence.
Focus on user experience: Ensure that the self-service interfaces, whether web-based or command-line, are intuitive and easy to use. Gather feedback from application teams and continuously refine the user experience.
Maintain stability and reliability: Even with self-service, the platform team is responsible for ensuring the underlying infrastructure and services remain stable and reliable. Implement robust monitoring, alerting, and incident response processes to maintain platform health.
Embrace a product mindset: Treat the platform as a product, with the application teams as your customers. Continuously gather feedback, measure key performance indicators, and iterate on the platform’s capabilities to meet evolving needs.

By gradually transitioning towards a self-service model, you’ll empower your application teams to move faster while maintaining the necessary guardrails and controls to ensure the platform’s long-term success.

Conclusion: Embracing the Platform Engineering Journey

Building a successful platform engineering strategy involves following a clear roadmap. But remember, implementing the platform engineering strategy is a journey, not a destination. By following the steps outlined in this guide, you’ll be well on your way to building a robust, secure, scalable, and customer-centric internal developer platform (IDP) that empowers your organization to deliver software more efficiently and effectively.

With the right approach and mindset, your platform engineering efforts will become a strategic enabler for your organization’s digital transformation.

Build secure, scalable platforms with confidence—register for the Platform Engineering Course

Frequently Asked Questions

What is Platform Engineering?

There are many definitions for platform engineering, one of which is designing, building, and maintaining internal platforms that streamline software delivery by providing reusable tools, services, and workflows for developers. It focuses on scalability, efficiency, and self-service capabilities for internal teams.

For a more in-depth explanation of real-world use cases, see What is platform engineering.

Who are the Platform Customers?

The platform customers are the end users, typically internal teams (e.g., application developers and DevOps engineers), who use the platform’s tools and services to build, deploy, and manage software.

What is an Internal Developer Platform (IDP)?

A centralized platform that offers developers the tools and environments they need to build, test, and deploy applications. It abstracts away the complexity of underlying infrastructure, allowing teams to focus on coding and delivery.

What is a Platform Team?

The platform engineering team is a dedicated group of engineers responsible for managing the internal platform. They ensure the platform meets the needs of the organization, providing reliability, security, and scalability.

What is a Self-Service Infrastructure?

A platform feature that allows development teams to provision, configure, and manage infrastructure resources on demand without relying on other teams, fostering agility.

What is Infrastructure as Code (IaC)?

The process of managing and provisioning infrastructure through code, allowing infrastructure to be versioned, automated, and managed consistently across environments.

For a more in-depth explanation, see the What is Infrastructure as Code? page.

What is Policy as Code (PaC)?

Policy as Code (PaC) defines security, compliance, and operational policies in code to automate their enforcement across infrastructure and application deployments.

What is Developer Experience (DevEx)?

The overall experience developers have with using the platform, tools, and workflows provided by platform engineering. Optimizing DevEx ensures that developers can work more efficiently and effectively.

For a more in-depth explanation, read Beyond Productivity: Developer Experience is Business Critical.

Pulumi and Redis Cloud: Real-Time Data for Modern Apps

Josh Kodroff — Tue, 28 Nov 2023 00:00:00 +0000

Redis Enterprise Cloud makes apps faster by providing an in-memory database that enables teams to create a real-time data platform. In this post, you’ll learn how to enable connectivity between Redis Enterprise Cloud and your workloads using Pulumi and the Redis Cloud provider.

The full code for this example is available at https://github.com/pulumi/examples/tree/master/redis-cloud-aws-ts. The full code differs slightly from the example below in that it has some parameterization that makes the code more portable, whereas in this post we show hardcoded values for simplicity.

About Redis Enterprise Cloud

Redis Enterprise Cloud is a fully managed database as-a-service (DBaaS) provider that offers managed Redis clusters. Redis Enterprise Cloud clusters can be accessed via network peering in AWS or Google Cloud, or over the internet. The Redis Cloud provider allows you to manage Redis Enterprise Cloud resources in Pulumi programs.

Some of Redis Enterprise Cloud’s key features include:

Automated failover, backups, and scaling.
Linear scaling based on a serverless architecture, with an optimized engine that provides higher throughput and lower latencies compared to solutions based on Redis’ open-source version.
A subscription-based pricing pricing model based on consumption (as opposed to being billed hourly on provisioned hardware).

Architecture

One of the most common scenarios for using Redis Enterprise Cloud is to run a workload in a hyperscaler (like AWS or Google Cloud), and then to establish a low-latency connection between your workload and your managed Redis cluster via network peering. In this post, you’ll create an architecture for a sample AWS workload. The architecture you’ll build will include:

A Redis Enterprise Cloud database cluster.
An AWS VPC across 3 Availability Zones, with public and private subnets.
VPC Peering to enable connectivity between the Redis cluster and the VPC.

You’ll also prove the architecture works by spinning up an EC2 instance and running simple commands that target the Redis cluster.

The following diagram shows the major components of the architecture:

Project Initialization and Redis Enterprise Cloud Resources

Before you begin this tutorial, you will need to add a payment method in the Redis Cloud console.

To begin, initialize your Pulumi program in an empty directory:

pulumi new typescript

Then, add a reference to the Redis Cloud provider:

npm i @rediscloud/pulumi-rediscloud

Then, in your index.ts file, add use the getPaymentMethodOutput function to obtain your payment information, and use the resulting card ID to create a Redis Subscription:

Please note the following:

The Redis Cloud provider does not support Fixed subscriptions. You’ll need to create a Flexible subscription in order to use the provider to manage resources with Pulumi.
The snippet below uses hard-coded values for things like payment info or CIDR blocks for simplicity. In a multi-stack Pulumi program, these should be refactored to configurable values.

import * as pulumi from "@pulumi/pulumi";
import * as rediscloud from "@rediscloud/pulumi-rediscloud";

const card = rediscloud.getPaymentMethodOutput({
 cardType: "<YOUR-CARD-TYPE>" // e.g. "Visa" (capitalization matters here),
 lastFourNumbers: "1234",
});

const subscription = new rediscloud.Subscription("redis-subscription", {
 name: "pulumi-redis-aws-example",
 paymentMethod: "credit-card",
 paymentMethodId: card.id,
 cloudProvider: {
 regions: [
 {
 region: "us-east-1",
 // This CIDR block must not conflict with the CIDR block of the AWS
 // VPC to which you wish to connect:
 networkingDeploymentCidr: "10.0.0.0/24",
 // We use a single AZ here to minimize the cost incurred from running
 // this example. For a production workload, consider multiple availability
 // zones:
 multipleAvailabilityZones: false,
 preferredAvailabilityZones: ["use1-az1"],
 },
 ],
 },

 creationPlan: {
 memoryLimitInGb: 10,
 quantity: 1,
 replication: true,
 supportOssClusterApi: false,
 throughputMeasurementBy: "operations-per-second",
 throughputMeasurementValue: 20000,
 modules: ["RedisJSON"],
 },
});

Then, create a database in your subscription:

const database = new rediscloud.SubscriptionDatabase("redis-db", {
 name: "my-db",
 subscriptionId: subscription.id,
 protocol: "redis",
 memoryLimitInGb: 10,
 dataPersistence: "aof-every-1-second",
 throughputMeasurementBy: "operations-per-second",
 throughputMeasurementValue: 20000,
 replication: true,
});

AWS Networking Resources

Now you are ready to add your AWS resources. First, you’ll need to add some reference to your Pulumi program. Add the Pulumi AWS Provider, which provides basic AWS resources, and the AWSX package, which provides a convenient VPC component:

npm i @pulumi/awsx @pulumi/aws

Now you can add some AWS resources to your program. Start by adding import statements for the packages you added to our project, then create a VPC with with the AWSX package:

import * as awsx from "@pulumi/awsx";
import * as aws from "@pulumi/aws";

const vpc = new awsx.ec2.Vpc("vpc", {
 cidrBlock: "10.1.0.0/16", // Cannot conflict with the Redis CIDR block,
 natGateways: {
 strategy: "Single"
 }
});

Next, peer the Redis Cloud subscription with your AWS VPC:

const callerIdentity = aws.getCallerIdentity();

const peering = new rediscloud.SubscriptionPeering("redis-peering", {
 subscriptionId: subscription.id,
 region: region,
 awsAccountId: callerIdentity.then(x => x.accountId),
 vpcId: vpc.vpcId,
 vpcCidr: vpc.vpc.cidrBlock,
});

new aws.ec2.VpcPeeringConnectionAccepter("aws-peering-accepter", {
 vpcPeeringConnectionId: peering.awsPeeringId,
 autoAccept: true
});

To complete network connectivity between your Redis Cloud database and your AWS VPC, add routes to the private subnets in which your workloads will run that allow traffic to to through the peering connection to the Redis Cloud database’s network:

vpc.privateSubnetIds.apply(ids => {
 ids.forEach((id, index) => {
 const routeTable = aws.ec2.getRouteTableOutput({
 subnetId: id,
 });

 new aws.ec2.Route(`peering-route-${index}`, {
 routeTableId: routeTable.id,
 destinationCidrBlock: redisVpcCidr,
 vpcPeeringConnectionId: peering.awsPeeringId,
 });
 });
});

If you are unfamiliar with the call to apply(), you can learn more about why this is necessary in Inputs and Outputs. Creating resources within apply is usually not necessary, but this is one of the few cases where it is.

Now, deploy the infrastructure using the pulumi up command:

pulumi up

Creating an EC2 Instance to Test

To test the architecture, create an EC2 instance running Amazon Linux 2 and enable AWS SSM Systems Manager to allow you easy access to a shell prompt without having to worry about SSH keys. From the shell prompt, you will run the Redis CLI to prove that network connectivity is working as expected.

First, create a Security Group to allow all egress traffic. This will allow connectivity to Redis, SSM Systems Manager, and HTTP/S to update the underlying operating system and install dependencies:

const sg = new aws.ec2.SecurityGroup("instance-sg", {
 description: "Allow all egress traffic.",
 vpcId: vpc.vpcId,
 egress: [{
 cidrBlocks: ["0.0.0.0/0"],
 description: "Allow all",
 protocol: "-1",
 fromPort: 0,
 toPort: 0,
 }]
});

Next, create an EC2 instance role that grants the necessary permissions to use SSM Systems Manager:

const instanceRole = new aws.iam.Role("instance-role", {
 assumeRolePolicy: JSON.stringify({
 "Version": "2012-10-17",
 "Statement": {
 "Effect": "Allow",
 "Principal": {
 "Service": "ec2.amazonaws.com",
 },
 "Action": "sts:AssumeRole",
 },
 })
});

new aws.iam.RolePolicyAttachment("instance-role-attachment", {
 role: instanceRole.name,
 policyArn: "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore",
});

const instanceProfile = new aws.iam.InstanceProfile("instance-profile", {
 role: instanceRole.name,
});

Then, create the EC2 instance based on the latest Amazon Linux 2 AMI. Note that the userData input of the EC2 instance resource installs the Redis CLI, which is not included in the base operating system by default:

const amazonLinux2 = aws.ec2.getAmiOutput({
 mostRecent: true,
 owners: ["amazon"],
 filters: [
 { name: "name", values: ["amzn2-ami-hvm-*-x86_64-gp2"] },
 { name: "owner-alias", values: ["amazon"] },
 ]
});

const instance = new aws.ec2.Instance("instance", {
 ami: amazonLinux2.id,
 instanceType: "t3.micro",
 vpcSecurityGroupIds: [sg.id],
 subnetId: vpc.privateSubnetIds[0],
 tags: {
 Name: "pulumi-redis-cloud-tester"
 },
 iamInstanceProfile: instanceProfile.name,
 userData: `#!/bin/bash
 sudo amazon-linux-extras install redis6
 `
});

And again, deploy your stack:

pulumi up

If you’re new to Pulumi, you may notice that when you run the pulumi up command this time, Pulumi does not attempt to re-create the resources you’ve already declared. This is because Pulumi is a declarative tool: You only need to specify the desired end state of your resources, and Pulumi will figure out what steps (if any) are necessary to get from your infrastructure’s current state to your declared desired state.

Testing the Architecture

Now that you have your EC2 instance up and running, it’s time to verify that you can connect to your Redis cluster and run commands:

In the AWS Console, find the EC2 instance named pulumi-redis-cloud-tester, and connect to it via SSM Systems Manager:
In the Redis Cloud console, locate your database, and select “Connect”:
In the connection dialog, copy the connection info for the Redis CLI:
Paste the command into the Systems Manager shell prompt, which will connect you to the Redis instance. From there you can run Redis commands like INCR that will write to the Redis cluster:

Now your workloads can use your Redis Cloud cluster! You can tear down all the resources you created with the following command:

pulumi destroy

Next Steps and Conclusion

Now that you have proven that the architecture works, here are some next steps you might want to take with this codebase:

Replace the EC2 test instance with a real workload that uses Redis:
- If your workload is running on EC2, then the code in this post should get you up and running. However, if you find you need some help Pulumi Neo can be a very helpful resource. Simply enter a natural language query and select a support Pulumi language, and Pulumi AI will generate a Pulumi program that can be copy/pasted into your program and iterated on in the same Pulumi AI conversation.
- If you’re using another AWS compute platform, Pulumi also has helpful guides for creating workloads on ECS, EKS, or Lambda.
Try creating Redis Cloud clusters peered with Google Cloud.
If you’re comfortable using Pulumi and are thinking of adopting it at a team or organizational level, consider creating an organization in Pulumi Cloud and enjoying a 2-week free trial of our full set of features, no credit card required!

With Redis Cloud you can enjoy lightning-fast access to whatever data your application needs. When Redis Cloud is coupled with Pulumi, you can manage your Redis databases and all your other cloud infrastructure using a single tool, in any language. That’s powerful stuff!

A Tour of the Pulumi Equinix Provider

Josh Kodroff — Wed, 28 Jun 2023 00:00:00 +0000

Equinix recently released their self-maintained, fully-supported Pulumi provider, available in the Pulumi Registry. In this post, you’ll get an overview of the Equinix resources the provider can manage and we’ll show you how to deploy a Kubernetes cluster and associated workloads on Equinix Metal.

Join Pulumi and Equinix on September 13, 2023, for a live workshop: Deploying a Kubernetes Cluster on Equinix Metal.

Introducing the Equinix Provider

The Equinix provider can manage resources for:

Equinix Metal, which provides high-performance, bare-metal compute resources.
Equinix Fabric, which provides software-defined networking that enables low-latency connections between private networks; select service providers like Salesforce, AWS, Azure, and more; and Equinix Metal devices.
Equinix Network Edge, which enables organizations to deploy virtual network functions (VNFs) (like virtual firewall devices) in a centralized point, greatly simplifying network traffic management for multi- and hybrid cloud scenarios.

Detailed documentation for the Equinix provider can be found in the Pulumi Registry along with 130+ (at the time of writing) other providers that can be used to manage cloud and SaaS resources.

Creating a Kubernetes cluster on Equinix Metal

In order to demonstrate the power and utility of Pulumi and the Equinix provider, Equinix Labs has produced a codebase that creates a Kubernetes cluster on Equinix Metal. The codebase is available in both TypeScript and Python.

NOTE: For an overview of how Pulumi works along with a guided tour of the codebase and deploying a workload onto the Kubernetes cluster, check out Pulumi’s presentation at Equinix Demo Day 2023. (Pulumi’s presentation begins at 3:30:00 below, or click the preceding link to jump directly to Pulumi’s presentation on YouTube.):

The codebase gives an excellent example of one of Pulumi’s most compelling features: the ability to manage and orchestrate many different kinds of resources in real programming languages with a single tool. In addition to the Equinix provider which is used to manage the bare metal compute resources, the codebase also uses the following providers:

Cloud-init, to run initialization scripts that install the necessary services to run Kubernetes on the bare metal instances once they are spun up. Much of this work is accomplished via Kubeadm. For an even deeper dive on the services installed on the control plane and worker nodes respectively, see Kubernetes the Hard Way.
TLS, to manage cryptographic resources that allow nodes to join the cluster.
Command, which is used to synchronize cluster joining operations and to read the kubeconfig from the control plane.
Random, which is used to generate a token for nodes to join the control plane.

Before running the Pulumi program, ensure you’ve configured your Equinix credentials. Additionally, if you wish to customize specific aspects of the deployment, you can set the template configuration values. Once that’s done, execute the following commands to deploy the cluster:

cd nodejs
npm i
pulumi up

cd python
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt
pulumi up

After executing the commands, the process will require approximately 5 minutes to finish, resulting in a fully operational cluster. From there, you can write the cluster’s Kubeconfig to a file and set it as your default file:

pulumi stack output kubeconfig --show-secrets > kubeconfig.yml
export KUBECONFIG=kubeconfig.yml

Now you can run commands against your cluster with kubectl (or the excellent K9s if you prefer a more GUI-like experience)! For example, to show all running pods:

kubectl get pods --all-namespaces

Deploying a workload with the Pulumi Kubernetes provider

Now that your cluster is provisioned, you can deploy a workload (or any other Kubernetes or Helm resource) using the Pulumi Kubernetes provider. In order to deploy Kubernetes resources to your cluster, you need to first declare an explicit provider. The explicit provider is needed because you are creating the cluster and deploying workloads to that cluster in the same Pulumi program. (For more details on default and explicit providers, see Resource providers.) Configure the provider to use the Kubeconfig of your newly created cluster:

const k8sProvider = new k8s.Provider("k8s-provider", {
 kubeconfig: kubeconfig,
});

k8s_provider = k8s.Provider(
 "k8s-provider",
 kubeconfig=kubeconfig
)

You can then add resources to the cluster by specifying your explicit provider as a resource option:

const nginxDeployment = new k8s.apps.v1.Deployment("nginx-deployment", {
 metadata: {
 // ...
 },
 spec: {
 // ...
 },
}, { provider: k8sProvider });

deployment = k8s.apps.v1.Deployment(
 "nginx-deployment",
 metadata={
 # ...
 },
 spec={
 # ...
 },
 opts=pulumi.ResourceOptions(
 provider=k8s_provider
 )
)

Adding storage

Our cluster as configured will not be able to run stateful workloads due to the lack of a StorageClass from which PersistentVolumes can be provisioned. Popular storage options for bare-metal/edge Kubernetes clusters include Portworx, Longhorn, Rook, or (for non-production scenarios as data loss is likely if a node goes down) NFS.

Conclusion

The Pulumi Equinix provider offers developers an intuitive and efficient way to interact with Equinix resources. By combining the power of Pulumi’s infrastructure-as-code tooling and ecosystem along with the utility of Equinix’s service offerings, you can create and manage networking and bare metal compute resources using a single tool, freeing practitioners from manual configuration so they can focus on value-driving innovation.

Advanced AWS Networking: Part 2

Josh Kodroff — Thu, 20 Apr 2023 00:00:00 +0000

In this blog series, you will learn how to create a hub-and-spoke network architecture in AWS with centralized egress and traffic inspection. In this second installment, we’ll show you how to create spoke VPCs to run your workloads, verify centralized egress is working, and then add centralized traffic inspection using Pulumi, the infrastructure as code tool that enables you to manage infrastructure with real programming languages!

Introduction

In AWS Advanced Networking Part 1 you:

Learned about the benefits of a hub-and-spoke network architecture with centralized egress.
Created the shared AWS Transit Gateway resources to be shared among our spoke VPCs and inspection VPC.
Defined your inspection VPC as a Pulumi component resource.

Now, to finish the architecture, you will need to:

Create a spoke VPC as a Pulumi component resource.
Test the network architecture by verifying that centralized egress is working.
Add traffic inspection with AWS Network Firewall.

In subsequent sections we will be showing code samples. Because this is a more advanced Pulumi program, we will be focusing on the most important parts of the architecture rather than covering every single line in the codebase. The complete code for this project is available on GitHub, which is a great place to go for full context.

Spoke VPC

Like your inspection VPC, we will implement the spoke VPC as a Pulumi component resource. Component resources allow you to logically group resources to form reusable components. Because your network will contain multiple spoke VPCs (at least 2, possibly more), and the spoke VPC will contain both the basic VPC and connectivity to Transit Gateway, the spoke VPC is a natural fit for a component resource.

First, add a file spoke.py to your Pulumi program. In Python, it’s good practice to put each component resource along with its associated arguments class in a separate file from your main program as it makes for intuitive code organization:

@dataclass
class SpokeVpcArgs:
 vpc_cidr_block: str
 tgw_id: pulumi.Input[str]
 tgw_route_table_id: pulumi.Input[str]


class SpokeVpc(pulumi.ComponentResource):
 def __init__(self, name: str, args: SpokeVpcArgs, opts: pulumi.ResourceOptions = None) -> None:
 super().__init__("awsAdvancedNetworking:index:SpokeVpc", name, None, opts)

Then, create a VPC with two sets of subnets, both isolated (with no pathway to the internet). One set of subnets will hold our spoke VPC workloads, and the other will hold our Transit Gateway attachment. We enable DNS hostnames and support so that we can later use VPC endpoints to enable AWS Systems Manager access to verify that centralized egress is working as expected without requiring direct host access via SSH:

self.vpc = awsx.ec2.Vpc(
 f"{self._name}-vpc",
 awsx.ec2.VpcArgs(
 cidr_block=args.vpc_cidr_block,
 subnet_specs=[
 awsx.ec2.SubnetSpecArgs(
 name="private",
 cidr_mask=28,
 type=awsx.ec2.SubnetType.ISOLATED,
 ),
 awsx.ec2.SubnetSpecArgs(
 name="tgw",
 cidr_mask=28,
 type=awsx.ec2.SubnetType.ISOLATED,
 ),
 ],
 nat_gateways=awsx.ec2.NatGatewayConfigurationArgs(
 strategy=awsx.ec2.NatGatewayStrategy.NONE,
 ),
 enable_dns_hostnames=True,
 enable_dns_support=True,
 )
)

Then, attach the Transit Gateway to the appropriate subnets:

self.tgw_attachment = aws.ec2transitgateway.VpcAttachment(
 f"{name}-tgw-vpc-attachment",
 aws.ec2transitgateway.VpcAttachmentArgs(
 transit_gateway_id=args.tgw_id,
 subnet_ids=tgw_subnets.apply(lambda x: x.ids),
 vpc_id=self.vpc.vpc_id,
 transit_gateway_default_route_table_association=False,
 transit_gateway_default_route_table_propagation=False,
 tags={
 "Name": f"{name}",
 },
 ),
 pulumi.ResourceOptions(
 delete_before_replace=True,
 depends_on=[self.vpc],
 parent=self,
 )
)

Add VPC endpoints to allow Systems Manager access to open an SSH connection from the AWS console without having to allow network access to the hosts. Because Pulumi uses real programming languages, you can use a Python for loop to create these resources with fewer lines of code using familiar syntax:

for service in ["ec2messages", "ssmmessages", "ssm"]:
 aws.ec2.VpcEndpoint(
 f"{self._name}-endpoint-{service}",
 aws.ec2.VpcEndpointArgs(
 vpc_id=self.vpc.vpc_id,
 service_name=f"com.amazonaws.{aws.config.region}.{service}",
 private_dns_enabled=True,
 security_group_ids=[vpc_endpoint_sg.id],
 vpc_endpoint_type="Interface",
 tags={
 "Name": f"{self._name}-{service}"
 },
 subnet_ids=subnet_ids
 )
 )

Add routes to the spoke VPC’s Transit Gateway route table that direct all non-local traffic to the Transit Gateway:

for subnet_id in private_subnet_ids:
 route_table = aws.ec2.get_route_table(
 subnet_id=subnet_id,
 )

 aws.ec2.Route(
 f"spoke{self._name}-tgw-route-{subnet_id}",
 aws.ec2.RouteArgs(
 route_table_id=route_table.id,
 destination_cidr_block="0.0.0.0/0",
 transit_gateway_id=self._args.tgw_id,
 ),
 pulumi.ResourceOptions(
 depends_on=[self.tgw_attachment],
 parent=self,
 ),
 )

Finally, add some outputs for the component. You’ll use these to place workloads in the proper subnets:

self.register_outputs({
 "vpc": self.vpc,
 "workload_subnet_ids": self.workload_subnet_ids
})

You can instantiate a spoke VPC like the following in __main__.py:

spoke1_vpc = SpokeVpc(
 "spoke1",
 SpokeVpcArgs(
 vpc_cidr_block="10.0.0.0/16",
 tgw_id=tgw.id,
 tgw_route_table_id=spoke_tgw_route_table.id,
 ),
)

Subsequent spoke VPCs should use a separate CIDR block:

spoke2_vpc = SpokeVpc(
 "spoke2",
 SpokeVpcArgs(
 vpc_cidr_block="10.1.0.0/16",
 tgw_id=tgw.id,
 tgw_route_table_id=spoke_tgw_route_table.id,
 ),
)

Now that you have your spoke VPCs added, it’s time to test that the architecture is working as expected. To do this, we’ll create a sample workload and deploy it to one of our spoke VPCs.

Creating a Sample Spoke VPC Workload

To test that internet traffic from your spoke VPC is going over the inspection VPC’s NAT gateway, you’ll add a workload (i.e., an EC2 instance running Amazon Linux) to your spoke VPC and verify that its public IP is that of the Elastic IP attached to the NAT gateway in the inspection VPC. Like the spoke VPC workload, the spoke VPC itself is a natural fit for a Pulumi component resource.

Your component should go in its own file spoke_workload.py, and will begin much like the spoke VPC component:

@dataclass
class SpokeWorkloadArgs:
 spoke_vpc_id: pulumi.Input[str]
 spoke_instance_subnet_id: str


class SpokeWorkload(pulumi.ComponentResource):
 def __init__(self, name: str, args: SpokeWorkloadArgs, opts: pulumi.ResourceOptions = None) -> None:
 super().__init__("awsAdvancedNetworking:index:SpokeWorkload", name, None, opts)

Start by creating a security group to allow outbound traffic. Note that this security group’s egress rules are overly permissive, but this is ok because the spoke workload is an ephemeral resource that only exists to verify network connectivity. In order for SSM access to work though, you must at least enable port 443:

sg = aws.ec2.SecurityGroup(
 f"{name}-instance-sg",
 aws.ec2.SecurityGroupArgs(
 description="Allow all outbound traffic",
 vpc_id=args.spoke_vpc_id,
 egress=[
 aws.ec2.SecurityGroupEgressArgs(
 cidr_blocks=["0.0.0.0/0"],
 description="Allow everything",
 protocol="-1",
 from_port=0,
 to_port=0
 ),
 ]
 ),
 opts=pulumi.ResourceOptions(
 parent=self
 ),
)

In addition to the security group which provides the necessary network access, you will also need to assign an EC2 instance role in order to complete the prerequisites for Systems Manager access to an EC2 instance which you’ll create shortly:

ec2_role = aws.iam.Role(
 f"{name}-instance-role",
 aws.iam.RoleArgs(
 assume_role_policy=json.dumps({
 "Version": "2012-10-17",
 "Statement": {
 "Effect": "Allow",
 "Principal": {
 "Service": "ec2.amazonaws.com",
 },
 "Action": "sts:AssumeRole",
 },
 })
 )
)

aws.iam.RolePolicyAttachment(
 f"{name}-role-policy-attachment",
 aws.iam.RolePolicyAttachmentArgs(
 role=ec2_role.name,
 policy_arn="arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
 )
)

instance_profile = aws.iam.InstanceProfile(
 f"{name}-instance-profile",
 aws.iam.InstanceProfileArgs(
 role=ec2_role.name,
 )
)

Finally, create an EC2 instance with the instance profile you created:

aws.ec2.Instance(
 f"{name}-instance",
 aws.ec2.InstanceArgs(
 ami=amazon_linux_2.id,
 instance_type="t3.micro",
 vpc_security_group_ids=[sg.id],
 subnet_id=args.spoke_instance_subnet_id,
 tags={
 "Name": f"{name}-instance",
 },
 iam_instance_profile=instance_profile.name,
 ),
 opts=pulumi.ResourceOptions(
 parent=self
 ),
)

Instantiate the workload in __main__.py:

spoke1_workload = SpokeWorkload(
 "spoke1",
 SpokeWorkloadArgs(
 spoke_instance_subnet_id=spoke1_vpc.workload_subnet_ids[0],
 spoke_vpc_id=spoke1_vpc.vpc.vpc_id,
 )
)

And deploy:

pulumi up

Now you have a workload with which you can verify that centralized egress is working!

Testing Centralized Egress

You can now test that centralized egress is working as expected. First, log into the AWS console and go the EC2 service homepage. Find your instance in the list of EC2 instances:

First, take note of the NAT gateway IP address which you included as a Pulumi stack output:

Then, in the AWS console, locate an instance running in a spoke VPC which you created as part of the workload component resource, and select Connect from the drop-down menu:

Connect to the instance to start an SSH session (note that the IAM principal must have permissions to perform SSM Systems Manager actions in order to connect):

Finally, verify your public IP address from the host in the spoke VPC:

If your public IP address for the instance in the spoke VPC matches the public IP address of the NAT gateway in the inspection VPC, you’ve successfully verified that centralized egress is working!

Adding AWS Network Firewall

AWS Network Firewall incurs significant costs (hundreds of USD per month). Be sure to tear down AWS Network Firewall and all other resources in this architecture with the pulumi destroy command to avoid incurring unwanted AWS charges.

Now that you’ve verified centralized egress is working, you can add centralized traffic inspection with AWS Network Firewall. In a new file called firewall_rules.py, add a rule to drop all traffic not destined for amazon.com:

allow_amazon = aws.networkfirewall.RuleGroup(
 "allow-amazon",
 aws.networkfirewall.RuleGroupArgs(
 capacity=100,
 name="allow-amazon",
 type="STATEFUL",
 rule_group=aws.networkfirewall.RuleGroupRuleGroupArgs(
 rules_source=aws.networkfirewall.RuleGroupRuleGroupRulesSourceArgs(
 rules_string='pass tcp any any <> $EXTERNAL_NET 443 (msg:"Allowing TCP in port 443"; flow:not_established; sid:892123; rev:1;)\n' +
 'pass tls any any -> $EXTERNAL_NET 443 (tls.sni; dotprefix; content:".amazon.com"; endswith; msg:"Allowing .amazon.com HTTPS requests"; sid:892125; rev:1;)'
 ),
 stateful_rule_options={
 "rule_order": "STRICT_ORDER",
 },
 )
 )
)

And add the rule to a firewall policy:

policy = aws.networkfirewall.FirewallPolicy(
 "firewall-policy",
 aws.networkfirewall.FirewallPolicyArgs(
 firewall_policy=aws.networkfirewall.FirewallPolicyFirewallPolicyArgs(
 stateless_default_actions=["aws:forward_to_sfe"],
 stateless_fragment_default_actions=["aws:forward_to_sfe"],
 stateful_default_actions=[
 "aws:drop_strict", "aws:alert_strict"],
 stateful_engine_options={
 "rule_order": "STRICT_ORDER"
 },
 stateless_rule_group_references=[{
 "priority": 10,
 "resource_arn": drop_remote.arn
 }],
 stateful_rule_group_references=[
 {
 "priority": 10,
 "resource_arn": allow_icmp.arn,
 },
 {
 "priority": 20,
 "resource_arn": allow_amazon.arn,
 },
 ]
 )
 )
)

Then, in inspection_vpc.py, you’ll need to add some additional network resources to create the firewall and route your egress traffic through the firewall. First create some subnets to contain AWS Network Firewall:

for i, inspection_subnet in enumerate(inspection_subnets):
 resource_name = f"{self.name}-inspection-{i+1}"
 subnet = aws.ec2.Subnet(
 resource_name,
 aws.ec2.SubnetArgs(
 vpc_id=self.vpc.vpc_id,
 availability_zone=inspection_subnet["az"],
 cidr_block=inspection_subnet["cidr"],
 tags={
 "Name": resource_name
 }
 ),
 opts=pulumi.ResourceOptions(
 parent=self,
 delete_before_replace=True,
 ),
 )

After creating route tables and associating with your subnets, create routes that send internal traffic to the TGW and external traffic to the NAT gateway:

aws.ec2.Route(
 f"{self.name}-insp-supernet-to-tgw-{i+1}",
 aws.ec2.RouteArgs(
 route_table_id=route_table.id,
 destination_cidr_block=self.args.supernet_cidr_block,
 transit_gateway_id=self.args.tgw_id
 ),
 opts=pulumi.ResourceOptions(
 parent=route_table,
 ),
)

aws.ec2.Route(
 f"{self.name}-insp-default-to-nat-{i+1}",
 aws.ec2.RouteArgs(
 route_table_id=route_table.id,
 destination_cidr_block="0.0.0.0/0",
 nat_gateway_id=self.nat_gateway.id
 ),
 opts=pulumi.ResourceOptions(
 parent=route_table,
 ),
)

Then create the AWS Network Firewall itself and deploy it to the subnets you created. AWS Network Firewall creates a VPC endpoint in each subnet to which it’s attached:

self.firewall = aws.networkfirewall.Firewall(
 f"{self.name}-firewall",
 aws.networkfirewall.FirewallArgs(
 vpc_id=self.vpc.vpc_id,
 firewall_policy_arn=self.args.firewall_policy_arn,
 subnet_mappings=subnet_mappings,
 ),
 opts=pulumi.ResourceOptions(
 parent=self,
 ),
)

You’ll also need to add some routes. In the public subnets of the inspection VPC, you’ll need to route all traffic bound for the spoke VPCs (i.e., the return traffic from the internet) through AWS Network Firewall’s generated VPC endpoints:

for subnet_id in public_subnet_ids:
 # ...

 aws.ec2.Route(
 f"{self.name}-{subnet_id}-to-firewall",
 aws.ec2.RouteArgs(
 route_table_id=route_table.id,
 destination_cidr_block=self.args.supernet_cidr_block,
 vpc_endpoint_id=subnet_attachments[0]["endpoint_id"],
 ),
 pulumi.ResourceOptions(
 parent=self,
 ),
 )

In the inspection VPC subnets with the Transit Gateway attachment, you’ll need to route all traffic bound for the internet through AWS Network Firewall. Previously, these routes went directly to the NAT gateway:

for subnet_id in tgw_subnet_ids:
 # ...

 aws.ec2.Route(
 f"{self.name}-{subnet_id}-to-firewall",
 aws.ec2.RouteArgs(
 route_table_id=route_table.id,
 destination_cidr_block="0.0.0.0/0",
 vpc_endpoint_id=subnet_attachments[0]["endpoint_id"],
 ),
 pulumi.ResourceOptions(
 parent=self,
 ),
 )

In order to test that our firewall rules are in effect, connect to one of your spoke EC2 instances like before. Traffic to amazon.com should be allowed:

But traffic to other domains should be blocked:

Conclusion

The hub-and-spoke architecture gives us several key advantages when our organization has multiple VPCs: cost savings via centralized egress and the ability to provide traffic inspection and apply network policies in a single place. While this architecture can be difficult to implement due to the resources needed, Pulumi’s approach of enabling practitioners to use real programming languages makes it easier to define complex, real-world infrastructure.

If you enjoyed this content, sign up for an upcoming workshop to learn more about how Pulumi can help you tame the complexity of the modern cloud, and join our community in the Pulumi Community Slack!

AWS Lambda Response Streaming with Pulumi

Isaac Harris — Thu, 06 Apr 2023 00:00:00 +0000

Since its introduction in 2014, AWS Lambda has rapidly expanded its capabilities from simple “functions in the cloud” at launch to a comprehensive serverless platform with support for containerized functions and public per-function URL endpoints.

As serverless applications have increased in sophistication, developers have used functions-as-a-service as a first-class tool in their microservices strategy. As organizations increasingly look to break up their monolithic applications into services, adoption of AWS Lambda has not been a viable option for applications that return payloads larger than the 6 MiB Lambda service limit. In addition to the payload response limit, AWS Lambda has been limited to returning the entire response once it has been fully buffered in memory within the Lambda service. This historical limitation meant that end users would not see any response until the entire response was available. Both of these limitations made Lambda difficult or impossible to use for data-intensive or latency-sensitive use cases.

Introducing Response Streaming

Today, AWS announced Lambda Response Streaming: a new feature that enables response payloads beyond the 6MiB limit, supports binary content in responses and reduces response times for latency-sensitive applications. Response Streaming reduces the time to first byte (TTFB) by enabling your functions to send partial responses to the client as soon as they are ready instead of waiting for a fully-generated and buffered response. This reduces multi-second TTFB to milliseconds, and, you can stream payloads that are gigabytes in size with a soft 20MiB limit for streamed payloads. The maximum throughput for a streamed response is 2MiB/s.

Enabling Response Streaming requires a new Lambda function handler signature that provides a stream object that the function can write to. When your function writes data to this stream, it is sent immediately to the client. To help clients understand the contents of the stream, you can optionally set the Content-Type header for a response.

This feature supports NodeJS v14.x, v16.x and v18.x at launch and comes with an updated billing model that charges for bytes processed and streamed from your Lambda Function.

Authoring a Lambda Function for Response Streaming

For an example of how response streaming works in practice, we’ve built a Lambda function that will stream the Gettysburg Address one sentence at a time with a one second pause between sentences. For brevity, each sentence is abbreviated below (but the code in GitHub, which demonstrates the same function with and without streaming, contains the full text of the speech):

const handleInternal = async (_event, responseStream, _context) => {
 responseStream.setContentType("text/plain");

 const sentences = [
 "Fourscore and seven years ago...",
 "Now we are engaged in a great civil war...",
 "We have come to dedicate...",
 "It is altogether fitting...",
 "But, in a larger sense...",
 "The brave men...",
 "The world will little note...",
 "It is for us the living...",
 ]

 const timer = ms => new Promise( res => setTimeout(res, ms));

 for(let i = 0; i < sentences.length; i++) {
 responseStream.write(`${sentences[i]}\n`);
 await timer(1000);
 }

 responseStream.end();
}

exports.handler = awslambda.streamifyResponse(handleInternal);

The notable changes to the Lambda function code that support response streaming are:

The function must be wrapped in the awslambda.streamifyResponse() middleware. (The streamifyResponse() function is defined in the Node runtime that’s supplied by the Lambda service.)
Handler methods require an additional parameter to the usual (event, context) signature to become (event, responseStream, context).
The responseStream parameter implements Node’s Writable interface to give it a familiar API with other streams that Node developers may be familiar with.
Responses to the client must be written via responseStream.write(), the method that writes to the stream.
The function must call responseStream.end() to signal that no more data is expected.
Note that functions authored for streaming can still be seamlessly invoked with buffered responses.

Streaming Performance Considerations

For this simple text-based example, there is little need to worry about overall performance. However, if the response is large or the client is expected to perform a lot of processing on the response data, AWS suggests using Node’s pipeline() in place of the built-in write() method in case data is produced faster than it can be consumed by the client. pipeline() effectively helps you to pipe between streams, forward errors and properly clean up – providing a callback when the pipeline is complete. You can learn more about this approach in Backpressuring in streams in the Node docs.

Packaging and deploying your Lambda Function for streaming with Pulumi

We can package our Lambda function for streaming with just a few dozen lines of code using Pulumi. First, we’ll need to import some libraries that will allow us to package our Lambda and create the necessary resources in AWS. For this exercise, we’ll be using both the AWS Cloud Control provider and the AWS provider. The AWS Cloud Control provider works with the AWS Cloud Control API to enable same-day support for new AWS features like Lambda response streaming.

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as awsNative from "@pulumi/aws-native";

Next, we’ll create an IAM role for our Lambda function to assume that has the standard required permissions for a Lambda function:

const role = new aws.iam.Role("role", {
 assumeRolePolicy: JSON.stringify({
 "Version": "2012-10-17",
 "Statement": [{
 "Effect": "Allow",
 "Principal": {
 "Service": "lambda.amazonaws.com",
 },
 "Action": "sts:AssumeRole",
 }],
 }),
});

new aws.iam.RolePolicyAttachment("role-policy-attachment", {
 role: role.name,
 policyArn: "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole",
});

Streamed responses are not currently supported with API Gateway nor with Application Load Balancers, so they must be invoked via Lambda Function URLs or invoked and streamed through the InvokeWithResponseStream API using the AWS SDK. Let’s create our function, add a function URL, and the necessary permissions to allow unauthenticated access to the function URL. Note that we set the invokeMode input of the function URL to RESPONSE_STREAM. This is what allows us to invoke the function with streaming enabled:

const streamingFunc = new aws.lambda.Function("streaming-func", {
 code: new pulumi.asset.FileArchive("../../lambda"),
 role: role.arn,
 handler: "index.handler",
 runtime: "nodejs14.x",
 timeout: 30,
});

new aws.lambda.Permission("streaming-permission", {
 action: "lambda:InvokeFunctionUrl",
 "function": streamingFunc.arn,
 principal: "*",
 functionUrlAuthType: "NONE",
});

const streamingUrl = new awsNative.lambda.Url("streaming-url", {
 authType: "NONE",
 targetFunctionArn: streamingFunc.arn,
 invokeMode: "RESPONSE_STREAM",
});

Finally, we’ll add a stack export so we can access the generated function URL from outside of our Pulumi program:

exports.streamingUrl = streamingUrl.functionUrl;

To deploy our resources, we run pulumi up -y:

$ pulumi up -y
Previewing update (dev)

View Live: https://app.pulumi.com/jkodrofftest/aws-lambda-streaming/dev/previews/d2e04257-222e-4704-b163-98e6bf4ff704

 Type Name Plan Info
 + pulumi:pulumi:Stack aws-lambda-streaming-dev create
 + ├─ aws:iam:Role role create
 + ├─ aws:iam:RolePolicyAttachment role-policy-attachment create
 + ├─ aws:lambda:Function streaming-func create
 + ├─ aws:lambda:Permission streaming-permission create
 + └─ aws-native:lambda:Url streaming-url create


Outputs:
 streamingUrl: output<string>


Updating (dev)

View Live: https://app.pulumi.com/jkodrofftest/aws-lambda-streaming/dev/updates/25

 Type Name Status Info
 + pulumi:pulumi:Stack aws-lambda-streaming-dev creating (43s)...
 + ├─ aws:iam:Role role created (1s)
 + ├─ aws:iam:RolePolicyAttachment role-policy-attachment created (0.33s)
 + ├─ aws:lambda:Function streaming-func created (22s)
 + ├─ aws-native:lambda:Url streaming-url created (2s)
 + └─ aws:lambda:Permission streaming-permission created (0.58s)

Outputs:
 streamingUrl: "https://s5a3pi4zzsufh5cgaigdxq3wva0psaib.lambda-url.eu-west-1.on.aws/"

Resources:
 + 6 created

Duration: 46s

Reading a Response Stream

Now that we’ve deployed our streaming function, we can read a response stream from the deployed Lambda URL. Per AWS, we can use any client that supports HTTP/1.1 chunked transfer coding. In cURL, streamed responses can be read via the -N flag. We can also add output that displays our time to connect, time to first byte (TTFB), and the total time to return the response:

curl -N -w "Connect: %{time_connect} TTFB: %{time_starttransfer} Total time: %{time_total} \n" $(pulumi stack output streamingUrl)

When successful, your output will look like this:

This example visually shows the advantage of response streaming because the first message appears shortly (approximately 1 second) after invocation. If we disable streaming, the fully buffered response takes several seconds to arrive. You can see the buffered response example in action. This is precisely why TFFB is important: it’s the difference between an app or service that appears to hang while awaiting a response and one that feels much more responsive to an end user.

Wrapping up

AWS Lambda Response Streaming continues the Lambda team’s steady drumbeat of innovation that adds new capabilities and use cases to serverless functions. Grab the sample code and give the feature a try. Then, let us know what you think in the Pulumi Community Slack.

Advanced AWS Networking: Part 1

Josh Kodroff — Wed, 05 Apr 2023 00:00:00 +0000

In this blog series you will learn how to create a hub-and-spoke network architecture in AWS with centralized egress and traffic inspection. In this first installment, we’ll talk about the benefits of this architecture and begin to lay out some of its parts in Python with Pulumi, the infrastructure as code tool that enables you to manage infrastructure with real programming languages!

The complete code for this project is available on GitHub, which is a great place to go for full context.

What is a hub-and-spoke network?

A hub-and-spoke network is a common architecture for creating a network topology that provides isolation and security for your workloads. The hub-and-spoke architecture you’ll be creating on AWS has three main components: an inspection VPC, AWS Transit Gateway, and a series of spoke VPCs:

The inspection VPC provides centralized egress. It is the only VPC that has a route to the internet, so all other VPCs in the architecture must route their traffic through the inspection VPC. In part II of this blog post, you’ll add AWS Firewall to the inspection VPC to allow all traffic to the internet to be governed by a single set of policies.
Network connectivity between VPCs is accomplished via AWS Transit Gateway. The Transit Gateway maintains a central routing table that is used to route traffic from the spoke VPCs to the internet. We also need to maintain routes so that return traffic from the internet can be routed back to the correct spoke VPC.
The spoke VPCs are where we run our application workloads. They are isolated from each other and cannot communicate with each other unless we explicitly allow a network path. They will be able to communicate with the internet by default, but only through the inspection VPC’s NAT gateways.

A fully fleshed-out version of the architecture, with multiple spoke VPCs follows:

We could also enable communication between the spoke VPCs by adding routes to the Transit Gateway routing table. Traffic between spoke VPCs would not be desirable for security reasons if those VPCs represent different environments of the same workload (e.g., QA, UAT, and production). Traffic between spoke VPCs would potentially be desirable if those VPCs represent multiple workloads in the same environment, for example, our production online store and our production ERP software.

For simplicity, we will not be enabling routes between spoke VPCs in this architecture.

For a full discussion of all the considerations of building multi-VPC networks on AWS, see the excellent AWS whitepaper Building a Scalable and Secure Multi-VPC AWS Network Infrastructure.

Why use a hub-and-spoke network with centralized egress?

There’s several advantages to this architecture:

Cost savings: We save costs by allowing all of our spoke VPCs to share a single set of highly-available NAT gateways in the inspection VPC. There are costs to sending our traffic over AWS Transit Gateway, but in most scenarios Transit Gateway costs are significantly less than the cost of running NAT gateways in each spoke VPC.
Stable public IP addresses: Because all of our egress traffic to the internet originate from the same set of stable, public elastic IP addresses (attached to the NAT gateways in the inspection VPC), we can more easily interface with systems that use high-trust, network-based security models (i.e., IP allowlists). Network administrators for the systems we want to interface with only need allow a limited number of IPs to establish connectivity once and up-front, as opposed to having to allow new IPs for every VPC we create. These high-trust systems are often based in the data center and may have small and infrequent change windows, so only having to go through the change management process once, not for every spoke VPC we add, is a big win.
Centralized traffic inspection: When all traffic to the internet flows through our inspection VPC, we only need to apply a firewall and associated rules in a single place. This way, we can easily ensure that all network traffic to the internet complies with our organization’s security policies. (In the part of this post, we’ll add in AWS Firewall to our inspection VPC to add centralized traffic inspection capabilities to our architecture.)

Creating the first Transit Gateway resources

After initializing your Pulumi project and adding the necessary references to the AWS and AWSX, you can add the resources relating to your Transit Gateway.

You’ll need to configure a CIDR block for your network’s supernet. All spoke VPCs in this codebase must fall under the supernet CIDR block. You will use the supernet CIDR to create a route in the inspection VPC that will route traffic returning from the internet to the Transit Gateway. The Transit Gateway will, in turn, route the traffic back to the specific originating spoke VPC.

You can get the supernet CIDR from the Pulumi configuration. If the configuration value is not set, you can default to a CIDR block of 10.0.0.0/8:

config = pulumi.Config()
supernet_cidr = config.get("supernet-cidr") or "10.0.0.0/8"

Now you can add some of your initial Transit Gateway resources. You’ll start with the Transit Gateway itself along with some route tables for your inspection and spoke VPCs, respectively. You will add routes to this route table later to allow traffic from our spoke VPCs to the internet, as well as routes from the internet back to the originating spoke VPC:

tgw = aws.ec2transitgateway.TransitGateway(
 "tgw",
 aws.ec2transitgateway.TransitGatewayArgs(
 description=f"Transit Gateway - {project}",
 default_route_table_association="disable",
 default_route_table_propagation="disable",
 tags={
 "Name": "Pulumi"
 }
 )
)

inspection_tgw_route_table = aws.ec2transitgateway.RouteTable(
 "post-inspection-tgw-route-table",
 aws.ec2transitgateway.RouteTableArgs(
 transit_gateway_id=tgw.id,
 tags={
 "Name": "post-inspection",
 }
 ),
 # Adding the TGW as the parent makes the output of `pulumi up` a little
 # easier to understand as it groups these resources visually under the TGW
 # on which they depend.
 opts=pulumi.ResourceOptions(
 parent=tgw,
 ),
)

spoke_tgw_route_table = aws.ec2transitgateway.RouteTable(
 "spoke-tgw-route-table",
 aws.ec2transitgateway.RouteTableArgs(
 transit_gateway_id=tgw.id,
 tags={
 "Name": "spoke-tgw",
 }
 ),
 opts=pulumi.ResourceOptions(
 parent=tgw,
 ),
)

Creating the inspection VPC

The code samples in this section are not exhaustive. That is, they do not contain every line of code required so that we can focus on the most illustrative portions. See the GitHub repo for full context.

Now that you’ve added some foundational Transit Gateway resources, you can create your inspection VPC.

The inspection VPC is implemented as a Pulumi component resource. While we do not need to create multiple inspection VPCs in this codebase, implementing the inspection VPC as a component resource enhances the readability of both the code and the output by grouping all resources that belong to the inspection VPC in the output of pulumi up. You can also improve the component’s readability by placing its code in a separate file (in this case, inspection.py).

When you create a component resource in Python, it’s helpful to create a class that contains the arguments to the component resource. This allows you to pass the arguments to the component resource as a single object, which makes the code easier to read and understand. You can create a terse and easily readable arguments class in Python with the @dataclass decorator:

@dataclass
class InspectionVpcArgs:
 supernet_cidr_block: str
 vpc_cidr_block: str
 tgw_id: pulumi.Input[str]
 spoke_tgw_route_table_id: pulumi.Input[str]
 inspection_tgw_route_table_id: pulumi.Input[str]
 firewall_policy_arn: pulumi.Input[str]

Then, you can create the component resource class itself. The component resource class must inherit from pulumi.ComponentResource. The constructor for the component resource class must call the constructor of the base class, passing in the resource type (a string that uniquely identifies the component type, in the form of {project}:{module}:{component_name}), the resource name, and the resource options:

class InspectionVpc(pulumi.ComponentResource):
 def __init__(self, name: str, args: InspectionVpcArgs, opts: pulumi.ResourceOptions = None) -> None:
 super().__init__("awsAdvancedNetworking:index:InspectionVpc", name, None, opts)

You’ll start by creating a VPC with the AWSX VPC component. We’ll leverage the sensible defaults the component provides to create a VPC across 3 availability zones for high availability. You can either create a single NAT gateway in our inspection VPC to save costs, or create 3 NAT gateways, one in each availability zone, to provide high availability. See the code for details.

After creating the VPC, you’ll then need to attach the inspection VPC to the Transit Gateway. This is how the inspection VPC will be able to utilize the Transit Gateway and accept traffic from the spoke VPCs:

self.tgw_attachment = aws.ec2transitgateway.VpcAttachment(
 f"{name}-tgw-vpc-attachment",
 aws.ec2transitgateway.VpcAttachmentArgs(
 transit_gateway_id=self.args.tgw_id,
 subnet_ids=self.vpc.isolated_subnet_ids,
 vpc_id=self.vpc.vpc_id,
 transit_gateway_default_route_table_association=False,
 transit_gateway_default_route_table_propagation=False,
 appliance_mode_support="enable",
 tags={
 "Name": f"{name}",
 },
 ),
 # We can only have one attachment per VPC, so we need to tell Pulumi
 # explicitly to delete the old one before creating a new one:
 pulumi.ResourceOptions(
 delete_before_replace=True,
 depends_on=[self.vpc],
 parent=self,
 )
)

Then add a route to the Transit Gateway route table for the spoke VPCs that routes all traffic outside of the VPC to the inspection VPC (or to be precise, the Transit Gateway attachment in the inspection VPC). This route only needs to be added once as it will apply to all spoke VPCs, so we add it here in the inspection VPC since that’s where we create the Transit Gateway attachment:

aws.ec2transitgateway.Route(
 f"{name}-default-spoke-to-inspection",
 aws.ec2transitgateway.RouteArgs(
 destination_cidr_block="0.0.0.0/0",
 transit_gateway_attachment_id=self.tgw_attachment.id,
 transit_gateway_route_table_id=args.spoke_tgw_route_table_id,
 ),
 opts=pulumi.ResourceOptions(
 parent=self,
 )
)

Next, add routes within the inspection VPC’s isolated subnets that route traffic to the internet through the NAT gateway. The Transit Gateway:

for subnet_id in isolated_subnet_ids:
 route_table = aws.ec2.get_route_table(
 subnet_id=subnet_id
 )

 aws.ec2.Route(
 f"{self.name}-route-{subnet_id}-to-tgw",
 aws.ec2.RouteArgs(
 route_table_id=route_table.id,
 destination_cidr_block="0.0.0.0/0",
 nat_gateway_id=self.nat_gateway.id,
 ),
 pulumi.ResourceOptions(
 parent=self,
 ),
 )

Finally, you’ll need to add routes within the inspection VPC’s public subnets (where the NAT gateways live) that route return traffic from the internet back through the Transit Gateway. This is how the inspection VPC will be able to route traffic to the spoke VPCs:

for subnet_id in public_subnet_ids:
 route_table = aws.ec2.get_route_table(
 subnet_id=subnet_id
 )

 aws.ec2.Route(
 f"{self.name}-route-{subnet_id}-to-tgw",
 aws.ec2.RouteArgs(
 route_table_id=route_table.id,
 destination_cidr_block=self.args.supernet_cidr_block,
 transit_gateway_id=self.args.tgw_id,
 ),
 pulumi.ResourceOptions(
 depends_on=[self.tgw_attachment],
 parent=self,
 ),
 )

You need to create both sets of these routes within an apply because of the need to access the raw values of each of the subnet ids. Apply can be conceptually difficult for beginners in Pulumi to grasp, but it essentially means “once you know all of the values that needed, pass them on to this function”.

Final steps

Now that you’ve created our inspection VPC component, you can reference it in the main project file:

from inspection import InspectionVpc, InspectionVpcArgs

and you can create an instance of the component:

insp_vpc = InspectionVpc(
 "inspection",
 InspectionVpcArgs(
 supernet_cidr_block=supernet_cidr,
 vpc_cidr_block="10.129.0.0/24",
 tgw_id=tgw.id,
 inspection_tgw_route_table_id=inspection_tgw_route_table.id,
 spoke_tgw_route_table_id=spoke_tgw_route_table.id,
 firewall_policy_arn=None, # We will add this in the next post
 )
)

At this point, we do not have have a fully functional architecture - we still need to create the spoke VPCs - but if we want to get some feedback on our architecture, we can run pulumi up and see what happens. We should see that Pulumi will create the VPC, the Transit Gateway attachment, and the routes.

In the next post, we’ll create our spoke VPCs and verify connectivity, then add centralized inspection of network traffic going to the internet. Stay tuned!

Google Cloud: Bulk Importing Resources into Pulumi

Josh Kodroff — Thu, 16 Mar 2023 00:00:00 +0000

Point and click in the console is great when you’re first starting out learning a new cloud or managed service, but it quickly becomes a hindrance when cloud infrastructure is widely adopted by an organization. The point at which the term “widely adopted” becomes applicable to your situation differs, but at some point in their careers, many infrastructure and platform engineers are faced with situations where a large number of critical infrastructure resources were created through “click ops” with no ability to track changes, reproduce environments consistently, and so on. When this happens (and it will probably happen to many of you), it’s time to import those resources into infrastructure as code.

Fortunately, Pulumi has one of the smoothest and most powerful import processes of any IaC tool. In this post, we’re going to show you how to automate the bulk importation of Google Cloud resources into Pulumi! This approach will also work on resources that were created by another IaC tool.

In a previous post titled Automating Pulumi Import with Manually Created Resources, we covered the details of how pulumi import works and demonstrated a workflow of bulk importing resources from AWS. We won’t repeat the details of how pulumi import works here, so please refer to “Automating Pulumi Import with Manually Created Resources” for those details.

Writing an account scraper

In our AWS account scraper, we took the approach of querying each resource type using boto3, the AWS Python SDK. In contrast, Google Cloud has a tool called Config Connector that allows us to query all resources in a Google Cloud project with a single command, and without needing to write the code to query each resource type via the Google Cloud SDK.

Our process for importing from Google Cloud has the following steps:

Query resources using Config Connector and output them to a YAML file.
In a Python script, take the Config Connector YAML output and transform it into JSON suitable for a bulk pulumi import operation.
Run the pulumi import command and place the generated code in our Pulumi program.

Config Connector

Config Connector is designed to manage Google Cloud resources represented as Kubernetes Custom Resource Definitions (CRDs). We will not be using this capability of Config Connector, but if you are interested in managing Pulumi stacks as CRDs in a GitOps workflow, check out the Pulumi Kubernetes Operator. Instead, we’ll use Config Connector’s bulk export capability to query our Google Cloud environment for resources and output them as Kubernetes manifests, then transform those Kubernetes manifests into JSON suitable for a bulk pulumi import operation via a script written in Python.

First, we need to install the config-connector CLI per the instructions in the Config Connector docs. Then, to generate our Kubernetes manifests, we run the following command:

config-connector bulk-export --project <YOUR PROJECT ID> --output bulk-export.yaml --iam-format policymember --on-error continue

This command will generate a YAML file with a series of Kubernetes manifests like the following:

---
apiVersion: artifactregistry.cnrm.cloud.google.com/v1beta1
kind: ArtifactRegistryRepository
metadata:
 annotations:
 cnrm.cloud.google.com/project-id: my-google-cloud-project
 labels:
 goog-managed-by: cloudfunctions
 name: gcf-artifacts
spec:
 description: This repository is created and used by Cloud Functions for storing
 function docker images.
 format: DOCKER
 location: europe-west1
 resourceID: gcf-artifacts
---
apiVersion: compute.cnrm.cloud.google.com/v1beta1
kind: ComputeDisk
metadata:
 labels:
 goog-gke-volume: ""
 name: gke-helloworld-2a71d4f-pvc-1e6aa931-d742-437e-a52b-413999ace2be
spec:
 description: '{"kubernetes.io/created-for/pv/name":"pvc-1e6aa931-d742-437e-a52b-413999ace2be","kubernetes.io/created-for/pvc/name":"wpdev-wordpress","kubernetes.io/created-for/pvc/namespace":"default"}'
 location: us-central1-a
 physicalBlockSizeBytes: 4096
 projectRef:
 external: my-google-cloud-project
 resourceID: gke-helloworld-2a71d4f-pvc-1e6aa931-d742-437e-a52b-413999ace2be
 size: 10
 type: pd-standard
---
# etc...

Now that we have a single file containing all of our Google Cloud resources, we can take this file as input for Python script.

Mapping from Kubernetes manifests to `pulumi import` JSON

We need to write some code in order to convert the Kubernetes manifests generated by Config Connector into a JSON format suitable for pulumi import. For detailed information on bulk import in Pulumi and all supported fields, see Bulk Import Operations in the Pulumi docs.

In order for a resource to be imported, three fields are required:

type, which indicates the Pulumi resource type, e.g. gcp:compute/network:Network.
name, which, when combined with type, uniquely identifies the resource within the Pulumi program. In the following example, the Pulumi name is my-network:
```
const network = new gcp.compute.Network("my-network");
```
id, which informs the pulumi import command how to query the Google Cloud API for the existing resource for its attributes and output the generated code with all attributes filled in. IDs for import vary by the type of resource, although in Google Cloud, they often follow a predictable pattern which we’ll discuss in further detail soon. The format of a resource’s ID can be found in the Pulumi Registry in the Import section of the resource’s API page (example).

In order to get the type, we need to translate the Kubernetes kind to a Pulumi type. We can accomplish this by keeping a straightforward mapping in a Python dict:

resource_type_mappings = {
 'ArtifactRegistryRepository': {
 'pulumi_type': 'gcp:artifactregistry/repository:Repository'
 },
 'ComputeDisk': {
 'pulumi_type': 'gcp:compute/disk:Disk',
 },
 'ComputeFirewall': {
 'pulumi_type': 'gcp:compute/firewall:Firewall'
 },
 # etc.

Obtaining the name is simple - virtually all of the Kubernetes manifests in our sample data contained a unique or near-unique identifier under k8s_resource['metadata']['name']. Occasionally, we saw naming collisions that cause errors, for example where a compute instance (VM) and its main disk resource shared the same name. At the time of writing, the pulumi import command requires that names be unique within the context of a bulk import. However, this is not a requirement when authoring Pulumi programs - the combination of (name, type) must be unique. In these cases, we simply ensured a unique name by appending -1 to the resource name. This isn’t the most elegant solution, but it works well enough when collisions are relatively infrequent:

def ensure_unique_name(pulumi_resource):
 """Ensures that each resource has a unique name. `pulumi import` will fail
 if multiple resources have the same name due to
 https://github.com/pulumi/pulumi/issues/6032"""

 for resource in resources:
 if 'name' in resource and 'name' in pulumi_resource and resource['name'] == pulumi_resource['name']:
 pulumi_resource['name'] += "-1"
 return

Finally, we need to map the id value. In many cases, the Google Cloud resource ID is in the form {{region}}/{{location}}/{{resource ID}}, and can be mapped from fields that are consistent in the Kubernetes manifest:

def get_default_id(k8s_resource):
 """Returns the most common form of an ID of a Google Cloud resource, adding
 region and zone if they can be determined"""
 id = ""

 if 'region' in k8s_resource['spec']:
 id += f"{k8s_resource['spec']['region']}/"

 if 'location' in k8s_resource['spec']:
 id += f"{k8s_resource['spec']['location']}/"

 id += k8s_resource['spec']['resourceID']

 return id

For any resource types that do not fit this pattern, we’ll need to do some additional work.

Mapping resources with custom ID requirements

If the Kubernetes manifest for our resource does not follow the standard pattern, we’ll need to write a custom function to get the ID. One example of resources that do not follow the standard pattern are compute instances and compute instance groups, which store the project ID in a different key:

apiVersion: compute.cnrm.cloud.google.com/v1beta1
kind: ComputeInstance
metadata:
 annotations:
 cnrm.cloud.google.com/project-id: my-project-id
# ...

Therefore, we need to write a custom function to handle grabbing the IDs from these types:

def get_compute_instance_id(k8s_resource):
 return f"{k8s_resource['metadata']['annotations']['cnrm.cloud.google.com/project-id']}/{k8s_resource['spec']['zone']}/{k8s_resource['metadata']['name']}"

And then map our custom function to the appropriate typed in our type mappings dict:

resource_type_mappings = {
 # ...
 'ComputeInstance': {
 'pulumi_type': 'gcp:compute/instance:Instance',
 'get_id': get_compute_instance_id,
 },
 'ComputeInstanceGroup': {
 'pulumi_type': 'gcp:compute/instanceGroup:InstanceGroup',
 'get_id': get_compute_instance_id,
 },
 # ...
}

The full code also contains some additional mapping and processing for Google Cloud IAM resources, but it follows the same basic patterns as the mapping for other types.

Importing to Pulumi

Now that we have our conversion script written, we can run it with the following command:

python3 main.py -i sample-output.yaml -o pulumi-import.json -p <your-project-id>

The command will generate output like the following:

{
 "resources": [
 {
 "type": "gcp:artifactregistry/repository:Repository",
 "name": "gcf-artifacts",
 "id": "europe-west1/gcf-artifacts"
 },
 {
 "type": "gcp:compute/disk:Disk",
 "name": "gke-helloworld-2a71d4f-pvc-1e6aa931-d742-437e-a52b-413999ace2be",
 "id": "us-central1-a/gke-helloworld-2a71d4f-pvc-1e6aa931-d742-437e-a52b-413999ace2be"
 },
// ...

Once we’re in our Pulumi program’s root, we can instruct pulumi import to do a bulk import and output the generated code to a file in the language of our program (this example includes TypeScript output, but pulumi import will automatically generate code in the correct language for your project).

pulumi import -f ../config-connector-transform/pulumi-import.json -y -s dev -o index.ts

And we can see that in our Pulumi program, all the attributes of our imported resources are present, which means no further actions are (typically - see note below for known exceptions) necessary! We are ready to manage our critical infrastructure as code from now on!

const gcf_artifacts = new gcp.artifactregistry.Repository("gcf-artifacts", {
 description: "This repository is created and used by Cloud Functions for storing function docker images.",
 format: "DOCKER",
 labels: {
 "goog-managed-by": "cloudfunctions",
 },
 location: "europe-west1",
 project: "your-project-id",
 repositoryId: "gcf-artifacts",
});

const gke_helloworld_2a71d4f_pvc_1e6aa931_d742_437e_a52b_413999ace2be = new gcp.compute.Disk("gke-helloworld-2a71d4f-pvc-1e6aa931-d742-437e-a52b-413999ace2be", {
 description: "{\"kubernetes.io/created-for/pv/name\":\"pvc-1e6aa931-d742-437e-a52b-413999ace2be\",\"kubernetes.io/created-for/pvc/name\":\"wpdev-wordpress\",\"kubernetes.io/created-for/pvc/namespace\":\"default\"}",
 labels: {
 "goog-gke-volume": "",
 },
 name: "gke-helloworld-2a71d4f-pvc-1e6aa931-d742-437e-a52b-413999ace2be",
 physicalBlockSizeBytes: 4096,
 project: "your-project-id",
 size: 10,
 zone: "us-central1-a",
});

You may need to massage some of the generated output for any *IAMPolicy resources because deleted principals cannot be contained in IAM policies due to a limitation of the Google Classic provider, but they are still contained in the generated code returned by pulumi import, which in turn comes from the Google API.

Next Steps

Now that we have our Google Cloud resources under Pulumi management, we might want to consider some additional steps to build upon our IaC solution:

Set up a CI/CD pipeline: By adding our Pulumi code to a delivery pipeline, we can help ensure that all infrastructure changes go through an automated process. Pulumi has helpful integrations and documentation for many popular tools. See Continuous Delivery in the Pulumi docs for more information.
Add policy as code: By adding policy as code to our IaC pipeline, we can ensure that our infrastructure is in compliance with any security or regulatory requirements before it’s ever provisioned in the cloud, as well as checks on any existing Pulumi resources. For more information on Pulumi’s policy as code capabilities, see CrossGuard (Policy as Code) in the Pulumi docs.

Conclusion

We’ve shown how the Google Cloud account scraper combined with pulumi import enables organizations to quickly get all of their Google Cloud resources under management with Pulumi. Managing cloud resources with Pulumi allows organizations to utilize modern, automated, code-centric approach, and pulumi import makes adopting Pulumi as painless as possible, even when critical resources have been created via point and click in the console!

If you use the Google Cloud account scraper and find something you need is not supported, please submit an issue, or better yet, submit a pull request!

How to Manage Kafka Clusters in Confluent Cloud with Pulumi

Josh Kodroff — Wed, 15 Feb 2023 00:00:00 +0000

Event streaming is used across diverse industries that demand real-time data processing. Apache Kafka is the most popular open-source streaming platform. Confluent Cloud lets you run Kafka on the cloud provider of your choice.

In this blog post, you’ll use the Confluent Cloud Pulumi provider and Pulumi to create a Kafka cluster, topic, and customer account.

About Apache Kafka and Confluent Cloud

What is Apache Kafka?

Apache Kafka is an event store and stream-processing platform, used by more than 30% of the Fortune 500 today. Using Kafka streams, developers can write modern, event-driven applications for real-time data streaming and processing. Kafka is used across many industries, including gaming, financial services, healthcare, retail, automotive, and manufacturing.

Kafka was created to allow scalable high-throughput applications to store, analyze, and reprocess streaming data. However, managing Kafka clusters can require significant operational expertise, leading many organizations to look for a managed solution.

What is Confluent Cloud?

Confluent Cloud provides managed Kafka clusters along with major value-add features such as elasticity, integrated security, stream governance, and improved monitoring. Clusters can be provisioned in AWS, Azure, or Google Cloud to reduce network latency and egress charges. Confluent Cloud also offers cluster linking capabilities to on-prem producers and consumers for hybrid cloud scenarios.

Using Pulumi, you can manage your Confluent resources and maximize your organization’s ability to quickly ship modern, secure, event-driven workloads.

Initializing the Project

Before you can add Confluent resources to your Pulumi program, you’ll need to ensure you have a Confluent Cloud account and an API key. You can sign up for a free trial of Confluent Cloud if you do not already have a Confluent Cloud account. Create an API key and set its values as environment variables:

export CONFLUENT_CLOUD_API_KEY=<your API key>
export CONFLUENT_CLOUD_API_SECRET=<your API secret>

Now you can create a new directory and initialize the Pulumi program:

mkdir confluent-blog-post
cd confluent-blog-post
pulumi new typescript

After a few seconds, the Pulumi program has been initialized. Next, you’ll need to add a reference to the Pulumi Confluent provider:

npm i @pulumi/confluentcloud

Finally, you’ll need to add a reference to the top of the scaffolded index.ts file:

import * as confluent from "@pulumi/confluentcloud";

Now you’re ready to create and manage Confluent resources!

Adding Resources

Your example architecture will have the following components:

A Kafka cluster for our messages (“inventory”).
An admin service account which you’ll use to create objects within the cluster (topics and users).
A Kafka topic for our cluster, which will hold our sample messages.
A producer service account, which you’ll use to write messages to the topic.
A consumer service account, which you’ll use to read messages from the topic.

You’ll be keeping all of your resources in a single file: index.ts.

The first resource you’ll need to create is a Confluent environment which is a container for the other Confluent resources you’ll be creating:

const env = new confluent.Environment("environment", {
 displayName: "pulumi-confluent-blog",
});

Next, you’ll create a standard Kafka cluster. A couple notes about the cluster you’re creating:

The cluster is a single-zone cluster for cost reasons, but if you’re creating a cluster for production scenarios you’ll likely want to use the MULTI_ZONE option for availability.
While this cluster is in AWS’ us-east-2 region, Confluent Cloud also supports Azure and Google Cloud as well as other regions within AWS. For a full list of supported options for the cloud and region attributes, see Cloud Providers and Regions for Confluent Cloud.

Add the following code to your Pulumi program:

const cluster = new confluent.KafkaCluster("cluster", {
 displayName: "inventory",
 availability: "SINGLE_ZONE",
 cloud: "AWS",
 region: "us-east-2",
 environment: {
 id: env.id,
 },
 standard: {}
});

Next, you’ll need to create the admin-level service account you’ll use to create our Kafka topic and our producer and consumer accounts. This app manager account is similar to the “DBA” account you may be familiar with in relational databases or the root account in Linux:

const serviceAccount = new confluent.ServiceAccount("app-manager", {
 description: "Service account to manage 'inventory' Kafka cluster",
});

const roleBinding = new confluent.RoleBinding("app-manager-kafka-cluster-admin", {
 principal: pulumi.interpolate`User:${serviceAccount.id}`,
 roleName: "CloudClusterAdmin",
 crnPattern: cluster.rbacCrn,
});

const managerApiKey = new confluent.ApiKey("app-manager-kafka-api-key", {
 displayName: "app-manager-kafka-api-key",
 description: "Kafka API Key that is owned by 'app-manager' service account",
 owner: {
 id: serviceAccount.id,
 kind: serviceAccount.kind,
 apiVersion: serviceAccount.apiVersion,
 },
 managedResource: {
 id: cluster.id,
 apiVersion: cluster.apiVersion,
 kind: cluster.kind,
 environment: {
 id: env.id,
 },
 }
}, {
 dependsOn: roleBinding
});

Next, you’ll create your Kafka topic using the cluster admin service account credentials you just created (see the credentials input in the following code):

const topic = new confluent.KafkaTopic("orders", {
 kafkaCluster: {
 id: cluster.id,
 },
 topicName: "orders",
 restEndpoint: cluster.restEndpoint,
 credentials: {
 key: managerApiKey.id,
 secret: managerApiKey.secret,
 },
});

Now that you have your topic, you need to create a producer service account and give that account permissions to write to the topic, again using the credentials of your cluster admin account:

const producerAccount = new confluent.ServiceAccount("producer", {
 description: "Service account to produce to 'orders' topic of 'inventory' Kafka cluster",
});

const producerApiKey = new confluent.ApiKey("producer-api-key", {
 owner: {
 id: producerAccount.id,
 kind: producerAccount.kind,
 apiVersion: producerAccount.apiVersion,
 },
 managedResource: {
 id: cluster.id,
 apiVersion: cluster.apiVersion,
 kind: cluster.kind,
 environment: {
 id: env.id,
 },
 },
});

new confluent.KafkaAcl("app-producer-write", {
 kafkaCluster: {
 id: cluster.id,
 },
 resourceType: "TOPIC",
 resourceName: topic.topicName,
 patternType: "LITERAL",
 principal: pulumi.interpolate`User:${producerAccount.id}`,
 host: "*",
 operation: "WRITE",
 permission: "ALLOW",
 restEndpoint: cluster.restEndpoint,
 credentials: {
 key: managerApiKey.id,
 secret: managerApiKey.secret,
 }
});

Now you create our consumer account which will read messages from your Kafka topic. It’s created similarly to the producer:

const consumerAccount = new confluent.ServiceAccount("consumer", {
 description: "Service account to consume from 'orders' topic of 'inventory' Kafka cluster",
});

const consumerApiKey = new confluent.ApiKey("consumer-api-key", {
 owner: {
 id: consumerAccount.id,
 kind: consumerAccount.kind,
 apiVersion: consumerAccount.apiVersion,
 },
 managedResource: {
 id: cluster.id,
 apiVersion: cluster.apiVersion,
 kind: cluster.kind,
 environment: {
 id: env.id,
 },
 },
});

new confluent.KafkaAcl("consumer-read-topic-acl", {
 kafkaCluster: {
 id: cluster.id,
 },
 resourceType: "TOPIC",
 resourceName: topic.topicName,
 patternType: "LITERAL",
 principal: pulumi.interpolate`User:${consumerAccount.id}`,
 host: "*",
 operation: "READ",
 permission: "ALLOW",
 restEndpoint: cluster.restEndpoint,
 credentials: {
 key: managerApiKey.id,
 secret: managerApiKey.secret,
 }
});

new confluent.KafkaAcl("consumer-read-group-acl", {
 kafkaCluster: {
 id: cluster.id,
 },
 resourceType: "GROUP",
 resourceName: "confluent_cli_consumer_",
 patternType: "PREFIXED",
 principal: pulumi.interpolate`User:${consumerAccount.id}`,
 host: "*",
 operation: "READ",
 permission: "ALLOW",
 restEndpoint: cluster.restEndpoint,
 credentials: {
 key: managerApiKey.id,
 secret: managerApiKey.secret,
 }
});

Finally, add some Pulumi stack outputs. Stack outputs allow you to access values from your Pulumi program in two ways:

Via stack references in other Pulumi programs, which you won’t use in this blog tutorial.
From the command line via the pulumi stack output command, which you will use to test your Kafka cluster.

The syntax for Pulumi stack outputs varies by language, but in TypeScript programs they are accomplished by a simple export statement:

export const ordersTopicName = topic.topicName;
export const environmentId = env.id;
export const clusterId = cluster.id;

export const producerApiKeyId = producerApiKey.id;
export const producerApiKeySecret = producerApiKey.secret;

export const consumerApiKeyId = consumerApiKey.id;
export const consumerApiKeySecret = consumerApiKey.secret;

Your Pulumi program is now complete! You can deploy our infrastructure by running the following command:

pulumi up

After a short wait, your cluster is up and running, and you are ready to test your infrastructure!

Testing Kafka Cluster

To simulate your producer and consumer, you can use the Confluent CLI to send messages to and read messages from your topic. You will use the values of your Pulumi stack outputs to formulate the command.

Make sure you run the confluent login command first if you have not yet done so. At the time of writing, failure to do so will lead to misleading error messages from the Confluent CLI. The confluent login command will need to be run periodically as the token it generates expires after a few hours.

To simulate a message producer, you can send messages to your Kafka topic with the following command:

confluent kafka topic produce $(pulumi stack output ordersTopicName) \
 --environment $(pulumi stack output environmentId) \
 --cluster $(pulumi stack output clusterId) \
 --api-key $(pulumi stack output producerApiKeyId) \
 --api-secret "$(pulumi stack output producerApiKeySecret --show-secrets)"

You can enter a few sample records like the following and then press Ctrl-C when you’re done to exit:

{"number":1,"date":18500,"shipping_address":"899 W Evelyn Ave, Mountain View, CA 94041, USA","cost":15.00}
{"number":2,"date":18501,"shipping_address":"1 Bedford St, London WC2E 9HG, United Kingdom","cost":5.00}
{"number":3,"date":18502,"shipping_address":"3307 Northland Dr Suite 400, Austin, TX 78731, USA","cost":10.00}

To simulate your consumer and read the records you just wrote, you can enter the following command:

$ confluent kafka topic consume $(pulumi stack output ordersTopicName) \
 --from-beginning \
 --environment $(pulumi stack output environmentId) \
 --cluster $(pulumi stack output clusterId) \
 --api-key $(pulumi stack output consumerApiKeyId) \
 --api-secret $(pulumi stack output consumerApiKeySecret --show-secrets)

You should see the events you entered as the producer:

Starting Kafka Consumer. Use Ctrl-C to exit.
{"number":3,"date":18502,"shipping_address":"3307 Northland Dr Suite 400, Austin, TX 78731, USA","cost":10.00}
{"number":1,"date":18500,"shipping_address":"899 W Evelyn Ave, Mountain View, CA 94041, USA","cost":15.00}
{"number":2,"date":18501,"shipping_address":"1 Bedford St, London WC2E 9HG, United Kingdom","cost":5.00}

The producer is able to write events to your topic, and the consumer is able to read them. Your architecture has been proven to work!

See it in action

Want to see this in action? Here is a Modern Infrastructure video demonstrating everything discussed in this blog post:

Video Transcript

Hello, and welcome to another episode of Modern Infrastructure Wednesday. I’m your host, Lee Zen, and today we’re going to be covering how easy it is to use Pulumi to deploy an application using AWS App Runner. App Runner is a new service from Amazon that lets you deploy a container image or a source code repository as an application without having to do too much configuration. You don’t have to configure load balancers or any of the typical stuff you usually have to set up. Here, you can see I’m running the main.go file, and it’s going to create all the necessary resources, along with, finally, the App Runner service.

We’re going to dive into all the source code to explain how this works, but right now you can see what’s happening: we’re actually doing the Docker build that’s going to publish the image. After the Docker build completes, it’s pushing the image into our ECR repository. Finally, you can see here the update succeeds, and we actually get the service URL. Now let’s go take a look at our service. While the service is updating, you’ll notice it takes a little bit of time—not too much—but just a little bit to get going.

Now we’re going to see how our code actually works. Let’s look at our deployment.go file, which main.go actually calls. We’ve built an Automation API-based program, and as you can see in the outputs, we create the image and various other resources. Let’s take a closer look at the code: the first thing we do is create an ECR repository using auto-naming, so it automatically generates the name. Next, we create a role so that App Runner can pull the image from our repository. We create this role and give it the appropriate policy. Afterward, we publish our image. You can see here we have the image resource, and there we feed it the repository credentials. Let’s also take a quick look at the Dockerfile we’re building—it’s just copying an index.html file over, making it a very simple setup.

That’s really all there is to the infrastructure—it’s super simple. If we look at the Automation API side of things, all we’re doing is the usual Automation API stuff: setting up a stack and deploying our program. Right here, we point out where we’re deploying the program. Once we’ve finished deploying, we’re going to use the outputs from that program—namely, the image URL and the access role—and we’ll feed those to App Runner. This will invoke the Create Service API call, which is what’s actually doing the work. Going back to the console, you can now see we have the app running, and voilà, exactly what we expected to see! Hopefully, you enjoyed this demo. Like I said, it’s just so easy to get everything running with App Runner in just a few simple lines of code and Pulumi. Thanks for watching, and have a great day!

Conclusion

By combining the operational simplicity and rich functionality of Confluent Cloud with the power of Pulumi’s infrastructure as code platform to manage Confluent resources using real programming languages, organizations can quickly and securely deploy Apache Kafka clusters.

No matter whether your organization is using data streaming today or looking to adopt it in the future, using Confluent Cloud with Pulumi will allow your organization to quickly spin up and manage Kafka infrastructure so you can focus on what really matters: delivering value to customers and stakeholders.

Additional Resources

Watch Collin James, Engineering Leader, and Software Architect at Dutchie, describe how a small team has enabled Kafka adoption by creating a monorepo of Pulumi projects that manage resources on Confluent Cloud.

Automating Pulumi Import with Manually Created Resources

Josh Kodroff — Mon, 30 Jan 2023 00:00:00 +0000

A few weeks ago, I was speaking with a consultant at one of the big firms who asked me how he could introduce Pulumi into a client’s organization when that client had created many infrastructure resources manually through the AWS console and was running production workloads on those resources.

Introducing modern cloud infrastructure tooling and automation is relatively simple (or at least more straightforward) when organizations decide to adopt IaC from the start of their cloud journey, but what about organizations who have gone far enough down the route of manually created cloud infrastructure to see the perils of that approach? Many teams come to this realization only when they’ve deployed too many production workloads to start over from scratch. If your organization is looking at Pulumi as an IaC solution, it’s worth bringing these resources under management because of the low effort and high value of having a single pane of glass to manage all of your resources.

Bringing resources under IaC (and Pulumi specifically) gives an organization an immediate return on investment. Note that while we list some benefits specific to the Pulumi Service, Pulumi supports a number of backends for both state management and secrets management and the Pulumi Service is not required in order to adopt Pulumi:

Version control/change history: With IaC placed under version control, teams can easily roll back to previous versions of infrastructure, see who changed what, and when, which provides critical visibility into how infrastructure has evolved over time. The Pulumi Service provides a great experience here, with a dashboard that collects version history and deployment logs along with git contextual data in a single pane of glass.
Automation and pipelines: Because IaC is code, it enjoys the same main benefits from automated pipelines as application code: a repeatable and reliable process. We can also enjoy significant security benefits: When IaC and automated pipelines become the standard way to deploy infrastructure, organizations can significantly reduce the number of users with direct access to critical infrastructure (i.e., via the cloud console or CLI) and the level of privileges needed (e.g., read-only rather than full admin). Once we get our pipeline established so that we have repeatable automated delivery of our infrastructure, we can add automated quality assurance to Pulumi programs via unit testing and policy as code.
Drift detection and correction: Most IaC tools will detect changes made out of band to resources, e.g., changes made manually in the console, and attempt to reconcile the resource back to its declared configuration. Thus, by bringing manually created resources under IaC, organizations can put a definitive end to the slow and error-prone practice of manually managing cloud resources. In Pulumi, the pulumi refresh command will refresh all resources in the state file. For more information on strategies to address configuration drift, check out Patterns of Drift Detection with Pulumi in the Pulumi blog.

About Pulumi Import

The pulumi import command is executed within the context of a Pulumi stack. The user supplies the type of resource to be imported, a name with which to identify the resource in the Pulumi program and state file, and a resource-dependent ID. For example, we might run the following command to import an AWS VPC (this command is sourced from the API docs for the AWS provider in the Pulumi Registry):

pulumi import aws:ec2/vpc:Vpc imported-vpc vpc-0b0a6ad0766eccf3b

When we run the command, Pulumi queries the AWS API for the VPC’s attributes. The resource’s attributes are added to the stack’s state file and the command prints out code to be added to our Pulumi program:

$ pulumi import aws:ec2/vpc:Vpc imported-vpc vpc-0b0a6ad0766eccf3b
Previewing import (dev)

View Live: https://app.pulumi.com/jkodrofftest/pulumi-import-blog-sample/dev/previews/2e462988-0771-4fd0-8e13-bf0b88eb9d4b

Type Name Plan
 + pulumi:pulumi:Stack pulumi-import-blog-sample-dev create
 = └─ aws:ec2:Vpc imported-vpc import

Resources:
 + 1 to create
 = 1 to import
 2 changes

Do you want to perform this import? yes
Importing (dev)

View Live: https://app.pulumi.com/jkodrofftest/pulumi-import-blog-sample/dev/updates/1

Type Name Status
 + pulumi:pulumi:Stack pulumi-import-blog-sample-dev created (4s)
 = └─ aws:ec2:Vpc imported-vpc imported (3s)

Resources:
 + 1 created
 = 1 imported
 2 changes

Duration: 5s

Please copy the following code into your Pulumi application. Not doing so
will cause Pulumi to report that an update will happen on the next update command.

Please note that the imported resources are marked as protected. To destroy them
you will need to remove the `protect` option and run `pulumi update` *before*
the destroy will take effect.

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

const imported_vpc = new aws.ec2.Vpc("imported-vpc", {
 cidrBlock: "10.0.0.0/16",
 tags: {
 Name: "vpc",
 },
}, {
 protect: true,
});

Note that in the preceding example we have TypeScript output, but pulumi import will automatically detect the language our Pulumi project uses and will output code in the correct language.

Our resource also has the protect: true option specified by default, which means that Pulumi will not delete (or recreate) the resource if the program changes. This is an overridable default setting chosen for safety: If a resource is being imported in the first place, it’s typically because there’s an important workload using it, and we should be very careful before deleting or re-creating it.

Now that we’ve run the pulumi import command and added the code that the command outputs, our resource can now be managed by Pulumi. If we run the pulumi preview command, we will see that our resource has been added to our stack and that no changes have been detected, meaning that Pulumi has correctly imported all of our VPC’s attributes:

$ pulumi preview
Previewing update (dev)

View Live: https://app.pulumi.com/jkodrofftest/pulumi-import-blog-sample/dev/previews/6ed858d9-c4f5-4770-9c22-c9aed39646bd

 Type Name Plan
 pulumi:pulumi:Stack pulumi-import-blog-sample-dev

Resources:
 2 unchanged

Automated Batch Import Capabilities

For projects with many resources, pulumi import also has an option that allows it to take a list of resources in a specially formatted JSON file for batch import. We’ll be using this batch import capability to add our manually-created resources into the Pulumi state. A basic sample JSON input follows:

{
 "resources": [
 {
 "type": "aws:ec2/vpc:Vpc",
 "name": "my-vpc",
 "id": "vpc-094958e4051c478c3"
 },
 {
 "type": "aws:ec2/vpc:Vpc",
 "name": "my-other-vpc",
 "id": "vpc-0f12a82357335a28f"
 }
 ]
}

Because pulumi import gives us the full code for an imported resource with all attributes (as demonstrated in the previous section), and because pulumi import has an option for batch import from an external file source (both key differentiators between Pulumi and other IaC tools) we can use pulumi import to import existing resources at scale in an automated fashion.

If you are looking to import resources that were created with a non-Pulumi IaC tool as opposed to manually in the console, the approach described in this article will still work, but you may also want to consider the pulumi convert --from terraform command and cf2pulumi tool for resources created with Terraform and CloudFormation, respectively. For a comprehensive guide on the various options for importing existing resources into Pulumi, see Migrating to Pulumi.

Designing an End-to-End Solution

Now that we’ve explained the benefits of bringing manually created resources under IaC, and the capabilities of the pulumi import command, we need to design a solution that will allow us to leverage pulumi import’s batch import capabilities so that we can bring all resources in our cloud environment under Pulumi management.

Our solution for an automated batch import of resources into Pulumi comprises the following steps:

Create an account scraper using our cloud provider’s SDK (or CLI) to query for the resources to be imported.
Take the output from the cloud provider’s SDK and transform it into a JSON file suitable as an import for a batch pulumi import operation.
Run pulumi import to bring the resources under Pulumi control.

Writing an Account Scraper

In our account scraper, we will use the AWS Python SDK, boto3, to query resources created in the AWS console, but a similar approach can be applied to the many cloud and SaaS providers supported by Pulumi. The full code for the sample solution can be found on GitHub at https://github.com/pulumi/pulumi-import-aws-account-scraper. If you find the tool useful, please submit issues and/or PRs to support additional resource types!

Because the ID attribute (or multiple attributes in the case of some resources) vary based on the resource type (in AWS at least, your cloud provider may vary), we need to write custom code for each resource type that we want to import. (Note that the account scraper code only queries a subset of AWS resource types, but the code can be easily modified to accommodate additional resource types.) For example, the following code shows a query for associations between route tables and VPC subnets:

def import_route_table_associations(ec2_client):
 pulumi_resources = []

 route_tables = ec2_client.describe_route_tables()['RouteTables']

 for route_table in route_tables:
 for association in route_table["Associations"]:
 if 'SubnetId' not in association:
 continue
 pulumi_resources.append({
 "type": "aws:ec2/routeTableAssociation:RouteTableAssociation",
 "name": f"import-{association['RouteTableAssociationId']}",
 "id": f"{association['SubnetId']}/{route_table['RouteTableId']}",
 })

 return pulumi_resources

We can run the entire account scraper program to output our resources to a single JSON file by running the following command. Detailed instructions are contained in the account scraper repository:

python account_scraper.py > /path/to/file.json

Batch Import

Now that we’ve generated our JSON file, we can use the pulumi import command to import all of our resources in a single command and output our code to its own file:

pulumi import -f /path/to/file.json -o imported-resources.ts -y

And with that, our resources are now managed by Pulumi!

Next Steps

Now that we have our resources under Pulumi management, we might want to consider some additional steps to build upon our IaC solution:

Set up a CI/CD pipeline: By adding our Pulumi code to a delivery pipeline, we can help ensure that all infrastructure changes go through an automated process. Pulumi has helpful integrations and documentation for many popular tools. See Continuous Delivery in the Pulumi docs for more information.
Add policy as code: By adding policy as code to our IaC pipeline, we can ensure that our infrastructure is in compliance with any security or regulatory requirements before it’s ever provisioned in the cloud, as well as checks on any existing Pulumi resources. For more information on Pulumi’s policy as code capabilities, see CrossGuard (Policy as Code) in the Pulumi docs.

Conclusion

We’ve shown how pulumi import enables organizations to quickly move from slow and error prone manual management of cloud resources to a modern, automated, code-centric approach, even when we need to preserve those manually created resources because they are running critical workloads. It’s never too late for an organization to transition to managing infrastructure resources as code, and Pulumi makes this transition as painless as possible!

AWS Lambda SnapStart with Pulumi

Josh Kodroff — Mon, 28 Nov 2022 14:00:00 -0800

As AWS Lambda has matured as a serverless platform, there are two key ways the service has evolved:

New capabilities that extend the platform to support new use cases like Lambda Container support, Lambda URLs and attribute-based access control support.
Performance enhancements that enable Lambda functions to be more responsive and cost-effective such as Tiered compilation, and Graviton2 support are just a few examples of the investments AWS made in this space.

With each advancement, the reasons to deploy a full VM or cluster to support your application get fewer and the time to deliver value in the cloud gets shorter.

One challenge that many teams have faced when adapting their services to take advantage of AWS Lambda’s simple deployment model and attractive economics is cold starts: the latency incurred when the service prepares a new execution environment for your function. There is good analysis on the source of this latency as well as some available optimizations here.

Functions using Java runtimes typically experience significantly longer initialization times than functions using other runtimes.

Lambda creates a fresh execution environment when a function is invoked for the first time or when the function gets scaled to handle a spike in traffic. Initialization requires downloading function code, initializing the runtime, and running any pre-invocation initialization code in your function code. (Examples of pre-invocation code are retrieving secrets from AWS Secrets Manager or initializing database or other network connections.) This initialization process can add seconds of latency to the first invocation of your function.

Reducing Function Latency with AWS Lambda SnapStart

Today, AWS released a major improvement for Java-based functions that specifically addresses the latency incurred from cold starts in Java functions: AWS Lambda SnapStart. SnapStart improves startup performance for latency-sensitive Java applications by up to 10x at no extra cost, and typically with no changes to your function code. With SnapStart, instead of initializing function code on invoke, initialization happens when a function version is published. The way AWS describes this new feature is that the Lambda service will take a Firecracker MicroVM snapshot of the memory and disk state of the function’s fully-initialized execution environment, persist the encrypted snapshot and place it in a cache it for rapid access. On a first invocation or on scaling the function, Lambda resumes new environments using this snapshot.

AWS Lambda further improves invoke times by dividing snapshots into chunks. This means only the chunks required for your function are downloaded from a distributed cache on invoke. In real terms, this process removes several seconds of latency from function execution and scaling times.

The new Lambda Telemetry API works with SnapStart so you can easily track platform metrics related to this new capability.

All of the benefits of SnapStart are a great reason to give this feature a try, and there is no additional cost for enabling SnapStart for your functions. However, there are some considerations to keep in mind before you get started:

SnapStart improvements require the Amazon Corretto 11 runtime - Amazon’s production-ready distribution of OpenJDK.
After 14 days without an invocation, your function version becomes inactive and Lambda will delete the snapshot. If you invoke an inactive function version, Lambda will initialize a new snapshot.
Network connections may not persist when a function is invoked from a snapshot, so, your code should assume connections need to be restored.
Ephemeral data such as temporary credentials created during initialization may expire between the time the snapshot is created and later resumed.
Since snapshots are reused many times to resume execution environments, there are additional requirements to ensure random numbers generated by your function remain unique.

Lambda functions that need to persist network connections, rely on ephemeral data created during initialization, or have strict requirements for randomness will require extra attention. The SnapStart documentation and AWS blog are helpful for understanding the code changes required to support these scenarios. For example, a common mistake is using a Pseudorandom Number Generator (PRNG) such as java.lang.Math.random() when a Cryptographically Secure Pseudorandom Number Generator (CPRNG) such as java.security.SecureRandom() will ensure your application code generates random values as expected with SnapStart.

Configuring Lambda Functions to work with SnapStart in Pulumi

To take advantage of faster cold starts, you must create your function resource with the option to use SnapStart. Note that beyond the previously described requirements to make your function code SnapSafe, no changes need to be made to the function code itself to use SnapStart. In the following example, we use the AWS Cloud Control provider to provision our function with SnapStart. The AWS Cloud Control provider uses the new Cloud Control API behind the scenes and has same-day support for new resources included in Cloud Control API - like SnapStart. This example also uses the AWS provider and the Pulumi Command Provider.

First, we add the necessary imports, create a role for our function to assume, and an S3 bucket to hold our Lambda code. Then we add our function’s zipped code as an object in the bucket. For this example, the Lambda function code is the AWS pet store example in Spring Boot:

import * as pulumi from "@pulumi/pulumi";
import * as awsNative from "@pulumi/aws-native";
import * as aws from "@pulumi/aws";
import * as command from "@pulumi/command";

const role = new aws.iam.Role("role", {
 assumeRolePolicy: JSON.stringify({
 "Version": "2012-10-17",
 "Statement": [{
 "Effect": "Allow",
 "Principal": {
 "Service": "lambda.amazonaws.com",
 },
 "Action": "sts:AssumeRole",
 }],
 }),
});

new aws.iam.RolePolicyAttachment("role-policy-attachment", {
 role: role.name,
 policyArn: "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole",
});

const bucket = new aws.s3.Bucket("snapstart-bucket", {
 versioning: {
 enabled: true,
 }
});

const functionCode = new aws.s3.BucketObject("function-code", {
 bucket: bucket.bucket,
 source: new pulumi.asset.FileArchive("../petstore.zip"),
});

Next, we create our function with SnapStart enabled using the AWS Cloud Control provider, publish a version of our function (a requirement to use SnapStart) via the AWS CLI, and create a function alias for the version we created:

const func = new awsNative.lambda.Function("snapstart-func", {
 code: {
 s3Bucket: functionCode.bucket,
 s3Key: functionCode.key,
 s3ObjectVersion: functionCode.versionId,
 },
 role: role.arn,
 runtime: "java11",

 handler: "com.amazonaws.serverless.sample.springboot2.StreamLambdaHandler::handleRequest",
 memorySize: 1512,
 timeout: 60,
 snapStart: {
 applyOn: "PublishedVersions",
 },
});

const publishVersion = new command.local.Command("publish-version", {
 create: func.arn.apply(arn => `aws lambda publish-version --function-name ${arn}`),
 triggers: [func],
}, {
 dependsOn: func,
});

const aliasName = "v1";

const alias = new aws.lambda.Alias("alias", {
 functionName: func.arn,
 functionVersion: "1",
 name: aliasName,
}, {
 dependsOn: publishVersion,
});

Finally, we create the necessary resources to integrate our function with API Gateway and add our test URL as a stack output. This allows us to access it from outside of our Pulumi program:

const api = new aws.apigatewayv2.Api("snapstart-api", {
 protocolType: "HTTP",
});

const integration = new aws.apigatewayv2.Integration("lambdaIntegration", {
 apiId: api.id,
 integrationType: "AWS_PROXY",
 integrationUri: alias.arn,
 integrationMethod: "GET",
 payloadFormatVersion: "1.0",
 passthroughBehavior: "WHEN_NO_MATCH",
 connectionType: "INTERNET"
});

const route = new aws.apigatewayv2.Route("apiRoute", {
 apiId: api.id,
 routeKey: "$default",
 target: pulumi.interpolate`integrations/${integration.id}`,
});

new aws.apigatewayv2.Stage("apiStage", {
 apiId: api.id,
 name: "$default",
 routeSettings: [
 {
 routeKey: route.routeKey,
 throttlingBurstLimit: 5000,
 throttlingRateLimit: 10000,
 },
 ],
 autoDeploy: true,
}, { dependsOn: [route] });

new aws.lambda.Permission("permission", {
 action: "lambda:InvokeFunction",
 principal: "apigateway.amazonaws.com",
 function: alias.arn,
 sourceArn: api.executionArn.apply(x => `${x}/*/*`),
});

export const apiUrl = pulumi.concat(api.apiEndpoint, "/pets");

Deployment and Performance Testing

To deploy our function, we simply run the pulumi up command:

pulumi up -y

As the command completes, you’ll see output like the following:

Updating (dev)

View Live: https://app.pulumi.com/jkodroff/blog-lambda-snapshots-springboot-snapstart/dev/updates/9

 Type Name Status Info
 + pulumi:pulumi:Stack blog-lambda-snapshots-springboot-snapstart-dev created (4s) 1 warning
 + ├─ aws:iam:Role role created (0.82s)
 + ├─ aws:s3:Bucket snapstart-bucket created (2s)
 + ├─ aws:apigatewayv2:Api snapstart-api created (1s)
 + ├─ aws:iam:RolePolicyAttachment role-policy-attachment created (0.24s)
 + ├─ aws:s3:BucketObject function-code created (13s)
 + ├─ aws-native:lambda:Function snapstart-func created (22s)
 + ├─ command:local:Command publish-version created (3s)
 + ├─ aws:lambda:Alias alias created (0.66s)
 + ├─ aws:lambda:Permission permission created (0.40s)
 + ├─ aws:apigatewayv2:Integration lambdaIntegration created (0.74s)
 + ├─ aws:apigatewayv2:Route apiRoute created (0.54s)
 + └─ aws:apigatewayv2:Stage apiStage created (1s)


Outputs:
 apiUrl: "https://56r9255ey4.execute-api.us-west-2.amazonaws.com/pets"

Resources:
 + 13 created

Duration: 52s

We invoke our function by issuing an HTTP GET request against our stack output:

$ curl $(pulumi stack output apiUrl)
[{"id":"24ecd2bc-5106-46f8-95f2-af7de19cba57","breed":"Bloodhound","name":"Cooper","dateOfBirth":1639148877751},{"id":"caf6e8fa-fc97-4b9c-bfe7-996f14dc7995","breed":"Bloodhound","name":"Zoe","dateOfBirth":1438614477751},{"id":"b3d28309-8c92-4ae8-b7bd-7b656a5ea0bf","breed":"Beagle","name":"Murphy","dateOfBirth":1282489677751},{"id":"3c572757-298a-4343-9ca7-5c0865f9fce1","breed":"Beagle","name":"Winston","dateOfBirth":1410188877751},{"id":"4f8ab29a-760d-4b29-801d-6e35a98c538b","breed":"Dalmatian","name":"Max","dateOfBirth":1619968077751},{"id":"31fe978f-5eb6-4116-9933-ba8d9edc89a8","breed":"Dalmatian","name":"Sasha","dateOfBirth":1615475277751},{"id":"772218c3-cc56-44b7-87be-1db1b46730a3","breed":"Afghan Hound","name":"Lucky","dateOfBirth":1579187277751},{"id":"e994816e-8a6a-448a-bc48-ed404b463e46","breed":"Dalmatian","name":"Roxy","dateOfBirth":1408806477752},{"id":"3987cae6-6db2-48f9-96cb-dcb9812b1ab1","breed":"Bernese Mountain Dog","name":"Bailey","dateOfBirth":1329059277752},{"id":"514c8db6-225e-4fab-8c5b-63acdc88da56","breed":"Bloodhound","name":"Maggie","dateOfBirth":1218812877752}]%

To benchmark performance, use the Apache ab utility which is installed by default on macOS. The following command will run 1000 GET requests against our function:

ab -n 1000 -c 50 $(pulumi stack output apiUrl)

Comparing performance before and after SnapStart

During extensive benchmarking, the Pulumi team tried both a simple “Hello World” function as well as the more complex Spring Boot example above. Invoking this example Lambda function with and without SnapStart over many iterations, the Pulumi team found cold start times for a simple “Hello World” were 40% faster with SnapStart. The team found a much larger performance improvement when deploying the SpringBoot example. Cold starts were more than 10x faster. In real-world usage, this will translate into more responsive applications and services.

Conclusion

Addressing the challenges of cold starts for AWS Lambda functions is a major performance improvement that will enable Lambda functions to be useful in more scenarios that were previously the domain of dedicated VMs and container services. A 10x improvement in latency with common frameworks is a generational leap that we expect many teams to find valuable as they seek to modernize their infrastructure while keeping services responsive for end users.

Check out our GitHub repo with examples to try out Java-based functions with and without SnapStart enabled and see for yourself how these improvements can improve performance.

Happy coding!

Build Self-Service DevOps with AWS Proton, Pulumi & CodeBuild

Isaac Harris — Wed, 16 Nov 2022 08:36:28 -0800

Self-service infrastructure is the holy grail of DevOps. When platform engineering teams can empower application teams to provision their own infrastructure without needing to understand the details of configuring networking, storage, and compute resources, IT organizations can drastically increase their ability to deliver on organizational goals. The first step in this process is to codify infrastructure best practices using platforms like Pulumi and the next step is to make these best practices available in a workflow that fits into the application team’s software development tools and process.

Today, AWS announced support for AWS CodeBuild as a provisioner in AWS Proton. Using CodeBuild as a provisioner allows Proton to integrate with the most popular Infrastructure as Code platforms, including Pulumi. With Codebuild provisioning, Proton enables platform teams to define the steps to provision resources based on their own custom workflow and IaC tool of choice.

What are AWS Proton and AWS CodeBuild?

AWS Proton is a service that enables platform engineers to define, vend, and maintain self-service infrastructure templates for deploying cloud resources in an automated workflow. Development and operations teams can leverage these templates to deploy well-architected, modern infrastructure with confidence. Platform teams can create Proton templates for environments (which typically include shared resources like VPCs and container scheduler clusters) and application services which utilize the resources in the environment. Templates may be stored in git repositories for easy version control and automatic syncing with the Proton service.

AWS CodeBuild is a managed continuous integration (CI) service. Using CodeBuild, engineers can specify a Docker build container image, either pre-built or custom, and a list of commands to execute in the container to perform the build. AWS handles concerns like creating and destroying the container, logging, and metrics. When using CodeBuild as a provisioner for AWS Proton, teams specify the steps to both provision and tear down our infrastructure templates.

Using AWS Proton with CodeBuild and Pulumi

Since CodeBuild provisioning in Proton allows us to execute arbitrary commands, we can script our template builds using the Pulumi CLI. AWS has built a complete, end-to-end example that can be found at GitHub, and for this post we are going to focus on the specific steps required to use CodeBuild provisioning in AWS Proton with Pulumi. Note that in this example we are still using the Pulumi Service as our backend, so we can still enjoy all the benefits the service provides like state file management, secure secrets handling, a shared history of changes to our stacks, and advanced features available in Enterprise and Business Critical editions.

Define Your Proton Manifest File

When using Pulumi with CodeBuild provisioning in AWS Proton, our main interface between Proton and Pulumi occurs in the manifest file where we define the steps to provision and deprovision our infrastructure. Our manifest file also contains mappings for Pulumi configuration. We define our manifest file in YAML and include it in the root of our Pulumi program.

The following is an example of a manifest file that uses CodeBuild provisioning with Proton and Pulumi:

infrastructure:
 templates:
 - rendering_engine: codebuild
 settings:
 image: aws/codebuild/amazonlinux2-x86_64-standard:4.0
 runtimes:
 nodejs: 16
 env:
 variables:
 PULUMI_ORG: <PULUMI-ORG-HERE>
 PULUMI_NAMESPACE: proton-service
 secrets_manager:
 PULUMI_ACCESS_TOKEN: "proton/pulumi-access-token:Secret"
 provision:
 - cat proton-inputs.json
 - curl -fsSL https://get.pulumi.com | sh
 - export PATH=$PATH:$HOME/.pulumi/bin
 - npm install
 - pulumi stack select $PULUMI_ORG/$PULUMI_NAMESPACE/$STACK_NAME || pulumi stack init $PULUMI_ORG/$PULUMI_NAMESPACE/$STACK_NAME
 - pulumi config set aws:region $AWS_DEFAULT_REGION
 - pulumi config set org $PULUMI_ORG
 - pulumi up --yes --stack $PULUMI_ORG/$PULUMI_NAMESPACE/$STACK_NAME
 - chmod +x ./pulumi-to-proton-outputs.sh
 - pulumi stack output --json | ./pulumi-to-proton-outputs.sh > outputs.json
 - aws proton notify-resource-deployment-status-change --resource-arn $RESOURCE_ARN --status IN_PROGRESS --outputs file://./outputs.json
 deprovision:
 - curl -fsSL https://get.pulumi.com | sh
 - export PATH=$PATH:$HOME/.pulumi/bin
 - npm install
 - pulumi stack select $PULUMI_ORG/$PULUMI_NAMESPACE/$STACK_NAME
 - pulumi config set aws:region $AWS_DEFAULT_REGION
 - pulumi config set org $PULUMI_ORG
 - pulumi destroy --yes -s $PULUMI_ORG/$STACK_NAME

The provisioning steps in the manifest are as follows:

Download and install any necessary dependencies not already present in the CodeBuild container, like the Pulumi CLI.
Run the Pulumi commands to provision our infrastructure.
Map our Pulumi outputs to the necessary JSON format required by AWS Proton.

The deprovisioning steps are fairly similar to the provisioning steps:

Download and install any dependencies.
Run the Pulumi commands to tear down our infrastructure.

Create a Schema File for Self-Service Inputs

Proton allows application teams to provision infrastructure via a self-service UI. In order to inform the UI’s controls, platform engineering teams must supply Proton with a template schema file that defines the configurable parameters of the template.

The following is an example of a simple template schema file which allows us to configure the VPC CIDR block in our Proton environment:

schema:
 format:
 openapi: "3.0.0"
 environment_input_type: "EnvironmentInputs"
 types:
 EnvironmentInputs:
 type: object
 description: "Environment inputs"
 properties:
 vpcCidrBlock:
 title: "VPC CIDR block"
 type: string
 default: "10.0.0.0/16"

Package and Sync Templates

Finally, we need to package our template in a suitable format for Proton. Proton templates can be sourced from either S3 bucket objects or synced via git repositories. Using git as the source of truth for Proton templates is fully automated out of the box and thus is recommended over sourcing templates from S3 objects for production scenarios. For more information on git configuration with Proton, see Template sync configurations in the AWS docs.

A sample directory structure for an AWS Proton template that uses Pulumi follows:

.
├── infrastructure
│ ├── Pulumi.yaml
│ ├── index.ts
│ ├── manifest.yaml
│ ├── package-lock.json
│ ├── package.json
│ ├── proton-inputs.json
│ └── tsconfig.json
└── schema
└── schema.yaml

Our sample package contains 2 points of interest:

The infrastructure directory, which contains our Pulumi program plus the Proton manifest file.
The schema directory, which contains our template schema file.

For full details on Proton template packaging, see Wrap up template files for AWS Proton in the AWS docs.

Tying it all together

So what does this all look like from a user’s perspective when provisioning a self-service template? Let’s look at some screenshots.

First, the user is presented with a list of the templates published by the platform team:

After selecting the template to deploy and filling in any configurable input parameters, the user can view the CodeBuild logs and is presented with a link to the CodeBuild job:

Finally, the user can view the fully provisioned template and reference any generated outputs:

Conclusion

Using Proton with CodeBuild and Pulumi, we can enjoy all the benefits of self-service infrastructure along with the benefits of the Pulumi service like secrets management and stack history, while using real programming languages to allow us to ship infrastructure faster. Give it a try today!

Pulumi Blog: Josh Kodroff

Day 2 Operations: Drift Detection and Remediation

The Reality of Infrastructure Drift

Understanding the Prevention vs. Detection Paradigm

Prevention: The First Line of Defense

Detection and Remediation: The Safety Net

What Is Infrastructure Drift?

Pulumi’s Drift Detection and Remediation Workflow

1. Drift Detection with pulumi refresh

2. State Reconciliation with pulumi refresh

3. Drift Remediation with pulumi up

Automating Drift Detection with Pulumi Deployments

Setting Up Automated Drift Detection

Step 1: Configure Deployment Settings

Step 2: Create Drift Detection Schedules

Step 3: Configure Webhooks for Notifications

Handling Common Drift Scenarios

Scenario 1: Emergency Hotfix

Scenario 2: Legitimate External Changes

Scenario 3: Configuration Drift

The Value of Automated Drift Detection

Getting Started with Drift Detection

Quick Start Checklist

Conclusion: Day 2 Operations as a Competitive Advantage

Amazon EKS Hybrid Nodes: Bridging Cloud and On-Premises

Automation and Operations

Summary

The Guide to Platform Engineering: 7 Steps to Get It Right

On this article:

Step 1: Securing Executive Buy-in

Step 2: Staffing the Platform Engineering Team

Step 3: Defining the Platform Mandate

Step 4: Implementing the Pre-Production Pipeline

Step 5: Embracing Infrastructure as Code

Step 6: Implementing Policy as Code

Step 7: Evolving Towards Self-Service

Conclusion: Embracing the Platform Engineering Journey

Frequently Asked Questions

What is Platform Engineering?

Who are the Platform Customers?

What is an Internal Developer Platform (IDP)?

What is a Platform Team?

What is a Self-Service Infrastructure?

What is Infrastructure as Code (IaC)?

What is Policy as Code (PaC)?

What is Developer Experience (DevEx)?

Pulumi and Redis Cloud: Real-Time Data for Modern Apps

About Redis Enterprise Cloud

Architecture

Project Initialization and Redis Enterprise Cloud Resources

AWS Networking Resources

Creating an EC2 Instance to Test

Testing the Architecture

Next Steps and Conclusion

A Tour of the Pulumi Equinix Provider

Introducing the Equinix Provider

Creating a Kubernetes cluster on Equinix Metal

Deploying a workload with the Pulumi Kubernetes provider

Adding storage

Conclusion

Advanced AWS Networking: Part 2

Introduction

Spoke VPC

Creating a Sample Spoke VPC Workload

Testing Centralized Egress

Adding AWS Network Firewall

Conclusion

AWS Lambda Response Streaming with Pulumi

Introducing Response Streaming

Authoring a Lambda Function for Response Streaming

Streaming Performance Considerations

Packaging and deploying your Lambda Function for streaming with Pulumi

Reading a Response Stream

Wrapping up

Advanced AWS Networking: Part 1

What is a hub-and-spoke network?

Why use a hub-and-spoke network with centralized egress?

Creating the first Transit Gateway resources

Creating the inspection VPC

Final steps

1. Drift Detection with `pulumi refresh`

2. State Reconciliation with `pulumi refresh`

3. Drift Remediation with `pulumi up`

Mapping from Kubernetes manifests to `pulumi import` JSON