Pulumi Blog: Networking

Cloudflare-First Networking as Code with Pulumi

Pablo Seibelt — Tue, 23 Jun 2026 00:00:00 +0000

Platform teams managing multi-cloud applications face a dangerous visibility gap. While origin infrastructure is tightly controlled, the edge configuration often drifts through manual console tweaks. DNS records point to stale origins, WAF rules are inconsistent across environments, and Zero Trust policies fail to keep pace with team changes. This edge drift leads to application exposure or routing failures that origin teams only notice after users report them.

As applications span multiple clouds, the edge often becomes the most consistent layer for enforcing security and traffic policies. The cost of waiting for a unified edge strategy is high. Every manual change is a potential security hole or a performance bottleneck that bypasses your standard CI/CD rigor.

What you’ll build

In this post, you will build a standardized Cloudflare edge baseline. You’ll use Pulumi to define:

DNS records to point to your multi-cloud origins.
WAF custom rules to block malicious traffic before it hits your network.
A Worker canary to handle edge logic and header validation.
Zero Trust Access policies to secure internal tools and origin access.

By the end, you will have a version-controlled edge that you can validate and deploy across any environment.

The Pulumi program

We’ll build a Pulumi program that sets up a complete edge environment. This approach ensures that your edge security and routing are always in sync with your application code.

Setting up the provider

First, ensure you have the @pulumi/cloudflare package installed and your credentials configured. The examples below are checked against @pulumi/cloudflare@6.15.0, the current v6 Pulumi Registry API at the time of writing.

npm install @pulumi/cloudflare@6.15.0
pulumi config set --secret cloudflare:apiToken "$CLOUDFLARE_API_TOKEN"

Defining the infrastructure

Here is the complete Pulumi program in TypeScript. If your zone already has a custom WAF ruleset for the same phase, import that ruleset or add the rule to your existing ruleset instead of creating another phase-level ruleset that may fail or conflict.

import * as cloudflare from "@pulumi/cloudflare";

const accountId = "<your-cloudflare-account-id>";
const trustedAdminIp = "198.51.100.10";

const zone = new cloudflare.Zone("my-zone", {
 account: {
 id: accountId,
 },
 name: "example.com",
});

const www = new cloudflare.DnsRecord("www", {
 zoneId: zone.id,
 name: "www",
 content: "1.2.3.4",
 ttl: 1,
 type: "A",
 proxied: true,
});

const admin = new cloudflare.DnsRecord("admin", {
 zoneId: zone.id,
 name: "admin",
 content: "1.2.3.4",
 ttl: 1,
 type: "A",
 proxied: true,
});

const wafRule = new cloudflare.Ruleset("waf-rule", {
 zoneId: zone.id,
 name: "Block suspicious traffic",
 description: "Custom WAF rule to block specific patterns",
 kind: "zone",
 phase: "http_request_firewall_custom",
 rules: [{
 action: "block",
 expression: `(http.request.uri.path contains "/admin" and not ip.src in {${trustedAdminIp}})`,
 description: "Block unauthorized admin access",
 }],
});

const worker = new cloudflare.WorkersScript("canary-worker", {
 accountId: accountId,
 scriptName: "edge-canary",
 compatibilityDate: "2026-01-01",
 mainModule: "worker.js",
 content: `
 export default {
 async fetch(request) {
 return new Response("Edge Baseline Validated", {
 headers: { "x-edge-baseline": "active" },
 });
 },
 };
 `,
});

const workerRoute = new cloudflare.WorkersRoute("canary-route", {
 zoneId: zone.id,
 pattern: "www.example.com/*",
 script: worker.scriptName,
});

const adminGroup = new cloudflare.ZeroTrustAccessGroup("admin-group", {
 accountId: accountId,
 name: "Administrators",
 includes: [{
 email: {
 email: "admin@example.com",
 },
 }],
});

const accessApp = new cloudflare.ZeroTrustAccessApplication("internal-tool", {
 accountId: accountId,
 name: "Internal Admin Tool",
 domain: "admin.example.com",
 type: "self_hosted",
 policies: [{
 name: "Allow administrators",
 precedence: 1,
 decision: "allow",
 includes: [{
 group: {
 id: adminGroup.id,
 },
 }],
 }],
});

Validation

Once you run pulumi up, you can validate your edge baseline with these steps:

DNS check: Run dig www.example.com to confirm the record points to Cloudflare’s proxied IPs.
WAF test: Attempt to access www.example.com/admin from an unauthorized IP and verify you receive a 403 Forbidden response.
Worker canary: Use curl -I https://www.example.com and check for the x-edge-baseline: active header.
Access policy: Navigate to admin.example.com and verify you are redirected to the Cloudflare Access login page.

By managing these resources as code, you eliminate the risk of manual drift and ensure that every environment starts with a secure, validated edge baseline.

How to Move to the Gateway API: post ingress-nginx Retirement

Engin Diri — Fri, 02 Jan 2026 11:35:51 +0100

The upcoming retirement of ingress-nginx in early 2026 gives infrastructure teams both a deadline and an opportunity to rethink traffic management. Configuring the Ingress API often meant relying on controller-specific annotations that varied between implementations. The Gateway API offers a cleaner, standardized alternative. This post investigates the practical reality of this migration and explores why kgateway emerges as a robust solution for the future.

With ingress-nginx entering its sunset phase in early 2026, the Kubernetes community faces a decision point. While the controller served as the default standard for a decade, its architecture now struggles to meet modern requirements. The transition to the Gateway API offers a chance to adopt a standard designed for contemporary traffic patterns.

The Gateway API addresses the portability issues of its predecessor by establishing a standardized, expressive approach that behaves consistently across implementations. A technical evaluation of the available options points to kgateway as a particularly strong candidate for production workloads.

Why ingress-nginx is retiring

The Kubernetes SIG Network and Security Response Committee was direct in its announcement regarding the project’s future. After March 2026, ingress-nginx will cease to receive releases, bug fixes, or security patches. The repositories will become read-only, leaving existing deployments functional but unmaintained.

This decision stems from both resource constraints and technical debt. The project relied on a very small group of maintainers working primarily in their spare time. Furthermore, features that once provided flexibility, such as “snippets” for arbitrary NGINX configuration injection, now pose significant security liabilities. With no path to modernize the codebase, retirement was inevitable.

It is worth noting that the Ingress API itself remains supported. The retirement affects only the ingress-nginx controller. NGINX as a web server continues unchanged. However, clusters relying on this specific controller must prepare for a transition.

To identify affected resources, checking the cluster for specific pods can reveal the scope of the dependency:

kubectl get pods --all-namespaces --selector app.kubernetes.io/name=ingress-nginx

Understanding the Gateway API

The Gateway API represents a fundamental shift in traffic management concepts. Some elements will look familiar to Ingress users, but the underlying philosophy addresses the “baggage” that the previous standard accumulated.

The core improvement lies in expressiveness. Advanced routing requirements (header matching, weighted traffic splitting, and traffic policies) are now native parts of the specification. This eliminates the need for the non-portable annotations that plagued the Ingress ecosystem.

The API also introduces a role-oriented structure that mirrors actual organizational workflows.

GatewayClass resources are managed by infrastructure teams to define the underlying controller.
Gateway resources are created by platform teams to specify entry points and listeners.
HTTPRoute resources are owned by application teams to define service traffic rules.

This separation allows for genuine self-service. Application teams can manage their routing logic without requiring broad cluster privileges.

The relationship between these resources defines the traffic flow:

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
 name: kgateway
spec:
 controllerName: kgateway.dev/kgateway

A Gateway references this class to establish the entry point:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
 name: my-gateway
 namespace: default
spec:
 gatewayClassName: kgateway
 listeners:
 - name: http
 port: 8080
 protocol: HTTP
 allowedRoutes:
 namespaces:
 from: All

An HTTPRoute then attaches to the gateway:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: my-route
 namespace: default
spec:
 parentRefs:
 - name: my-gateway
 hostnames:
 - "example.com"
 rules:
 - matches:
 - path:
 type: PathPrefix
 value: /api
 backendRefs:
 - name: api-service
 port: 8080

Beyond standard HTTP, the API supports GRPCRoute, TCPRoute, UDPRoute, and TLSRoute, offering a protocol diversity that the original Ingress spec lacked.

Cross-namespace routing and ReferenceGrant

The Gateway API enables a security model where routes can cross namespace boundaries, but only with explicit permission. The ReferenceGrant resource controls these connections. Without a grant, an HTTPRoute in one namespace cannot reference a Service in another.

apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
 name: allow-routes-from-apps
 namespace: gateway-system
spec:
 from:
 - group: gateway.networking.k8s.io
 kind: HTTPRoute
 namespace: my-app
 to:
 - group: ""
 kind: Service

This mechanism allows platform teams to strictly control which namespaces can attach to shared gateways.

Evaluating Gateway API implementations

Several implementations have emerged to support the new standard, each with a distinct philosophy.

NGINX Gateway Fabric leverages NGINX as the data plane, offering continuity for teams in the F5 ecosystem. It provides impressive throughput and integrates with enterprise security tools, making it a logical choice for existing NGINX shops.

Traefik focuses on simplicity. As a single binary with no external dependencies, it aligns well with environments that prioritize developer experience and rapid iteration. Its declarative configuration is particularly friendly to GitOps workflows.

Envoy Gateway is built on the Envoy Proxy and operates under a community-driven governance model. It appeals to those seeking strong conformance to the Gateway API spec without vendor lock-in. Its companion project, Envoy AI Gateway, adds support for LLM routing.

kgateway, formerly Gloo Gateway, brings seven years of production history to the table. Donated to the CNCF in early 2025, it combines an Envoy data plane with a control plane optimized for scale. Its internal architecture uses the “krt” framework to handle massive route tables efficiently, avoiding the performance bottlenecks often seen in snapshot-based systems.

Aspect	NGINX Gateway Fabric	Traefik	Envoy Gateway	kgateway
Base Technology	NGINX (C)	Custom Go	Envoy Proxy (C++)	Envoy Proxy (C++)
Maturity	Production-ready	Mature	GA (v1.0+)	Battle-tested (7+ years as Gloo)
Setup Complexity	Medium	Low	Low	Low
Commercial Support	F5 Enterprise	Traefik Labs	Multiple vendors	Solo.io
AI/LLM Support	No	No	Via Envoy AI Gateway	Native (Agentgateway)
Learning Curve	Familiar for NGINX users	Gentle	Moderate	Moderate
Community Size	Growing	Large	Large	Established
Best For	Teams with NGINX expertise, high-throughput requirements	Teams prioritizing simplicity and rapid deployment	Standards-focused teams, multi-vendor environments	Enterprise scale, AI workloads, Istio integration

Decision framework

NGINX Gateway Fabric suits teams needing F5 support or deep NGINX tuning.
Traefik fits best where simplicity and operational ease are paramount.
Envoy Gateway works for those valuing community governance and strict standards adherence.
kgateway is the choice for enterprise scale, native AI gateway needs, and Istio integration.

This exploration uses kgateway for its proven track record and native handling of both traditional microservices and AI traffic.

Migrating from Ingress to Gateway API

The official migration guide outlines the transition process. It’s less translation, more restructuring.

The process typically involves:

Defining a Gateway resource: Unlike Ingress, listeners must be explicitly defined. This provides granular control over ports and protocols.
Creating HTTPRoute resources: These replace Ingress routing rules. You can split complex Ingress resources into multiple HTTPRoutes for better management.
Configuring filters: Annotations for headers or redirects are replaced by standardized filters within the HTTPRoute spec.

The ingress2gateway tool can automate much of the initial conversion, providing a baseline of resources to review and refine.

Migration planning

Scope the migration effort first:

1. Assessment Identify the volume and complexity of existing resources.

# Count your Ingress resources across all namespaces
kubectl get ingress -A | wc -l

# List all Ingress resources with their hosts
kubectl get ingress -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}: {.spec.rules[*].host}{"\n"}{end}'

2. Parallel Operations Running kgateway alongside ingress-nginx allows for incremental migration. Services can be moved one by one, validating the new configuration without risking the entire cluster’s traffic.

3. Rollback Strategy Keeping the original Ingress manifests ensures that traffic can be quickly reverted to the old controller if issues arise during the transition.

Why kgateway stands out

kgateway distinguishes itself through its maturity and architectural decisions. Having started as Gloo Gateway in 2018, it has hardened through years of production use.

Its control plane uses the krt framework, originally developed for Istio. This allows it to track dependencies precisely and recalculate only the parts of the configuration that have changed. In large clusters where pods churn constantly, this incremental approach prevents the control plane from becoming a bottleneck, enabling it to handle over 10,000 routes efficiently.

For teams managing AI workloads, kgateway includes the Agentgateway component. This Rust-based data plane is built for LLM traffic, supporting the Model Context Protocol (MCP) and providing token-based rate limiting. It unifies the management of AI and traditional traffic under a single control plane.

Additional features include:

Route Delegation: Allows large routing tables to be split across teams with clear inheritance.
Security Integration: Seamless mTLS with Istio ambient mesh and external authorization support.
Traffic Management: Advanced capabilities like traffic mirroring and session affinity are available without complex Envoy configuration.

Putting it all together with Pulumi

Seeing these concepts in code clarifies the architecture. The following Pulumi program demonstrates a complete setup: a DigitalOcean Kubernetes cluster, the kgateway installation via Helm, Gateway API resources, and an httpbin application to validate the configuration.

The program provisions infrastructure in a specific sequence. The cluster comes first, followed by the Gateway API CRDs, then kgateway itself. Once the control plane is running, the program creates a GatewayClass, a Gateway listener, and an HTTPRoute that directs traffic to the sample application. A ReferenceGrant permits the cross-namespace reference between the HTTPRoute and the backend Service.

import * as pulumi from "@pulumi/pulumi";
import * as digitalocean from "@pulumi/digitalocean";
import * as k8s from "@pulumi/kubernetes";

// Configuration
const config = new pulumi.Config();
const clusterName = config.get("clusterName") || "kgateway-demo-cluster";
const region = config.get("region") || "fra1";
const nodeSize = config.get("nodeSize") || "s-2vcpu-4gb";
const nodeCount = config.getNumber("nodeCount") || 2;
const k8sVersion = config.get("k8sVersion") || "1.32.10-do.2";

// Create a DigitalOcean Kubernetes cluster
const cluster = new digitalocean.KubernetesCluster(clusterName, {
 name: clusterName,
 region: region,
 version: k8sVersion,
 nodePool: {
 name: "default-pool",
 size: nodeSize,
 nodeCount: nodeCount,
 },
});

// Create a Kubernetes provider using the cluster's kubeconfig
const k8sProvider = new k8s.Provider("k8s-provider", {
 kubeconfig: cluster.kubeConfigs[0].rawConfig,
});

// Install Gateway API CRDs (standard channel)
const gatewayApiCrds = new k8s.yaml.ConfigFile("gateway-api-crds", {
 file: "https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.0/standard-install.yaml",
}, { provider: k8sProvider });

// Create kgateway-system namespace
const kgatewayNamespace = new k8s.core.v1.Namespace("kgateway-system", {
 metadata: {
 name: "kgateway-system",
 },
}, { provider: k8sProvider });

// Install kgateway CRDs via Helm
const kgatewayCrds = new k8s.helm.v3.Release("kgateway-crds", {
 chart: "oci://cr.kgateway.dev/kgateway-dev/charts/kgateway-crds",
 version: "v2.1.2",
 namespace: kgatewayNamespace.metadata.name,
 createNamespace: false,
}, { provider: k8sProvider, dependsOn: [gatewayApiCrds, kgatewayNamespace] });

// Install kgateway control plane via Helm
const kgateway = new k8s.helm.v3.Release("kgateway", {
 chart: "oci://cr.kgateway.dev/kgateway-dev/charts/kgateway",
 version: "v2.1.2",
 namespace: kgatewayNamespace.metadata.name,
 createNamespace: false,
}, { provider: k8sProvider, dependsOn: [kgatewayCrds] });

// Create hello-world namespace
const helloWorldNamespace = new k8s.core.v1.Namespace("hello-world", {
 metadata: {
 name: "hello-world",
 },
}, { provider: k8sProvider });

// Deploy hello world application (using httpbin as shown in kgateway docs)
const helloWorldLabels = { app: "httpbin" };

const helloWorldDeployment = new k8s.apps.v1.Deployment("httpbin", {
 metadata: {
 name: "httpbin",
 namespace: helloWorldNamespace.metadata.name,
 },
 spec: {
 replicas: 1,
 selector: {
 matchLabels: helloWorldLabels,
 },
 template: {
 metadata: {
 labels: helloWorldLabels,
 },
 spec: {
 containers: [{
 name: "httpbin",
 image: "mccutchen/go-httpbin:v2.15.0",
 ports: [{
 containerPort: 8080,
 name: "http",
 }],
 }],
 },
 },
 },
}, { provider: k8sProvider, dependsOn: [helloWorldNamespace] });

const helloWorldService = new k8s.core.v1.Service("httpbin", {
 metadata: {
 name: "httpbin",
 namespace: helloWorldNamespace.metadata.name,
 },
 spec: {
 selector: helloWorldLabels,
 ports: [{
 port: 8000,
 targetPort: 8080,
 name: "http",
 }],
 },
}, { provider: k8sProvider, dependsOn: [helloWorldDeployment] });

// Create GatewayClass for kgateway
const gatewayClass = new k8s.apiextensions.CustomResource("kgateway-gateway-class", {
 apiVersion: "gateway.networking.k8s.io/v1",
 kind: "GatewayClass",
 metadata: {
 name: "kgateway",
 },
 spec: {
 controllerName: "kgateway.dev/kgateway",
 },
}, { provider: k8sProvider, dependsOn: [kgateway] });

// Create Gateway with LoadBalancer service (matching kgateway docs pattern)
const gateway = new k8s.apiextensions.CustomResource("http-gateway", {
 apiVersion: "gateway.networking.k8s.io/v1",
 kind: "Gateway",
 metadata: {
 name: "http",
 namespace: kgatewayNamespace.metadata.name,
 },
 spec: {
 gatewayClassName: "kgateway",
 listeners: [{
 name: "http",
 port: 8080,
 protocol: "HTTP",
 allowedRoutes: {
 namespaces: {
 from: "All",
 },
 },
 }],
 },
}, { provider: k8sProvider, dependsOn: [gatewayClass] });

// Create HTTPRoute for httpbin application
const httpRoute = new k8s.apiextensions.CustomResource("httpbin-route", {
 apiVersion: "gateway.networking.k8s.io/v1",
 kind: "HTTPRoute",
 metadata: {
 name: "httpbin",
 namespace: helloWorldNamespace.metadata.name,
 },
 spec: {
 parentRefs: [{
 name: "http",
 namespace: kgatewayNamespace.metadata.name,
 }],
 hostnames: [
 "*.nip.io",
 ],
 rules: [{
 backendRefs: [{
 name: "httpbin",
 port: 8000,
 }],
 }],
 },
}, { provider: k8sProvider, dependsOn: [gateway, helloWorldService] });

// Create a reference grant to allow the HTTPRoute to reference the service cross-namespace
const referenceGrant = new k8s.apiextensions.CustomResource("httpbin-reference-grant", {
 apiVersion: "gateway.networking.k8s.io/v1beta1",
 kind: "ReferenceGrant",
 metadata: {
 name: "allow-gateway-to-httpbin",
 namespace: helloWorldNamespace.metadata.name,
 },
 spec: {
 from: [{
 group: "gateway.networking.k8s.io",
 kind: "HTTPRoute",
 namespace: helloWorldNamespace.metadata.name,
 }],
 to: [{
 group: "",
 kind: "Service",
 }],
 },
}, { provider: k8sProvider, dependsOn: [gatewayApiCrds] });

// Create a LoadBalancer Service for the Gateway proxy
const gatewayProxyService = new k8s.core.v1.Service("gateway-proxy", {
 metadata: {
 name: "gateway-proxy",
 namespace: kgatewayNamespace.metadata.name,
 },
 spec: {
 type: "LoadBalancer",
 selector: {
 "gateway.networking.k8s.io/gateway-name": "http",
 },
 ports: [{
 name: "http",
 port: 8080,
 targetPort: 8080,
 protocol: "TCP",
 }],
 },
}, { provider: k8sProvider, dependsOn: [gateway] });

// Extract the LoadBalancer IP from our managed Service
const gatewayIP = gatewayProxyService.status.apply(status => {
 const ingress = status?.loadBalancer?.ingress;
 if (ingress && ingress.length > 0) {
 return ingress[0].ip || ingress[0].hostname || "pending";
 }
 return "pending";
});

// Exports
export const clusterNameOutput = cluster.name;
export const clusterEndpoint = cluster.endpoint;
export const kubeconfig = pulumi.secret(cluster.kubeConfigs[0].rawConfig);

// Export the LoadBalancer IP and nip.io URL
export const gatewayLoadBalancerIP = gatewayIP;
export const httpbinUrl = gatewayIP.apply(ip =>
 ip !== "pending" ? `http://httpbin.${ip}.nip.io:8080/get` : "Waiting for LoadBalancer IP..."
);

The demo uses nip.io, a wildcard DNS service that resolves *.IP.nip.io hostnames to the specified IP address. This eliminates the need to configure separate DNS records during testing. Once deployed, the httpbinUrl output provides a ready-to-use endpoint for validating the gateway configuration.

Migrate to the Gateway API

You can provision your cluster, install kgateway, and define your Gateway API resources in one Pulumi program, then migrate services incrementally and review each change. Start a project to plan your migration.

Get started

The short-term option: Chainguard’s ingress-nginx fork

For organizations that cannot immediately migrate to the Gateway API, there is another path forward. Chainguard has forked ingress-nginx as part of their EmeritOSS program, providing continued maintenance after the upstream project’s retirement.

This fork is not about adding new features. Chainguard is explicit that they are maintaining stability, not continuing development. Their commitment includes keeping dependencies updated, addressing CVEs on a best-efforts basis, and providing commercial container images with low vulnerability counts and SLAs. FIPS-compliant versions are also available for regulated environments.

The chainguard-forks/ingress-nginx repository on GitHub provides the maintained codebase. For teams running ingress-nginx in production, this fork offers a viable bridge while evaluating the Gateway API or other alternatives. However, this buys time rather than solving the problem permanently. The architectural limitations of the Ingress API remain, and the eventual move to a more expressive standard like the Gateway API is still the recommended path forward.

Looking ahead

The retirement of ingress-nginx marks the end of an era and the beginning of a more structured approach to Kubernetes networking. The Gateway API offers a robust framework that reflects how teams actually build and secure applications today.

While March 2026 provides a comfortable runway, early planning prevents rushed decisions. Setting up a test environment with kgateway to validate the new API constructs is a prudent next step.

Moving to the Gateway API means building a networking foundation ready for the next decade of infrastructure challenges.

Get Started with Pulumi

Use Pulumi's open-source SDK to create, deploy, and manage infrastructure on any cloud.

Resources

Gateway API Documentation – Comprehensive guides and examples.
Migration Guide – Official steps for transitioning from Ingress.
ingress2gateway – Tooling to automate resource conversion.
kgateway Documentation – Installation and configuration details.
Designing kgateway for Scalability – Deep dive into the krt framework.
Kubernetes Blog: ingress-nginx Retirement – Full context on the deprecation.

Advanced AWS Networking: Part 2

Josh Kodroff — Thu, 20 Apr 2023 00:00:00 +0000

In this blog series, you will learn how to create a hub-and-spoke network architecture in AWS with centralized egress and traffic inspection. In this second installment, we’ll show you how to create spoke VPCs to run your workloads, verify centralized egress is working, and then add centralized traffic inspection using Pulumi, the infrastructure as code tool that enables you to manage infrastructure with real programming languages!

Introduction

In AWS Advanced Networking Part 1 you:

Learned about the benefits of a hub-and-spoke network architecture with centralized egress.
Created the shared AWS Transit Gateway resources to be shared among our spoke VPCs and inspection VPC.
Defined your inspection VPC as a Pulumi component resource.

Now, to finish the architecture, you will need to:

Create a spoke VPC as a Pulumi component resource.
Test the network architecture by verifying that centralized egress is working.
Add traffic inspection with AWS Network Firewall.

In subsequent sections we will be showing code samples. Because this is a more advanced Pulumi program, we will be focusing on the most important parts of the architecture rather than covering every single line in the codebase. The complete code for this project is available on GitHub, which is a great place to go for full context.

Spoke VPC

Like your inspection VPC, we will implement the spoke VPC as a Pulumi component resource. Component resources allow you to logically group resources to form reusable components. Because your network will contain multiple spoke VPCs (at least 2, possibly more), and the spoke VPC will contain both the basic VPC and connectivity to Transit Gateway, the spoke VPC is a natural fit for a component resource.

First, add a file spoke.py to your Pulumi program. In Python, it’s good practice to put each component resource along with its associated arguments class in a separate file from your main program as it makes for intuitive code organization:

@dataclass
class SpokeVpcArgs:
 vpc_cidr_block: str
 tgw_id: pulumi.Input[str]
 tgw_route_table_id: pulumi.Input[str]


class SpokeVpc(pulumi.ComponentResource):
 def __init__(self, name: str, args: SpokeVpcArgs, opts: pulumi.ResourceOptions = None) -> None:
 super().__init__("awsAdvancedNetworking:index:SpokeVpc", name, None, opts)

Then, create a VPC with two sets of subnets, both isolated (with no pathway to the internet). One set of subnets will hold our spoke VPC workloads, and the other will hold our Transit Gateway attachment. We enable DNS hostnames and support so that we can later use VPC endpoints to enable AWS Systems Manager access to verify that centralized egress is working as expected without requiring direct host access via SSH:

self.vpc = awsx.ec2.Vpc(
 f"{self._name}-vpc",
 awsx.ec2.VpcArgs(
 cidr_block=args.vpc_cidr_block,
 subnet_specs=[
 awsx.ec2.SubnetSpecArgs(
 name="private",
 cidr_mask=28,
 type=awsx.ec2.SubnetType.ISOLATED,
 ),
 awsx.ec2.SubnetSpecArgs(
 name="tgw",
 cidr_mask=28,
 type=awsx.ec2.SubnetType.ISOLATED,
 ),
 ],
 nat_gateways=awsx.ec2.NatGatewayConfigurationArgs(
 strategy=awsx.ec2.NatGatewayStrategy.NONE,
 ),
 enable_dns_hostnames=True,
 enable_dns_support=True,
 )
)

Then, attach the Transit Gateway to the appropriate subnets:

self.tgw_attachment = aws.ec2transitgateway.VpcAttachment(
 f"{name}-tgw-vpc-attachment",
 aws.ec2transitgateway.VpcAttachmentArgs(
 transit_gateway_id=args.tgw_id,
 subnet_ids=tgw_subnets.apply(lambda x: x.ids),
 vpc_id=self.vpc.vpc_id,
 transit_gateway_default_route_table_association=False,
 transit_gateway_default_route_table_propagation=False,
 tags={
 "Name": f"{name}",
 },
 ),
 pulumi.ResourceOptions(
 delete_before_replace=True,
 depends_on=[self.vpc],
 parent=self,
 )
)

Add VPC endpoints to allow Systems Manager access to open an SSH connection from the AWS console without having to allow network access to the hosts. Because Pulumi uses real programming languages, you can use a Python for loop to create these resources with fewer lines of code using familiar syntax:

for service in ["ec2messages", "ssmmessages", "ssm"]:
 aws.ec2.VpcEndpoint(
 f"{self._name}-endpoint-{service}",
 aws.ec2.VpcEndpointArgs(
 vpc_id=self.vpc.vpc_id,
 service_name=f"com.amazonaws.{aws.config.region}.{service}",
 private_dns_enabled=True,
 security_group_ids=[vpc_endpoint_sg.id],
 vpc_endpoint_type="Interface",
 tags={
 "Name": f"{self._name}-{service}"
 },
 subnet_ids=subnet_ids
 )
 )

Add routes to the spoke VPC’s Transit Gateway route table that direct all non-local traffic to the Transit Gateway:

for subnet_id in private_subnet_ids:
 route_table = aws.ec2.get_route_table(
 subnet_id=subnet_id,
 )

 aws.ec2.Route(
 f"spoke{self._name}-tgw-route-{subnet_id}",
 aws.ec2.RouteArgs(
 route_table_id=route_table.id,
 destination_cidr_block="0.0.0.0/0",
 transit_gateway_id=self._args.tgw_id,
 ),
 pulumi.ResourceOptions(
 depends_on=[self.tgw_attachment],
 parent=self,
 ),
 )

Finally, add some outputs for the component. You’ll use these to place workloads in the proper subnets:

self.register_outputs({
 "vpc": self.vpc,
 "workload_subnet_ids": self.workload_subnet_ids
})

You can instantiate a spoke VPC like the following in __main__.py:

spoke1_vpc = SpokeVpc(
 "spoke1",
 SpokeVpcArgs(
 vpc_cidr_block="10.0.0.0/16",
 tgw_id=tgw.id,
 tgw_route_table_id=spoke_tgw_route_table.id,
 ),
)

Subsequent spoke VPCs should use a separate CIDR block:

spoke2_vpc = SpokeVpc(
 "spoke2",
 SpokeVpcArgs(
 vpc_cidr_block="10.1.0.0/16",
 tgw_id=tgw.id,
 tgw_route_table_id=spoke_tgw_route_table.id,
 ),
)

Now that you have your spoke VPCs added, it’s time to test that the architecture is working as expected. To do this, we’ll create a sample workload and deploy it to one of our spoke VPCs.

Creating a Sample Spoke VPC Workload

To test that internet traffic from your spoke VPC is going over the inspection VPC’s NAT gateway, you’ll add a workload (i.e., an EC2 instance running Amazon Linux) to your spoke VPC and verify that its public IP is that of the Elastic IP attached to the NAT gateway in the inspection VPC. Like the spoke VPC workload, the spoke VPC itself is a natural fit for a Pulumi component resource.

Your component should go in its own file spoke_workload.py, and will begin much like the spoke VPC component:

@dataclass
class SpokeWorkloadArgs:
 spoke_vpc_id: pulumi.Input[str]
 spoke_instance_subnet_id: str


class SpokeWorkload(pulumi.ComponentResource):
 def __init__(self, name: str, args: SpokeWorkloadArgs, opts: pulumi.ResourceOptions = None) -> None:
 super().__init__("awsAdvancedNetworking:index:SpokeWorkload", name, None, opts)

Start by creating a security group to allow outbound traffic. Note that this security group’s egress rules are overly permissive, but this is ok because the spoke workload is an ephemeral resource that only exists to verify network connectivity. In order for SSM access to work though, you must at least enable port 443:

sg = aws.ec2.SecurityGroup(
 f"{name}-instance-sg",
 aws.ec2.SecurityGroupArgs(
 description="Allow all outbound traffic",
 vpc_id=args.spoke_vpc_id,
 egress=[
 aws.ec2.SecurityGroupEgressArgs(
 cidr_blocks=["0.0.0.0/0"],
 description="Allow everything",
 protocol="-1",
 from_port=0,
 to_port=0
 ),
 ]
 ),
 opts=pulumi.ResourceOptions(
 parent=self
 ),
)

In addition to the security group which provides the necessary network access, you will also need to assign an EC2 instance role in order to complete the prerequisites for Systems Manager access to an EC2 instance which you’ll create shortly:

ec2_role = aws.iam.Role(
 f"{name}-instance-role",
 aws.iam.RoleArgs(
 assume_role_policy=json.dumps({
 "Version": "2012-10-17",
 "Statement": {
 "Effect": "Allow",
 "Principal": {
 "Service": "ec2.amazonaws.com",
 },
 "Action": "sts:AssumeRole",
 },
 })
 )
)

aws.iam.RolePolicyAttachment(
 f"{name}-role-policy-attachment",
 aws.iam.RolePolicyAttachmentArgs(
 role=ec2_role.name,
 policy_arn="arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
 )
)

instance_profile = aws.iam.InstanceProfile(
 f"{name}-instance-profile",
 aws.iam.InstanceProfileArgs(
 role=ec2_role.name,
 )
)

Finally, create an EC2 instance with the instance profile you created:

aws.ec2.Instance(
 f"{name}-instance",
 aws.ec2.InstanceArgs(
 ami=amazon_linux_2.id,
 instance_type="t3.micro",
 vpc_security_group_ids=[sg.id],
 subnet_id=args.spoke_instance_subnet_id,
 tags={
 "Name": f"{name}-instance",
 },
 iam_instance_profile=instance_profile.name,
 ),
 opts=pulumi.ResourceOptions(
 parent=self
 ),
)

Instantiate the workload in __main__.py:

spoke1_workload = SpokeWorkload(
 "spoke1",
 SpokeWorkloadArgs(
 spoke_instance_subnet_id=spoke1_vpc.workload_subnet_ids[0],
 spoke_vpc_id=spoke1_vpc.vpc.vpc_id,
 )
)

And deploy:

pulumi up

Now you have a workload with which you can verify that centralized egress is working!

Testing Centralized Egress

You can now test that centralized egress is working as expected. First, log into the AWS console and go the EC2 service homepage. Find your instance in the list of EC2 instances:

First, take note of the NAT gateway IP address which you included as a Pulumi stack output:

Then, in the AWS console, locate an instance running in a spoke VPC which you created as part of the workload component resource, and select Connect from the drop-down menu:

Connect to the instance to start an SSH session (note that the IAM principal must have permissions to perform SSM Systems Manager actions in order to connect):

Finally, verify your public IP address from the host in the spoke VPC:

If your public IP address for the instance in the spoke VPC matches the public IP address of the NAT gateway in the inspection VPC, you’ve successfully verified that centralized egress is working!

Adding AWS Network Firewall

AWS Network Firewall incurs significant costs (hundreds of USD per month). Be sure to tear down AWS Network Firewall and all other resources in this architecture with the pulumi destroy command to avoid incurring unwanted AWS charges.

Now that you’ve verified centralized egress is working, you can add centralized traffic inspection with AWS Network Firewall. In a new file called firewall_rules.py, add a rule to drop all traffic not destined for amazon.com:

allow_amazon = aws.networkfirewall.RuleGroup(
 "allow-amazon",
 aws.networkfirewall.RuleGroupArgs(
 capacity=100,
 name="allow-amazon",
 type="STATEFUL",
 rule_group=aws.networkfirewall.RuleGroupRuleGroupArgs(
 rules_source=aws.networkfirewall.RuleGroupRuleGroupRulesSourceArgs(
 rules_string='pass tcp any any <> $EXTERNAL_NET 443 (msg:"Allowing TCP in port 443"; flow:not_established; sid:892123; rev:1;)\n' +
 'pass tls any any -> $EXTERNAL_NET 443 (tls.sni; dotprefix; content:".amazon.com"; endswith; msg:"Allowing .amazon.com HTTPS requests"; sid:892125; rev:1;)'
 ),
 stateful_rule_options={
 "rule_order": "STRICT_ORDER",
 },
 )
 )
)

And add the rule to a firewall policy:

policy = aws.networkfirewall.FirewallPolicy(
 "firewall-policy",
 aws.networkfirewall.FirewallPolicyArgs(
 firewall_policy=aws.networkfirewall.FirewallPolicyFirewallPolicyArgs(
 stateless_default_actions=["aws:forward_to_sfe"],
 stateless_fragment_default_actions=["aws:forward_to_sfe"],
 stateful_default_actions=[
 "aws:drop_strict", "aws:alert_strict"],
 stateful_engine_options={
 "rule_order": "STRICT_ORDER"
 },
 stateless_rule_group_references=[{
 "priority": 10,
 "resource_arn": drop_remote.arn
 }],
 stateful_rule_group_references=[
 {
 "priority": 10,
 "resource_arn": allow_icmp.arn,
 },
 {
 "priority": 20,
 "resource_arn": allow_amazon.arn,
 },
 ]
 )
 )
)

Then, in inspection_vpc.py, you’ll need to add some additional network resources to create the firewall and route your egress traffic through the firewall. First create some subnets to contain AWS Network Firewall:

for i, inspection_subnet in enumerate(inspection_subnets):
 resource_name = f"{self.name}-inspection-{i+1}"
 subnet = aws.ec2.Subnet(
 resource_name,
 aws.ec2.SubnetArgs(
 vpc_id=self.vpc.vpc_id,
 availability_zone=inspection_subnet["az"],
 cidr_block=inspection_subnet["cidr"],
 tags={
 "Name": resource_name
 }
 ),
 opts=pulumi.ResourceOptions(
 parent=self,
 delete_before_replace=True,
 ),
 )

After creating route tables and associating with your subnets, create routes that send internal traffic to the TGW and external traffic to the NAT gateway:

aws.ec2.Route(
 f"{self.name}-insp-supernet-to-tgw-{i+1}",
 aws.ec2.RouteArgs(
 route_table_id=route_table.id,
 destination_cidr_block=self.args.supernet_cidr_block,
 transit_gateway_id=self.args.tgw_id
 ),
 opts=pulumi.ResourceOptions(
 parent=route_table,
 ),
)

aws.ec2.Route(
 f"{self.name}-insp-default-to-nat-{i+1}",
 aws.ec2.RouteArgs(
 route_table_id=route_table.id,
 destination_cidr_block="0.0.0.0/0",
 nat_gateway_id=self.nat_gateway.id
 ),
 opts=pulumi.ResourceOptions(
 parent=route_table,
 ),
)

Then create the AWS Network Firewall itself and deploy it to the subnets you created. AWS Network Firewall creates a VPC endpoint in each subnet to which it’s attached:

self.firewall = aws.networkfirewall.Firewall(
 f"{self.name}-firewall",
 aws.networkfirewall.FirewallArgs(
 vpc_id=self.vpc.vpc_id,
 firewall_policy_arn=self.args.firewall_policy_arn,
 subnet_mappings=subnet_mappings,
 ),
 opts=pulumi.ResourceOptions(
 parent=self,
 ),
)

You’ll also need to add some routes. In the public subnets of the inspection VPC, you’ll need to route all traffic bound for the spoke VPCs (i.e., the return traffic from the internet) through AWS Network Firewall’s generated VPC endpoints:

for subnet_id in public_subnet_ids:
 # ...

 aws.ec2.Route(
 f"{self.name}-{subnet_id}-to-firewall",
 aws.ec2.RouteArgs(
 route_table_id=route_table.id,
 destination_cidr_block=self.args.supernet_cidr_block,
 vpc_endpoint_id=subnet_attachments[0]["endpoint_id"],
 ),
 pulumi.ResourceOptions(
 parent=self,
 ),
 )

In the inspection VPC subnets with the Transit Gateway attachment, you’ll need to route all traffic bound for the internet through AWS Network Firewall. Previously, these routes went directly to the NAT gateway:

for subnet_id in tgw_subnet_ids:
 # ...

 aws.ec2.Route(
 f"{self.name}-{subnet_id}-to-firewall",
 aws.ec2.RouteArgs(
 route_table_id=route_table.id,
 destination_cidr_block="0.0.0.0/0",
 vpc_endpoint_id=subnet_attachments[0]["endpoint_id"],
 ),
 pulumi.ResourceOptions(
 parent=self,
 ),
 )

In order to test that our firewall rules are in effect, connect to one of your spoke EC2 instances like before. Traffic to amazon.com should be allowed:

But traffic to other domains should be blocked:

Conclusion

The hub-and-spoke architecture gives us several key advantages when our organization has multiple VPCs: cost savings via centralized egress and the ability to provide traffic inspection and apply network policies in a single place. While this architecture can be difficult to implement due to the resources needed, Pulumi’s approach of enabling practitioners to use real programming languages makes it easier to define complex, real-world infrastructure.

If you enjoyed this content, sign up for an upcoming workshop to learn more about how Pulumi can help you tame the complexity of the modern cloud, and join our community in the Pulumi Community Slack!

Advanced AWS Networking: Part 1

Josh Kodroff — Wed, 05 Apr 2023 00:00:00 +0000

In this blog series you will learn how to create a hub-and-spoke network architecture in AWS with centralized egress and traffic inspection. In this first installment, we’ll talk about the benefits of this architecture and begin to lay out some of its parts in Python with Pulumi, the infrastructure as code tool that enables you to manage infrastructure with real programming languages!

The complete code for this project is available on GitHub, which is a great place to go for full context.

What is a hub-and-spoke network?

A hub-and-spoke network is a common architecture for creating a network topology that provides isolation and security for your workloads. The hub-and-spoke architecture you’ll be creating on AWS has three main components: an inspection VPC, AWS Transit Gateway, and a series of spoke VPCs:

The inspection VPC provides centralized egress. It is the only VPC that has a route to the internet, so all other VPCs in the architecture must route their traffic through the inspection VPC. In part II of this blog post, you’ll add AWS Firewall to the inspection VPC to allow all traffic to the internet to be governed by a single set of policies.
Network connectivity between VPCs is accomplished via AWS Transit Gateway. The Transit Gateway maintains a central routing table that is used to route traffic from the spoke VPCs to the internet. We also need to maintain routes so that return traffic from the internet can be routed back to the correct spoke VPC.
The spoke VPCs are where we run our application workloads. They are isolated from each other and cannot communicate with each other unless we explicitly allow a network path. They will be able to communicate with the internet by default, but only through the inspection VPC’s NAT gateways.

A fully fleshed-out version of the architecture, with multiple spoke VPCs follows:

We could also enable communication between the spoke VPCs by adding routes to the Transit Gateway routing table. Traffic between spoke VPCs would not be desirable for security reasons if those VPCs represent different environments of the same workload (e.g., QA, UAT, and production). Traffic between spoke VPCs would potentially be desirable if those VPCs represent multiple workloads in the same environment, for example, our production online store and our production ERP software.

For simplicity, we will not be enabling routes between spoke VPCs in this architecture.

For a full discussion of all the considerations of building multi-VPC networks on AWS, see the excellent AWS whitepaper Building a Scalable and Secure Multi-VPC AWS Network Infrastructure.

Why use a hub-and-spoke network with centralized egress?

There’s several advantages to this architecture:

Cost savings: We save costs by allowing all of our spoke VPCs to share a single set of highly-available NAT gateways in the inspection VPC. There are costs to sending our traffic over AWS Transit Gateway, but in most scenarios Transit Gateway costs are significantly less than the cost of running NAT gateways in each spoke VPC.
Stable public IP addresses: Because all of our egress traffic to the internet originate from the same set of stable, public elastic IP addresses (attached to the NAT gateways in the inspection VPC), we can more easily interface with systems that use high-trust, network-based security models (i.e., IP allowlists). Network administrators for the systems we want to interface with only need allow a limited number of IPs to establish connectivity once and up-front, as opposed to having to allow new IPs for every VPC we create. These high-trust systems are often based in the data center and may have small and infrequent change windows, so only having to go through the change management process once, not for every spoke VPC we add, is a big win.
Centralized traffic inspection: When all traffic to the internet flows through our inspection VPC, we only need to apply a firewall and associated rules in a single place. This way, we can easily ensure that all network traffic to the internet complies with our organization’s security policies. (In the part of this post, we’ll add in AWS Firewall to our inspection VPC to add centralized traffic inspection capabilities to our architecture.)

Creating the first Transit Gateway resources

After initializing your Pulumi project and adding the necessary references to the AWS and AWSX, you can add the resources relating to your Transit Gateway.

You’ll need to configure a CIDR block for your network’s supernet. All spoke VPCs in this codebase must fall under the supernet CIDR block. You will use the supernet CIDR to create a route in the inspection VPC that will route traffic returning from the internet to the Transit Gateway. The Transit Gateway will, in turn, route the traffic back to the specific originating spoke VPC.

You can get the supernet CIDR from the Pulumi configuration. If the configuration value is not set, you can default to a CIDR block of 10.0.0.0/8:

config = pulumi.Config()
supernet_cidr = config.get("supernet-cidr") or "10.0.0.0/8"

Now you can add some of your initial Transit Gateway resources. You’ll start with the Transit Gateway itself along with some route tables for your inspection and spoke VPCs, respectively. You will add routes to this route table later to allow traffic from our spoke VPCs to the internet, as well as routes from the internet back to the originating spoke VPC:

tgw = aws.ec2transitgateway.TransitGateway(
 "tgw",
 aws.ec2transitgateway.TransitGatewayArgs(
 description=f"Transit Gateway - {project}",
 default_route_table_association="disable",
 default_route_table_propagation="disable",
 tags={
 "Name": "Pulumi"
 }
 )
)

inspection_tgw_route_table = aws.ec2transitgateway.RouteTable(
 "post-inspection-tgw-route-table",
 aws.ec2transitgateway.RouteTableArgs(
 transit_gateway_id=tgw.id,
 tags={
 "Name": "post-inspection",
 }
 ),
 # Adding the TGW as the parent makes the output of `pulumi up` a little
 # easier to understand as it groups these resources visually under the TGW
 # on which they depend.
 opts=pulumi.ResourceOptions(
 parent=tgw,
 ),
)

spoke_tgw_route_table = aws.ec2transitgateway.RouteTable(
 "spoke-tgw-route-table",
 aws.ec2transitgateway.RouteTableArgs(
 transit_gateway_id=tgw.id,
 tags={
 "Name": "spoke-tgw",
 }
 ),
 opts=pulumi.ResourceOptions(
 parent=tgw,
 ),
)

Creating the inspection VPC

The code samples in this section are not exhaustive. That is, they do not contain every line of code required so that we can focus on the most illustrative portions. See the GitHub repo for full context.

Now that you’ve added some foundational Transit Gateway resources, you can create your inspection VPC.

The inspection VPC is implemented as a Pulumi component resource. While we do not need to create multiple inspection VPCs in this codebase, implementing the inspection VPC as a component resource enhances the readability of both the code and the output by grouping all resources that belong to the inspection VPC in the output of pulumi up. You can also improve the component’s readability by placing its code in a separate file (in this case, inspection.py).

When you create a component resource in Python, it’s helpful to create a class that contains the arguments to the component resource. This allows you to pass the arguments to the component resource as a single object, which makes the code easier to read and understand. You can create a terse and easily readable arguments class in Python with the @dataclass decorator:

@dataclass
class InspectionVpcArgs:
 supernet_cidr_block: str
 vpc_cidr_block: str
 tgw_id: pulumi.Input[str]
 spoke_tgw_route_table_id: pulumi.Input[str]
 inspection_tgw_route_table_id: pulumi.Input[str]
 firewall_policy_arn: pulumi.Input[str]

Then, you can create the component resource class itself. The component resource class must inherit from pulumi.ComponentResource. The constructor for the component resource class must call the constructor of the base class, passing in the resource type (a string that uniquely identifies the component type, in the form of {project}:{module}:{component_name}), the resource name, and the resource options:

class InspectionVpc(pulumi.ComponentResource):
 def __init__(self, name: str, args: InspectionVpcArgs, opts: pulumi.ResourceOptions = None) -> None:
 super().__init__("awsAdvancedNetworking:index:InspectionVpc", name, None, opts)

You’ll start by creating a VPC with the AWSX VPC component. We’ll leverage the sensible defaults the component provides to create a VPC across 3 availability zones for high availability. You can either create a single NAT gateway in our inspection VPC to save costs, or create 3 NAT gateways, one in each availability zone, to provide high availability. See the code for details.

After creating the VPC, you’ll then need to attach the inspection VPC to the Transit Gateway. This is how the inspection VPC will be able to utilize the Transit Gateway and accept traffic from the spoke VPCs:

self.tgw_attachment = aws.ec2transitgateway.VpcAttachment(
 f"{name}-tgw-vpc-attachment",
 aws.ec2transitgateway.VpcAttachmentArgs(
 transit_gateway_id=self.args.tgw_id,
 subnet_ids=self.vpc.isolated_subnet_ids,
 vpc_id=self.vpc.vpc_id,
 transit_gateway_default_route_table_association=False,
 transit_gateway_default_route_table_propagation=False,
 appliance_mode_support="enable",
 tags={
 "Name": f"{name}",
 },
 ),
 # We can only have one attachment per VPC, so we need to tell Pulumi
 # explicitly to delete the old one before creating a new one:
 pulumi.ResourceOptions(
 delete_before_replace=True,
 depends_on=[self.vpc],
 parent=self,
 )
)

Then add a route to the Transit Gateway route table for the spoke VPCs that routes all traffic outside of the VPC to the inspection VPC (or to be precise, the Transit Gateway attachment in the inspection VPC). This route only needs to be added once as it will apply to all spoke VPCs, so we add it here in the inspection VPC since that’s where we create the Transit Gateway attachment:

aws.ec2transitgateway.Route(
 f"{name}-default-spoke-to-inspection",
 aws.ec2transitgateway.RouteArgs(
 destination_cidr_block="0.0.0.0/0",
 transit_gateway_attachment_id=self.tgw_attachment.id,
 transit_gateway_route_table_id=args.spoke_tgw_route_table_id,
 ),
 opts=pulumi.ResourceOptions(
 parent=self,
 )
)

Next, add routes within the inspection VPC’s isolated subnets that route traffic to the internet through the NAT gateway. The Transit Gateway:

for subnet_id in isolated_subnet_ids:
 route_table = aws.ec2.get_route_table(
 subnet_id=subnet_id
 )

 aws.ec2.Route(
 f"{self.name}-route-{subnet_id}-to-tgw",
 aws.ec2.RouteArgs(
 route_table_id=route_table.id,
 destination_cidr_block="0.0.0.0/0",
 nat_gateway_id=self.nat_gateway.id,
 ),
 pulumi.ResourceOptions(
 parent=self,
 ),
 )

Finally, you’ll need to add routes within the inspection VPC’s public subnets (where the NAT gateways live) that route return traffic from the internet back through the Transit Gateway. This is how the inspection VPC will be able to route traffic to the spoke VPCs:

for subnet_id in public_subnet_ids:
 route_table = aws.ec2.get_route_table(
 subnet_id=subnet_id
 )

 aws.ec2.Route(
 f"{self.name}-route-{subnet_id}-to-tgw",
 aws.ec2.RouteArgs(
 route_table_id=route_table.id,
 destination_cidr_block=self.args.supernet_cidr_block,
 transit_gateway_id=self.args.tgw_id,
 ),
 pulumi.ResourceOptions(
 depends_on=[self.tgw_attachment],
 parent=self,
 ),
 )

You need to create both sets of these routes within an apply because of the need to access the raw values of each of the subnet ids. Apply can be conceptually difficult for beginners in Pulumi to grasp, but it essentially means “once you know all of the values that needed, pass them on to this function”.

Final steps

Now that you’ve created our inspection VPC component, you can reference it in the main project file:

from inspection import InspectionVpc, InspectionVpcArgs

and you can create an instance of the component:

insp_vpc = InspectionVpc(
 "inspection",
 InspectionVpcArgs(
 supernet_cidr_block=supernet_cidr,
 vpc_cidr_block="10.129.0.0/24",
 tgw_id=tgw.id,
 inspection_tgw_route_table_id=inspection_tgw_route_table.id,
 spoke_tgw_route_table_id=spoke_tgw_route_table.id,
 firewall_policy_arn=None, # We will add this in the next post
 )
)

At this point, we do not have have a fully functional architecture - we still need to create the spoke VPCs - but if we want to get some feedback on our architecture, we can run pulumi up and see what happens. We should see that Pulumi will create the VPC, the Transit Gateway attachment, and the routes.

In the next post, we’ll create our spoke VPCs and verify connectivity, then add centralized inspection of network traffic going to the internet. Stay tuned!