Kubernetes Ingress with AWS ALB Ingress Controller and Pulumi Crosswalk for AWS
Posted on
Kubernetes Ingress is an API object that allows you manage external (or) internal HTTP[s] access to Kubernetes services running in a cluster. Amazon Elastic Load Balancing Application Load Balancer (ALB) is a popular AWS service that load balances incoming traffic at the application layer across multiple targets, such as Amazon EC2 instances, in a region. ALB supports multiple features including host or path based routing, TLS (Transport layer security) termination, WebSockets, HTTP/2, AWS WAF (web application firewall) integration, integrated access logs, and health checks.
The AWS ALB Ingress controller is a Kubernetes SIG-AWS subproject - it was the second sub-project added to SIG-AWS after the aws-authenticator subproject. The ALB Ingress controller triggers the creation of an ALB and the necessary supporting AWS resources whenever a Kubernetes user declares an Ingress resource on the cluster. TargetGroups are created for each backend specified in the Ingress resource. Listeners are created for every port specified as Ingress resource annotation. When no port is specified, sensible defaults (80 or 443) are used. Rules are created for each path specified in your ingress resource. This ensures that traffic to a specific path is routed to the correct TargetGroup.
In this post, we will work through a simple example of running ALB based Kubernetes Ingresses with Pulumi EKS, AWS, and AWSX packages.
Step 1: Initialize Pulumi project and stack
Install pulumi CLI
and set up your AWS credentials.
Initialize a new Pulumi project
and Pulumi stack from
available programming language
templates. We will use the
aws-typescript
template here and install all library
dependencies.
$ brew install pulumi/tap/pulumi # download pulumi CLI
$ mkdir eks-alb-ingress && cd eks-alb-ingress
$ pulumi new aws-typescript
$ npm install --save @pulumi/kubernetes @pulumi/eks
$ ls -la
drwxr-xr-x 10 nishidavidson staff 320 Jun 18 18:22 .
drwxr-xr-x+ 102 nishidavidson staff 3264 Jun 18 18:13 ..
-rw------- 1 nishidavidson staff 21 Jun 18 18:22 .gitignore
-rw-r--r-- 1 nishidavidson staff 32 Jun 18 18:22 Pulumi.dev.yaml
-rw------- 1 nishidavidson staff 91 Jun 18 18:22 Pulumi.yaml
-rw------- 1 nishidavidson staff 273 Jun 18 18:22 index.ts
drwxr-xr-x 95 nishidavidson staff 3040 Jun 18 18:22 node_modules
-rw-r--r-- 1 nishidavidson staff 50650 Jun 18 18:22 package-lock.json
-rw------- 1 nishidavidson staff 228 Jun 18 18:22 package.json
-rw------- 1 nishidavidson staff 522 Jun 18 18:22 tsconfig.json
Step 2: Create an EKS cluster
Once the steps above are complete, we update the typescript code in
index.ts
file to create an EKS cluster and run pulumi
up from the command line:
import * as awsx from "@pulumi/awsx";
import * as eks from "@pulumi/eks";
import * as k8s from "@pulumi/kubernetes";
const vpc = new awsx.ec2.Vpc("vpc-alb-ingress-eks", {});
const cluster = new eks.Cluster("eks-cluster", {
vpcId: vpc.id,
subnetIds: vpc.publicSubnetIds,
instanceType: "t2.medium",
version: "1.12",
nodeRootVolumeSize: 200,
desiredCapacity: 3,
maxSize: 4,
minSize: 3,
deployDashboard: false,
vpcCniOptions: {
warmIpTarget: 4
}
});
export const clusterName = cluster.eksCluster.name;
export const kubeconfig = cluster.kubeconfig;
export const clusterNodeInstanceRoleName = cluster.instanceRoles.apply(
roles => roles[0].name
);
export const nodesubnetId = cluster.core.subnetIds;
Configure the Public subnets in the console as defined in this guide.
Step 3: Deploy AWS ALB Ingress Controller
Lets confirm that the EKS cluster is up using the following commands:
$ pulumi stack export kubeconfig > kubeconfig.yaml
$ export KUBECONFIG=kubeconfig.yaml
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-10-10-0-58.ec2.internal Ready <none> 7h8m v1.12.7
ip-10-10-1-167.ec2.internal Ready <none> 7h8m v1.12.7
ip-10-10-1-84.ec2.internal Ready <none> 7h8m v1.12.7
Adequate roles and policies must be configured in AWS and available to the node(s) running the controller. How access is granted is up to you. Some will attach the needed rights to node’s role in AWS. Others will use projects like kube2iam. We attach a minimal IAM policy to the EKS worker nodes and then declare this on the EKS cluster as shown in the code below.
When declaring the ALB Ingress controller we simply re-use the Helm chart as part of the code. There is no need to rewrite all the logic or install Tiller in the EKS cluster. This frees you from thinking about RBAC for Helm, Tiller and the k8s cluster per se'.
With the default “instance mode” Ingress traffic starts from the ALB and
reaches the
NodePort
opened for the service. Traffic is then routed to the container Pods
within cluster. This is all encoded using Pulumi libraries below. If you
wish to use “ip-mode” with your Ingress such that traffic directly
reaches your pods, you will need to modify the
alb.ingress.kubernetes.io/target-type
annotation when using the helm
chart.
Append index.ts
file from Step 2 with the code below and run
pulumi up
:
// STEP 3: Declare the AWS ALB Ingress Controller
// Create IAM Policy for the IngressController called "ingressController-iam-policy” and read the policy ARN.
const ingressControllerPolicy = new aws.iam.Policy(
"ingressController-iam-policy",
{
policy: {
Version: "2012-10-17",
Statement: [
{
Effect: "Allow",
Action: [
"acm:DescribeCertificate",
"acm:ListCertificates",
"acm:GetCertificate"
],
Resource: "*"
},
{
Effect: "Allow",
Action: [
"ec2:AuthorizeSecurityGroupIngress",
"ec2:CreateSecurityGroup",
"ec2:CreateTags",
"ec2:DeleteTags",
"ec2:DeleteSecurityGroup",
"ec2:DescribeInstances",
"ec2:DescribeInstanceStatus",
"ec2:DescribeSecurityGroups",
"ec2:DescribeSubnets",
"ec2:DescribeTags",
"ec2:DescribeVpcs",
"ec2:ModifyInstanceAttribute",
"ec2:ModifyNetworkInterfaceAttribute",
"ec2:RevokeSecurityGroupIngress"
],
Resource: "*"
},
{
Effect: "Allow",
Action: [
"elasticloadbalancing:AddTags",
"elasticloadbalancing:CreateListener",
"elasticloadbalancing:CreateLoadBalancer",
"elasticloadbalancing:CreateRule",
"elasticloadbalancing:CreateTargetGroup",
"elasticloadbalancing:DeleteListener",
"elasticloadbalancing:DeleteLoadBalancer",
"elasticloadbalancing:DeleteRule",
"elasticloadbalancing:DeleteTargetGroup",
"elasticloadbalancing:DeregisterTargets",
"elasticloadbalancing:DescribeListeners",
"elasticloadbalancing:DescribeLoadBalancers",
"elasticloadbalancing:DescribeLoadBalancerAttributes",
"elasticloadbalancing:DescribeRules",
"elasticloadbalancing:DescribeSSLPolicies",
"elasticloadbalancing:DescribeTags",
"elasticloadbalancing:DescribeTargetGroups",
"elasticloadbalancing:DescribeTargetGroupAttributes",
"elasticloadbalancing:DescribeTargetHealth",
"elasticloadbalancing:ModifyListener",
"elasticloadbalancing:ModifyLoadBalancerAttributes",
"elasticloadbalancing:ModifyRule",
"elasticloadbalancing:ModifyTargetGroup",
"elasticloadbalancing:ModifyTargetGroupAttributes",
"elasticloadbalancing:RegisterTargets",
"elasticloadbalancing:RemoveTags",
"elasticloadbalancing:SetIpAddressType",
"elasticloadbalancing:SetSecurityGroups",
"elasticloadbalancing:SetSubnets",
"elasticloadbalancing:SetWebACL"
],
Resource: "*"
},
{
Effect: "Allow",
Action: ["iam:GetServerCertificate", "iam:ListServerCertificates"],
Resource: "*"
},
{
Effect: "Allow",
Action: [
"waf-regional:GetWebACLForResource",
"waf-regional:GetWebACL",
"waf-regional:AssociateWebACL",
"waf-regional:DisassociateWebACL"
],
Resource: "*"
},
{
Effect: "Allow",
Action: ["tag:GetResources", "tag:TagResources"],
Resource: "*"
},
{
Effect: "Allow",
Action: ["waf:GetWebACL"],
Resource: "*"
}
]
}
}
);
// Attach this policy to the NodeInstanceRole of the worker nodes.
export const nodeinstanceRole = new aws.iam.RolePolicyAttachment(
"eks-NodeInstanceRole-policy-attach",
{
policyArn: ingressControllerPolicy.arn,
role: clusterNodeInstanceRoleName
}
);
// Declare the ALBIngressController in 1 step with the Helm Chart.
const albingresscntlr = new k8s.helm.v2.Chart(
"alb",
{
chart:
"http://storage.googleapis.com/kubernetes-charts-incubator/aws-alb-ingress-controller-0.1.9.tgz",
values: {
clusterName: clusterName,
autoDiscoverAwsRegion: "true",
autoDiscoverAwsVpcID: "true"
}
},
{ provider: cluster.provider }
);
Confirm the alb-ingress-controller was created as follows:
$ kubectl get pods -n default | grep alb
alb-aws-alb-ingress-controller-58f44d4bb8lxs6w
$ kubectl logs alb-ingress-controller-58f44d4bb8lxs6w
-------------------------------------------------------------------------------
AWS ALB Ingress controller
Release: v1.1.2
Build: git-cc1c5971
Repository: https://github.com/kubernetes-sigs/aws-alb-ingress-controller.git
-------------------------------------------------------------------------------
Make sure the ingress-controller logs do not show errors about missing subnet tags or missing cluster name before proceeding to Step 4.
Step 4: Deploy Sample Application
The Ingress controller should now be running on the EKS worker nodes.
Let’s now create a sample “2048-game” and expose it as an Ingress on
our EKS cluster. The code below will let you do so. Append this piece of
code into index.ts
file from Step 3 and run pulumi up
:
function createNewNamespace(name: string): k8s.core.v1.Namespace {
//Create new namespace
return new k8s.core.v1.Namespace(
name,
{ metadata: { name: name } },
{ provider: cluster.provider }
);
}
// Define the 2048 namespace, deployment, and service.
const nsgame = createNewNamespace("2048-game");
const deploymentgame = new k8s.extensions.v1beta1.Deployment(
"deployment-game",
{
metadata: { name: "deployment-game", namespace: "2048-game" },
spec: {
replicas: 5,
template: {
metadata: { labels: { app: "2048" } },
spec: {
containers: [
{
image: "alexwhen/docker-2048",
imagePullPolicy: "Always",
name: "2048",
ports: [{ containerPort: 80 }]
}
]
}
}
}
},
{ provider: cluster.provider }
);
const servicegame = new k8s.core.v1.Service(
"service-game",
{
metadata: { name: "service-2048", namespace: "2048-game" },
spec: {
ports: [{ port: 80, targetPort: 80, protocol: "TCP" }],
type: "NodePort",
selector: { app: "2048" }
}
},
{ provider: cluster.provider }
);
//declare 2048 ingress
const ingressgame = new k8s.extensions.v1beta1.Ingress(
"ingress-game",
{
metadata: {
name: "2048-ingress",
namespace: "2048-game",
annotations: {
"kubernetes.io/ingress.class": "alb",
"alb.ingress.kubernetes.io/scheme": "internet-facing"
},
labels: { app: "2048-ingress" }
},
spec: {
rules: [
{
http: {
paths: [
{
path: "/*",
backend: { serviceName: "service-2048", servicePort: 80 }
}
]
}
}
]
}
},
{ provider: cluster.provider }
);
After few seconds, verify the Ingress resource as follows:
$ kubectl get ingress/2048-ingress -n 2048-game
NAME HOSTS ADDRESS PORTS AGE
2048-ingress * DNS-Name-Of-Your-ALB 80 3m
Open a browser. Copy and paste your “DNS-Name-Of-Your-ALB”. You should be to access your newly deployed 2048 game – have fun!
Pulumi is open source and free to use. For more information on our product platform, check out the following resources: