Simplify Kubernetes RBAC in Amazon EKS with open source Pulumi packages

Posted on Wednesday, Apr 24, 2019

One of the most common areas Kubernetes operators struggle with in production involves creating and managing role-based access control (RBAC). This is so daunting that RBAC is often not implemented, or implemented halfway, or the configuration becomes impossible to maintain.

Fortunately, Pulumi makes RBAC on Kuberenetes so easy that you’ll never create an insecure cluster again. In this post, we will contrast the traditional way of working with RBAC on EKS with using Pulumi.

Here are a few highlights:

NO MORE YAMLs! Configuring YAMLs, operators or custom resources is now a thing in the past! You use TypeScript or JavaScript to program directly with our cloud SDK and connect all cloud services to your Kubernetes services with a simple reference to the object in your program.
INCREASED DEVELOPMENT VELOCITY: You intuitively program Kubernetes objects with our SDK abstractions using minimal amount of code within hours instead of months. You “autocomplete” AWS, EKS, Kubernetes specifications within your IDE without understanding the entire API.
EASY UPDATES: Changing a roleRef in a RoleBinding, on one or multiple clusters involves updating your TypeScript file index.ts and running pulumi up. The Pulumi Service allows you to share your stack with your team in your GitHub, GitLab, or Atlassian-based organization.
WORKFLOW AUTOMATION FOR RBAC AT SCALE: You can delete or update multiple RoleBindings or Roles from your Pulumi stack source code. As you commit these changes to your repository, you can plan automated triggers that validate such changes as part of your CI/CD flow, whether you use Travis, CircleCI, AzureDevOps and more. Pulumi even has a GitHub Application for surfacing results within pull requests.

Prerequisites to work with Pulumi

Install pulumi CLI and set up your AWS credentials. Initialize a new Pulumi project from available templates. We use aws-typescript template here to install all dependencies and save the configuration.

$ brew install pulumi/tap/pulumi # download pulumi CLI

$ mkdir eks-rbac && cd eks-rbac

$ pulumi new aws-typescript

$ ls -l
-rw-r--r--   1 nishidavidson  staff     32 Apr 18 14:49 Pulumi.dev.yaml
-rw-------   1 nishidavidson  staff     84 Apr 18 14:48 Pulumi.yaml
-rw-------   1 nishidavidson  staff    273 Apr 18 14:48 index.ts
drwxr-xr-x  92 nishidavidson  staff   2944 Apr 18 14:49 node_modules
-rw-r--r--   1 nishidavidson  staff  48352 Apr 18 14:49 package-lock.json
-rw-------   1 nishidavidson  staff    228 Apr 18 14:48 package.json
-rw-------   1 nishidavidson  staff    522 Apr 18 14:48 tsconfig.json

With Pulumi, you will modify and update the default index.ts file with AWS and EKS resource variable declarations. We show you how to add this code as we contrast Pulumi’s approach with the sequential traditional approach in the steps below. In the end, you will do a one-time run of pulumi up and watch all the steps in the Traditional Way come alive simultaneously.

Step 1: Create three IAM Roles with a Trust-policy to map to Amazon EKS RBAC.

The Traditional-approach:

You sequentially create three IAM roles (clusterAdminRole; AutomationRole; EnvProdRole) with aws command line tool as shown below:

$ aws iam create-role --role-name clusterAdminRole --assume-role-policy-document file://Role-Trust-Policy.json
 
{
    "Role": {
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Action": "sts:AssumeRole",
                    "Principal": {
                        "AWS": "arn:aws:iam::xxxxxxxxxxxx:root"
                    },
                    "Effect": "Allow",
                    "Sid": ""
                }
            ]
        },
        "RoleId": "AROASHIVKXX3SFFMUUEU6",
        "CreateDate": "2019-04-17T17:43:03Z",
        "RoleName": "clusterAdminRole",
        "Path": "/",
        "Arn": "arn:aws:iam::xxxxxxxxxxxx:role/clusterAdminRole"
    }
}
 
$ cat Role-Trust-Policy.json
{
    "Version": "2012-10-17",
    "Statement": [
        {
          "Effect": "Allow",
          "Action": "sts:AssumeRole",
          "Resource": "*"
        }
    ]
}

The Pulumi-approach:

You update the default index.ts file in your source code editor such as VSCode as follows:

import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";
import * as eks from "@pulumi/eks";
import * as k8s from "@pulumi/kubernetes";

/*
 * 1) Single step deployment of three IAM Roles
 */

function createIAMRole(name: string): aws.iam.Role {
    // Create an IAM Role...
    return new aws.iam.Role(`${name}`, {
        assumeRolePolicy: `{
            "Version": "2012-10-17",
            "Statement":[
              {
                "Sid": "",
                "Effect": "Allow",
                "Principal": {
                  "AWS": "arn:aws:iam::153052954103:root"
                },
                "Action": "sts:AssumeRole"
              }
            ]
           }
        `,
          tags: {
              "clusterAccess": `${name}-usr`,
          },
        });
    };
}

// Administrator AWS IAM clusterAdminRole with full access to all AWS resources
const clusterAdminRole = createIAMRole("clusterAdminRole");

// Administer Automation role for use in pipelines, e.g. gitlab CI, Teamcity, etc.
const AutomationRole = createIAMRole("AutomationRole");

// Administer Prod role for use in Prod environment
const EnvProdRole = createIAMRole("EnvProdRole");

Step 2: Create one EKS cluster. Validate cluster creation. Add the namespaces you need.

The Traditional-approach:

You go through the steps below to validate the cluster and k8s resource deployment based on your tool chain and your understanding of Kubernetes:

$ eksctl create cluster eks-nd-test
 
$ kubectl get no
NAME                                           STATUS   ROLES    AGE   VERSION
ip-192-168-41-125.us-east-2.compute.internal   Ready    <none>   11h   v1.11.5
ip-192-168-5-250.us-east-2.compute.internal    Ready    <none>   11h   v1.11.5
 
$ kubectl create -f automation-ns.yaml && kubectl create -f prod-ns.yaml
 
$ cat automation-ns.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: automation
  labels:
    name: automation

The Pulumi-approach:

You use our API docs and your source-code editor to autocomplete the default index.ts file.

/*
    * 2) Single step deployment of EKS cluster with the most important variables and a Simple Function to create namespaces
    * automation and prod
    */

const vpc = new awsx.ec2.Vpc("vpc", {});
const cluster = new eks.Cluster("eks-cluster", {
  vpcId             : vpc.id,
  subnetIds         : vpc.publicSubnetIds,
  instanceType      : "t2.medium",
  nodeRootVolumeSize: 200,
  desiredCapacity   : 1,
  maxSize           : 2,
  minSize           : 1,
  deployDashboard   : false,
  vpcCniOptions     : {
    warmIpTarget    : 4,
  },
  roleMappings      : [
    // Provides full administrator cluster access to the k8s cluster
    {
      groups    : ["system:masters"],
      roleArn   : clusterAdminRole.arn,
      username  : "pulumi:admin-usr",
    },
    // Map IAM role arn "AutomationRoleArn" to the k8s user with name "automation-usr", e.g. gitlab CI
    {
      groups    : ["pulumi:automation-grp"],
      roleArn   : AutomationRole.arn,
      username  : "pulumi:automation-usr",
    },
    // Map IAM role arn "EnvProdRoleArn" to the k8s user with name "prod-usr"
    {
      groups    : ["pulumi:prod-grp"],
      roleArn   : EnvProdRole.arn,
      username  : "pulumi:prod-usr",
    },
  ],
});

export const clusterName = cluster.eksCluster.name;

function createNewNamespace(name: string): k8s.core.v1.Namespace {
  // Create new namespace
  return new k8s.core.v1.Namespace(name, { metadata: { name: name } }, { provider: cluster.provider });
}

// Declare namespaces automation and prod.
const automation = createNewNamespace("automation");
const prod = createNewNamespace("prod");

Step 3: Understand Kubernetes RBAC. Declare the Kubernetes objects on the EKS cluster.

The Kubernetes RBAC API declares four top-level types that can be defined as YAMLs syntaxes: a) Role - represents a set of additive rules within a namespace; b) RoleBinding - grants namespace-wide access to k8s subjects and resources; c) ClusterRole - represents a set of additive rules within the cluster; d) ClusterRoleBinding - grants cluster-wide access to k8s subjects and resources.

The Traditional-approach:

You define three k8s users with different privileges in your cluster and test them sequentially: User type1 called pulumi:admin-usr for users have cluster admin rights

 $ cat user1.yaml
 kind: ClusterRole
 apiVersion: rbac.authorization.k8s.io/v1
 metadata:
  name: ClusterAdminRole
  # no namespace needed
 rules:
 - apiGroups: ["*"]
  resources: ["*"]
  verbs: ["*"]
  
 ---
 kind: ClusterRoleBinding
 apiVersion: rbac.authorization.k8s.io/v1
 metadata:
   name: cluster-admin-binding
 subjects:
 - kind: User
  name: "pulumi:admin-usr"
  apiGroup: rbac.authorization.k8s.io
 roleRef:
  kind: ClusterRole
  name: ClusterAdminRole
  apiGroup: rbac.authorization.k8s.io`

User type2 called pulumi:automation-usr for users that have permissions to all k8s resources in namespace automation. An e.g would be your CI/CD pipeline

$ cat user2.yaml
 
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: AutomationRole
  namespace: automation
rules:
- apiGroups: ["*"]
  resources: ["*"]
  verbs: ["*"]
 
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: automation-binding
  namespace: automation
subjects:
- kind: User
  name: "pulumi:automation-usr"
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: AutomationRole
  apiGroup: rbac.authorization.k8s.io

User type 3 called prod-usr for users that have read access to all k8s resources in the namespace prod

$ cat user3.yaml
 
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: EnvProdRole
  namespace: prod
rules:
- apiGroups: ["*"]
  resources: ["*"]
  verbs: ["get", "list", "watch"]
 
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: env-prod-binding
  namespace: prod
subjects:
- kind: User
  name: "pulumi:prod-usr"
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: EnvProdRole
  apiGroup: rbac.authorization.k8s.io`

$ kubectl apply -f user1.yaml && kubectl apply -f user2.yaml && kubectl apply -f user3.yaml`
 
clusterrole.rbac.authorization.k8s.io/ClusterAdminRole created
clusterrolebinding.rbac.authorization.k8s.io/cluster-admin-binding created
role.rbac.authorization.k8s.io/AutomationRole created
rolebinding.rbac.authorization.k8s.io/automation created
role.rbac.authorization.k8s.io/EnvProdRole created
rolebinding.rbac.authorization.k8s.io/env-prod-binding created

The Pulumi-approach:

Update your index.ts file with more code as follows:

/*
 * 3) Single Step deployment of k8s RBAC configuration for user1, user2 and user3 per our example
 */

// Grant cluster admin access to all admins with k8s ClusterRole and ClusterRoleBinding
new k8s.rbac.v1.ClusterRole("clusterAdminRole", {
  metadata: {
    name: "clusterAdminRole",
  },
  rules: [{
    apiGroups: ["*"],
    resources: ["*"],
    verbs: ["*"],
  }]
}, {provider: cluster.provider});

new k8s.rbac.v1.ClusterRoleBinding("cluster-admin-binding", {
  metadata: {
    name: "cluster-admin-binding",
  },
  subjects: [{
     kind: "User",
     name: "pulumi:admin-usr",
  }],
  roleRef: {
    kind: "ClusterRole",
    name: "clusterAdminRole",
    apiGroup: "rbac.authorization.k8s.io",
  },
}, {provider: cluster.provider});

// User2 called automation-usr for users that have permissions to all k8s resources in the namespace automation
new k8s.rbac.v1.Role("AutomationRole", {
  metadata: {
    name: "AutomationRole",
    namespace: "automation",
  },
  rules: [{
    apiGroups: ["*"],
    resources: ["*"],
    verbs: ["*"],
  }]
}, {provider: cluster.provider});
  
new k8s.rbac.v1.RoleBinding("automation-binding", {
  metadata: {
    name: "automation-binding",
    namespace: "automation",
  },
  subjects: [{
     kind: "User",
     name: "pulumi:automation-usr",
     apiGroup: "rbac.authorization.k8s.io",
  }],
  roleRef: {
    kind: "Role",
    name: "AutomationRole",
    apiGroup: "rbac.authorization.k8s.io",
  },
}, {provider: cluster.provider});

// User3 called prod-usr for users that have read access to all k8s resources in the namespace env-prod
new k8s.rbac.v1.Role("EnvProdRole", {
  metadata: {
    name: "EnvProdRole",
    namespace: "prod",
  },
  rules: [{
    apiGroups: ["*"],
    resources: ["*"],
    verbs: ["get", "watch", "list"],
  }],
}, {provider: cluster.provider});
  
new k8s.rbac.v1.RoleBinding("env-prod-binding", {
  metadata: {
    name: "env-prod-binding",
    namespace: "prod",
  },
  subjects: [{
     kind: "User",
     name: "pulumi:prod-usr",
     apiGroup: "rbac.authorization.k8s.io",
    }],
  roleRef: {
    kind: "Role",
    name: "EnvProdRole",
    apiGroup: "rbac.authorization.k8s.io",
  },
}, {provider: cluster.provider});

export const kubeconfig = cluster.kubeconfig

Step 4: Update aws-iam-authenticator ConfigMap to map the IAM Roles to the Kubernetes usernames.

The Traditional-approach:

You get the three IAM role arns for clusterAdminRole, AutomationRole, EnvProdRole and update the configmap with k8s usernames pulumi:admin-usr, pulumi:automation-usr and pulumi:prod-usr.

mapRoles:
----
- groups:
  - system:masters
  rolearn: arn:aws:iam::XXXXXXXXXXXX:role/clusterAdminRole
  username: pulumi:admin-usr
- groups:
  rolearn: arn:aws:iam::XXXXXXXXXXXX:role/AutomationRole
  username: pulumi:automation-usr
- groups:
  rolearn: arn:aws:iam::XXXXXXXXXXXX:role/EnvProdRole
  username: pulumi:prod-usr
- groups:
  - system:bootstrappers
  - system:nodes
  rolearn: arn:aws:iam::XXXXXXXXXXXX:role/eksctl-eks-rbac-nd-test-nodegroup-NodeInstanceRole-NP542EG8JX8U
  username: system:node:`

The Pulumi-approach:

This step is not required as you have already updated the EKS ConfigMap at cluster creation time in STEP 2 using “RoleMappings”. Simply run pulumi up with the full index.ts file. Watch all your components come alive simultaneously. Setting up RBAC on one EKS cluster is a long convoluted sequential process that requires multiple validations along the way. Imagine the complexity involved when working with multiple tools for an environment that requires multiple groups with many users, namespaces, and clusters.

Testing the Pulumi approach worked

Make sure you run pulumi up with this index.ts file.

$ pulumi stack output kubeconfig | jq > kubeconfig.yaml
$ export KUBECONFIG = kubeconfig.yaml

Assume the IAM role AutomationRole with access to all Kubernetes resources in namespace automation and test if the permissions work.

  "users": [
    {
      "name": "aws",
      "user": {
        "exec": {
          "apiVersion": "client.authentication.k8s.io/v1alpha1",
          "args": [
            "token",
            "-i",
            "eks-cluster-eksCluster-196b0de",
            "-r",
            "arn:aws:iam::xxxxxxxxxxxx:role/AutomationRole"
          ],
          "command": "aws-iam-authenticator"
        }
      }
    }
  ]
}
 
$ kubectl get po --namespace=automation
No resources found.
 
$ kubectl get po --namespace=prod
Error from server (Forbidden): pods is forbidden: User "pulumi: automation-usr" cannot list resource "pods" in API group "" in the namespace "prod"

Upon assuming the IAM role AutomationRole which is mapped to Kubernernetes username pulumi:automation-usr in the EKS cluster configmap, you are only restricted to the resources and verbs allowed in the namespace “automation” and not in namespace “prod”.

Next Step

In this post, we discussed how setting up Kubernetes RBAC with Pulumi is simple, comprehensive, non-sequential and part of your everyday programming experience. You can find the complete pulumi code for our example and try it out yourself.

Pulumi is open source and free to use. For more examples, visit our GitHub examples page here. To learn more about Pulumi and how to manage Kubernetes through code, have a look at our “Get Started with Kubernetes” guide.

Simplify Kubernetes RBAC in Amazon EKS with open source Pulumi packages

Prerequisites to work with Pulumi

Step 1: Create three IAM Roles with a Trust-policy to map to Amazon EKS RBAC.

The Traditional-approach:

The Pulumi-approach:

Step 2: Create one EKS cluster. Validate cluster creation. Add the namespaces you need.

The Traditional-approach:

The Pulumi-approach:

Step 3: Understand Kubernetes RBAC. Declare the Kubernetes objects on the EKS cluster.

The Traditional-approach:

The Pulumi-approach:

Step 4: Update aws-iam-authenticator ConfigMap to map the IAM Roles to the Kubernetes usernames.

The Traditional-approach:

The Pulumi-approach:

Testing the Pulumi approach worked

Next Step

Subscribe to the Pulumi Monthly Newsletter