Provisioning a Staging EKS Cluster for Big Data Workloads

Provisioning a Staging EKS Cluster for Big Data Workloads

In this guide, we will provision an Amazon EKS (Elastic Kubernetes Service) cluster tailored for big data workloads. This setup will include the creation of a VPC, subnets, security groups, and the EKS cluster itself. We will also configure the necessary IAM roles and policies to ensure proper access control.

Step-by-Step Explanation

1. Create a VPC: We start by creating a Virtual Private Cloud (VPC) to house our EKS cluster. This VPC will contain multiple subnets across different availability zones for high availability.
2. Create Subnets: Within the VPC, we will create public and private subnets. Public subnets will be used for internet-facing resources, while private subnets will be used for internal resources.
3. Create Security Groups: Security groups will be configured to control inbound and outbound traffic to the EKS cluster.
4. Create IAM Roles and Policies: IAM roles and policies will be created to grant the EKS cluster the necessary permissions to interact with other AWS services.
5. Provision the EKS Cluster: Finally, we will create the EKS cluster and configure it to use the VPC, subnets, and security groups created earlier.

Summary

By following this guide, you will have a staging EKS cluster set up and ready for big data workloads. This setup ensures high availability, security, and proper access control, making it suitable for handling large-scale data processing tasks.

Full Code Example

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

// Create a VPC
const vpc = new aws.ec2.Vpc("eks-vpc", {
    cidrBlock: "10.0.0.0/16",
    enableDnsHostnames: true,
    enableDnsSupport: true,
    tags: { Name: "eks-vpc" },
});

// Create Subnets
const subnet1 = new aws.ec2.Subnet("eks-subnet-1", {
    vpcId: vpc.id,
    cidrBlock: "10.0.1.0/24",
    availabilityZone: "us-west-2a",
    tags: { Name: "eks-subnet-1" },
});

const subnet2 = new aws.ec2.Subnet("eks-subnet-2", {
    vpcId: vpc.id,
    cidrBlock: "10.0.2.0/24",
    availabilityZone: "us-west-2b",
    tags: { Name: "eks-subnet-2" },
});

// Create Security Group
const securityGroup = new aws.ec2.SecurityGroup("eks-security-group", {
    vpcId: vpc.id,
    description: "EKS security group",
    ingress: [
        { protocol: "tcp", fromPort: 80, toPort: 80, cidrBlocks: ["0.0.0.0/0"] },
    ],
    egress: [
        { protocol: "-1", fromPort: 0, toPort: 0, cidrBlocks: ["0.0.0.0/0"] },
    ],
    tags: { Name: "eks-security-group" },
});

// Create IAM Role
const eksRole = new aws.iam.Role("eks-role", {
    assumeRolePolicy: JSON.stringify({
        Version: "2012-10-17",
        Statement: [
            {
                Action: "sts:AssumeRole",
                Principal: {
                    Service: "eks.amazonaws.com",
                },
                Effect: "Allow",
                Sid: "",
            },
        ],
    }),
    tags: { Name: "eks-role" },
});

// Attach Policy to IAM Role
const eksRolePolicy = new aws.iam.RolePolicy("eks-role-policy", {
    role: eksRole.id,
    policy: JSON.stringify({
        Version: "2012-10-17",
        Statement: [
            {
                Effect: "Allow",
                Action: [
                    "ec2:Describe*",
                    "ec2:List*",
                    "ec2:Get*",
                    "ec2:Create*",
                    "ec2:Delete*",
                    "ec2:Modify*",
                    "ec2:Authorize*",
                    "ec2:Revoke*",
                    "ec2:Allocate*",
                    "ec2:Release*",
                    "ec2:Associate*",
                    "ec2:Disassociate*",
                    "ec2:Attach*",
                    "ec2:Detach*",
                    "ec2:Start*",
                    "ec2:Stop*",
                    "ec2:Terminate*",
                    "ec2:Reboot*",
                    "ec2:Monitor*",
                    "ec2:Unmonitor*",
                    "ec2:Describe*",
                    "ec2:List*",
                    "ec2:Get*",
                    "ec2:Create*",
                    "ec2:Delete*",
                    "ec2:Modify*",
                    "ec2:Authorize*",
                    "ec2:Revoke*",
                    "ec2:Allocate*",
                    "ec2:Release*",
                    "ec2:Associate*",
                    "ec2:Disassociate*",
                    "ec2:Attach*",
                    "ec2:Detach*",
                    "ec2:Start*",
                    "ec2:Stop*",
                    "ec2:Terminate*",
                    "ec2:Reboot*",
                    "ec2:Monitor*",
                    "ec2:Unmonitor*",
                ],
                Resource: "*",
            },
        ],
    }),
});

// Create EKS Cluster
const eksCluster = new aws.eks.Cluster("eks-cluster", {
    roleArn: eksRole.arn,
    vpcConfig: {
        subnetIds: [subnet1.id, subnet2.id],
        securityGroupIds: [securityGroup.id],
    },
    tags: { Name: "eks-cluster" },
});

// Export outputs
export const vpcId = vpc.id;
export const subnetIds = [subnet1.id, subnet2.id];
export const securityGroupId = securityGroup.id;
export const eksClusterName = eksCluster.name;