Configuring Persistent Storage in EKS With Amazon EFS for Data Analysis

Introduction

In this solution, we will configure persistent storage in Amazon Elastic Kubernetes Service (EKS) using Amazon Elastic File System (EFS) for data analysis. Amazon EKS is a managed Kubernetes service that makes it easy to run Kubernetes on AWS without needing to install and operate your own Kubernetes control plane or nodes. Amazon EFS provides a simple, scalable, and fully managed elastic NFS file system for use with AWS Cloud services and on-premises resources. By integrating EFS with EKS, we can ensure that our data is persistently stored and accessible across multiple pods and nodes, which is crucial for data analysis workloads.

Step-by-Step Explanation

Step 1: Set Up the EKS Cluster

First, we need to set up an EKS cluster. This involves creating the necessary IAM roles, VPC, subnets, and security groups. We will use Pulumi to define and deploy these resources.

Step 2: Create the EFS File System

Next, we will create an EFS file system. This file system will be used to store our data persistently. We will also create mount targets for the file system in each subnet of our VPC.

Step 3: Configure EFS Access Points

We will configure EFS access points to manage access to the file system. Access points simplify the process of managing access permissions for different applications and users.

Step 4: Deploy the EFS CSI Driver

To enable our EKS cluster to use EFS, we need to deploy the EFS Container Storage Interface (CSI) driver. This driver allows Kubernetes to interact with EFS and manage persistent volumes.

Step 5: Create Persistent Volume and Persistent Volume Claim

We will create a Persistent Volume (PV) that references our EFS file system and a Persistent Volume Claim (PVC) that pods can use to request storage. The PVC will be bound to the PV, ensuring that our data is stored in the EFS file system.

Step 6: Deploy a Sample Application

Finally, we will deploy a sample application that uses the PVC to store data. This application will demonstrate how data can be persistently stored and accessed across multiple pods and nodes in the EKS cluster.

Key Points

• Amazon EKS: Managed Kubernetes service for running Kubernetes on AWS.
• Amazon EFS: Fully managed elastic NFS file system for scalable and persistent storage.
• EFS CSI Driver: Allows Kubernetes to interact with EFS and manage persistent volumes.
• Persistent Volume (PV): Represents a piece of storage in the cluster that has been provisioned by an administrator.
• Persistent Volume Claim (PVC): A request for storage by a user that is bound to a PV.
• IAM Roles and Policies: Necessary permissions for EKS and EFS to interact with other AWS services.
• VPC and Subnets: Networking components required for the EKS cluster and EFS file system.

Conclusion

By following this solution, we have successfully configured persistent storage in Amazon EKS using Amazon EFS. This setup ensures that our data is persistently stored and accessible across multiple pods and nodes, which is essential for data analysis workloads. Using Pulumi to define and deploy these resources allows us to manage our infrastructure as code, making it easier to maintain and scale our solution as needed.

Full Code Example

import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as k8s from "@pulumi/kubernetes";

// Create a VPC for the EKS cluster
const vpc = new aws.ec2.Vpc("eks-vpc", {
    cidrBlock: "10.0.0.0/16",
    enableDnsHostnames: true,
    enableDnsSupport: true,
    tags: { Name: "eks-vpc" },
});

// Create subnets for the VPC
const subnet = new aws.ec2.Subnet("eks-subnet", {
    vpcId: vpc.id,
    cidrBlock: "10.0.1.0/24",
    availabilityZone: "us-west-2a",
    tags: { Name: "eks-subnet" },
});

// Create an IAM role for the EKS cluster
const eksRole = new aws.iam.Role("eks-role", {
    assumeRolePolicy: JSON.stringify({
        Version: "2012-10-17",
        Statement: [{
            Action: "sts:AssumeRole",
            Principal: { Service: "eks.amazonaws.com" },
            Effect: "Allow",
        }],
    }),
});

// Attach the AmazonEKSClusterPolicy to the role
new aws.iam.RolePolicyAttachment("eks-role-policy", {
    role: eksRole.name,
    policyArn: "arn:aws:iam::aws:policy/AmazonEKSClusterPolicy",
});

// Create the EKS cluster
const eksCluster = new aws.eks.Cluster("eks-cluster", {
    roleArn: eksRole.arn,
    vpcConfig: { subnetIds: [subnet.id] },
});

// Create an EFS file system
const efsFileSystem = new aws.efs.FileSystem("efs-file-system", {
    tags: { Name: "efs-file-system" },
});

// Create an EFS access point
const efsAccessPoint = new aws.efs.AccessPoint("efs-access-point", {
    fileSystemId: efsFileSystem.id,
    posixUser: { uid: 1000, gid: 1000 },
    rootDirectory: { path: "/data" },
});

// Deploy the EFS CSI driver
const efsCsiDriver = new k8s.helm.v3.Chart("efs-csi-driver", {
    chart: "aws-efs-csi-driver",
    fetchOpts: { repo: "https://kubernetes-sigs.github.io/aws-efs-csi-driver" },
});

// Create a Persistent Volume (PV)
const pv = new k8s.core.v1.PersistentVolume("efs-pv", {
    spec: {
        capacity: { storage: "5Gi" },
        accessModes: ["ReadWriteMany"],
        persistentVolumeReclaimPolicy: "Retain",
        csi: {
            driver: "efs.csi.aws.com",
            volumeHandle: efsFileSystem.id,
        },
    },
});

// Create a Persistent Volume Claim (PVC)
const pvc = new k8s.core.v1.PersistentVolumeClaim("efs-pvc", {
    spec: {
        accessModes: ["ReadWriteMany"],
        resources: { requests: { storage: "5Gi" } },
    },
});

export const eksClusterName = eksCluster.name;
export const efsFileSystemId = efsFileSystem.id;
export const efsAccessPointId = efsAccessPoint.id;
export const persistentVolumeName = pv.metadata.name;
export const persistentVolumeClaimName = pvc.metadata.name;
export const iamRoleArn = eksRole.arn;
export const vpcId = vpc.id;