Pulumi Blog: Ecs

How to Use ECS Anywhere with Pulumi [Step-by-Step Guide]

Piers Karsenbarg — Thu, 27 May 2021 00:00:00 +0000

This post is outdated and contains references to a pre-release version of Pulumi Crosswalk (@pulumi/awsx). For updated AWSx documentation and examples, see the AWS Guides.

When Amazon’s Elastic Container Service (ECS) first launched in 2014, it enabled an easy and convenient way of deploying and scheduling containers in the AWS ecosystem. Back then, you would run a set of EC2 instances, and ECS would deploy containers to instances based on the size, resources, and placement requirements you specified.

In 2018, AWS released Fargate, bringing a serverless feel to the platform. You could have ECS run your container without having to manage any infrastructure.

The next iteration is ECS Anywhere, which allows you to deploy your containers to almost any server. You can deploy to an EC2 instance inside AWS, a virtual machine (VM) in another cloud provider, or even a bare metal server in your private data center. Best of all, you can control the deployments in the same way you would with EC2 or Fargate.

Why would I want to use this?

As we’ve seen over the past few years, the ECS system makes it a convenient service to use. There are several reasons why you might want to use ECS as a control plane for your container-based applications but not run the actual applications within your AWS infrastructure. These include:

Latency requirements: You have systems on another network, and you need your containers to be running on that same network to give you the low latency required.
Regulatory requirements: There may be legal, security, or privacy needs that can only be satisfied by running your application in your own data center.
Operational requirements: you don’t want to manage a complex container cluster yourself on another network and are already running workloads on ECS.

Behind the scenes

The crucial part of understanding how ECS Anywhere works is that you’re using ECS to manage the deployment of the containers to your VMs (which we’ll call nodes going forward). The actual interaction between your application and whatever network traffic or instructions it receives is separate. You’ll see this shown in the example below where we deploy two VMs and a load balancer. Once the application is up and running, ECS takes a back seat and waits for the instructions to deploy a new version or stop the application from running altogether.

There are three parts to getting this to work:

1. Setting up the AWS infrastructure

AWS’ Systems Manager has a feature called “Hybrid Activations.” We’ll use it to link the external nodes to the ECS cluster using an activation code and an activation id. We’ll also create various IAM roles necessary to activate the nodes and assign them to the ECS cluster. In our example, we’ll also set up IAM roles so that our application can send logs to CloudWatch as well.

2. Deploying the nodes

During the node setup process, we’re going to download and run a custom script from AWS that performs several tasks:

It installs the docker engine.
It installs the SSM agent and registers it with the Systems Manager’s Fleet manager (which keeps track of the nodes and assigns an IAM role)
It installs the ECS agent, which connects to the ECS cluster and listens for instructions.

3. Deploying the application

There are very few differences between deploying an application to ECS internally on AWS infrastructure here. We set up a Task Definition and a Service within the ECS infrastructure, and the only difference is that we’ll use the new launch type of EXTERNAL instead of EC2 or FARGATE, which tells the cluster to push the instructions for running the application to our nodes.

Deploying a Node.js application using ECS Anywhere

In this example, we will be deploying a basic Node.js application within a container to a pair of virtual machines (VMs) behind a load balancer. You’ll need multiple VMs because ECS won’t deploy the multiple task definitions with the same open network port to the same node (this also happens within the EC2 flavor of ECS).

Let’s be constant

Before setting up our cloud resources, we begin by setting some variables that we’ll use later on throughout our Pulumi program.

We’ll first set up our configuration settings in our command line:

pulumi config set aws:region us-east-1
pulumi config set numberNodes 2

The first tells Pulumi which region to deploy the resources to (and we’ll also use it when setting up our nodes and CloudWatch logging). The second tells us how many nodes to create, and we’ll need at least two to allow ECS to perform a rolling deployment.

Since we’re using a third-party compute provider to prove that ECS can deploy the containers to non-AWS infrastructure, we need to provide an API token. In this example, we’ll add it to the config, but you should read our Setup Guide for more information on how to deploy.

To add your token to the config (as a secret), you should run the following command:

pulumi config set digitalocean:token XXXXXXXXXXXX --secret

Exchange XXXXXXXXXXXX for a personal access token located in the Control Panel.

We’ll use these to set variables in our code:

import * as pulumi from "@pulumi/pulumi";

const awsConfig = new pulumi.Config("aws");
const awsRegion = awsConfig.get("region");

const projectConfig = new pulumi.Config();
const numberNodes = projectConfig.getNumber("numberNodes") || 2;

(notice that for the number of nodes, we’re setting a default value in case this configuration setting does not exist)

Setting up IAM roles

We’re going to need three IAM roles:

The first is for the AWS Systems manager. We’re going to attach two managed policies to this: one to enable its core functionality (AmazonSSMManagedInstanceCore) and the default EC2 role for ECS (AmazonEC2ContainerServiceforEC2Role).
The second is the ECS execution role, and to that, we’re going to attach the managed policy to allow other AWS services to run ECS tasks (AmazonECSTaskExecutionRolePolicy)
The final one is the Task role, and we’re going to add a custom policy to that role to allow us to push logs from the nodes to CloudWatch.

const ssmRole = new aws.iam.Role("ssmRole", {
 assumeRolePolicy: aws.iam.assumeRolePolicyForPrincipal(
 aws.iam.Principals.SsmPrincipal
 ),
});

new aws.iam.RolePolicyAttachment("rpa-ssmrole-ssminstancecore", {
 policyArn: aws.iam.ManagedPolicy.AmazonSSMManagedInstanceCore,
 role: ssmRole,
});

new aws.iam.RolePolicyAttachment("rpa-ssmrole-ec2containerservice", {
 policyArn: aws.iam.ManagedPolicy.AmazonEC2ContainerServiceforEC2Role,
 role: ssmRole,
});

const executionRole = new aws.iam.Role("taskExecutionRole", {
 assumeRolePolicy: aws.iam.assumeRolePolicyForPrincipal(
 aws.iam.Principals.EcsTasksPrincipal
 ),
});

new aws.iam.RolePolicyAttachment("rpa-ecsanywhere-ecstaskexecution", {
 role: executionRole,
 policyArn: aws.iam.ManagedPolicy.AmazonECSTaskExecutionRolePolicy,
});

const taskRole = new aws.iam.Role("taskRole", {
 assumeRolePolicy: aws.iam.assumeRolePolicyForPrincipal(
 aws.iam.Principals.EcsTasksPrincipal
 ),
});

new aws.iam.RolePolicy("taskRolePolicy", {
 role: taskRole.id,
 policy: {
 Version: "2012-10-17",
 Statement: [
 {
 Effect: "Allow",
 Action: [
 "ssmmessages:CreateControlChannel",
 "ssmmessages:CreateDataChannel",
 "ssmmessages:OpenControlChannel",
 "ssmmessages:OpenDataChannel",
 ],
 Resource: "*",
 },
 {
 Effect: "Allow",
 Action: ["logs:DescribeLogGroups"],
 Resource: "*",
 },
 {
 Effect: "Allow",
 Action: [
 "logs:CreateLogStream",
 "logs:CreateLogGroup",
 "logs:DescribeLogStreams",
 "logs:PutLogEvents",
 ],
 Resource: "*",
 },
 ],
 }
});

Deploying the ECS cluster and node infrastructure

Next, we’re going to set up the System Manager activation, which provides an Id and a code that lets our nodes connect to the cluster. We’ll set them up at the same time as the CloudWatch log group that we’ll push application logs to later on:

const ssmActivation = new aws.ssm.Activation("ecsanywhere-ssmactivation", {
 iamRole: ssmRole.name,
 registrationLimit: numberNodes,
});

const cluster = new aws.ecs.Cluster("cluster");

export const clusterName = cluster.name;

const logGroup = new aws.cloudwatch.LogGroup("logGroup");

We’re using the ssmRole we created earlier to allow the Systems Manager to do this. You’ll also notice we’re exporting the clusterName here. We will use this to tidy up later on when we tear down our infrastructure.

When we create the nodes, we will run an AWS bash script provided as part of the setup and pass it into the resource as userData. In this script, we pass in the cluster name, the activation id, the activation code, and the AWS region where the cluster will be deployed.

const userData = pulumi
 .all([ssmActivation.activationCode, ssmActivation.id, cluster.name])
 .apply(
 ([activationCode, activationId, clusterName]) => `#!/bin/bash
# Download the ecs-anywhere install Script
curl -o "ecs-anywhere-install.sh" "https://amazon-ecs-agent.s3.amazonaws.com/ecs-anywhere-install-latest.sh" && sudo chmod +x ecs-anywhere-install.sh

# (Optional) Check integrity of the shell script
curl -o "ecs-anywhere-install.sh.sha256" "https://amazon-ecs-agent.s3.amazonaws.com/ecs-anywhere-install-latest.sh.sha256" && sha256sum -c ecs-anywhere-install.sh.sha256

# Run the install script
sudo ./ecs-anywhere-install.sh \
 --cluster ${clusterName} \
 --activation-id ${activationId} \
 --activation-code ${activationCode} \
 --region ${awsRegion}
`
 );

This next section of code deploys the VMs.

We’re going to create a tag that attaches the nodes to a load balancer. Next, we’ll create the VMs and assign the tag. Finally, we’ll deploy the load balancer and assign the same tag to route traffic to the nodes. We’ll also take the IP address of the load balancer and mark it as an output so we have something we can either use in a browser or call with curl to ensure that our deployment has worked.

In the end, we’re going to output the load balancer IP address so we have something we can either use in a browser or call with curl to ensure that our deployment has worked.

const loadBalancerTag = new digitalocean.Tag("lb");

for(let i = 1; i <= numberNodes; i++) {
 new digitalocean.Droplet(`droplet-${i}`, {
 region: digitalocean.Regions.NYC1,
 size: "s-1vcpu-2gb",
 image: "ubuntu-20-04-x64",
 userData: userData,
 tags: [loadBalancerTag.id]
 });
}

const lb = new digitalocean.LoadBalancer("lb", {
 region: digitalocean.Regions.NYC1,
 forwardingRules: [{
 entryPort: 80,
 entryProtocol: digitalocean.Protocols.HTTP,
 targetPort: 80,
 targetProtocol: digitalocean.Protocols.HTTP
 }],
 healthcheck: {
 port: 80,
 protocol: digitalocean.Protocols.HTTP,
 path: "/"
 },
 dropletTag: loadBalancerTag.name
});

export const ip = lb.ip;

The last part of the example will be familiar to anyone using ECS as a container scheduler: setting up the container repository, describing the task definition, and creating the service. The difference is that we use new compatibility in the task definition and a new launch type in the service - EXTERNAL. This tells ECS to push the container and start them on the external nodes.

const repo = new awsx.ecr.Repository("app");

const image = repo.buildAndPushImage("./app");

const taskDefinition = pulumi.all([image, logGroup.name, logGroup.namePrefix]).apply(
 ([img, logGroupName, nameprefix]) =>
 new aws.ecs.TaskDefinition("taskdefinition", {
 family: "ecs-anywhere",
 requiresCompatibilities: ["EXTERNAL"],
 taskRoleArn: taskRole.arn,
 executionRoleArn: executionRole.arn,
 containerDefinitions: JSON.stringify([
 {
 name: "app",
 image: img,
 cpu: 256,
 memory: 256,
 essential: true,
 portMappings: [
 {
 containerPort: 80,
 hostPort: 80
 },
 ],
 logConfiguration: {
 logDriver: "awslogs",
 options: {
 "awslogs-group": logGroupName,
 "awslogs-region": awsRegion,
 "awslogs-stream-prefixs": nameprefix
 }
 }
 },
 ]),
 })
);

const service = new aws.ecs.Service("service", {
 launchType: "EXTERNAL",
 taskDefinition: taskDefinition.arn,
 cluster: cluster.id,
 desiredCount: numberNodes - 1,
 }
);

Now you can run pulumi up, and once it’s all deployed, run curl ${pulumi stack output ip}.

If you want to update the Dockerfile or the Node.js application, run pulumi up again. You’ll see the new container deployed. Then you can run curl ${pulumi stack output ip} again to see the new response.

Finally, when you want to destroy the infrastructure, there is an additional step before running the usual pulumi destroy. This is because the nodes are registered to AWS Systems Manager and the ECS cluster as part of the node setup and happen outside of the Pulumi stack.

Run the following in your command line (you’ll need to install jq for this to work):

aws ssm describe-instance-information | jq ".InstanceInformationList | .[] | .InstanceId" | grep "mi-" | xargs -L 1 aws ssm deregister-managed-instance --instance-id

# Cleanup ECS resources
aws ecs list-container-instances --cluster ${pulumi stack output clusterName} | jq ".containerInstanceArns | .[]" | xargs -L 1 aws ecs deregister-container-instance --cluster ${pulumi stack output clusterName} --force --container-instance

# Refresh Pulumi stack
pulumi refresh -y

And then you can tear down all your infrastructure with the usual pulumi destroy.

You can see this code in its entirety in our examples repository on GitHub.

Summary

ECS Anywhere is an excellent addition to Amazon Web Services’ suite of container orchestration services. I can see many companies using it to run containers in their private infrastructure but don’t necessarily have the means, know-how, or drive to set up and run their Kubernetes cluster or another such tool.

We’ve shown you how to set up your ECS cluster to use external nodes, build and deploy the cluster and application and see for yourself how easy it is to deploy subsequent changes. Pulumi makes it simple to create container images and provision and manage cloud infrastructure on any cloud using familiar programming languages, including TypeScript, Python, Go, and .NET. Cloud engineers can manage container images, ECR registries, and multi-cloud deployments within the same infrastructure definition.

Deploying an OAuth Server for Netlify's CMS

Zephyr Zhou — Wed, 09 Sep 2020 00:00:00 +0000

Some of the code in this post is out of date. See the AWS guides for an updated overview and examples.

In our previous post, we deployed our CMS app on AWS instead of Netlify. We couldn’t use Netlify’s Identity Service, which manages GitHub access to Netlify CMS, because we deployed on AWS. As a result, we needed to implement an external OAuth Server.

We used Netlify’s Go example to deploy on ECS Fargate and configure the domain and certificate. To deploy the application on Fargate, we used a Typescript Pulumi project. This is a polyglot application where the OAuth server is implemented in Go and the infrastructure is deployed with Typescript. We’ll show how we accomplished the deployment.

Backend is a package that supports communications between Netlify CMS and repositories like GitHub, GitLab, and Bitbucket. The Pulumi example code uses backend to authenticate to the CMS. The OAuth Server also enables authorization for GitLab and Bitbucket by changing the callback URL to https://{{YOUR_OAUTH_SERVER_URL}}/callback/{{YOUR_BACKEND_NAME}}. To learn more about configuring OAuth Server, refer to the Environment Variable and Pulumi Stack Configuration section.

Building the OAuth Server

Netlify’s CMS website provides External OAuth Client examples for various languages and platforms. We used the Go example as a template for our server.

In Netlify’s example, the ./dotenv/dotenv.go retrieves environment variables from a file. The main.go file uses goth to instantiate the OAuth provider. The ./randstr/randstr.go file generates a random string for the SESSION_SECRET environment variable, which is used for authentication. However, Pulumi can implement both functions, replacing the code in goth by using Pulumi’s random package.

// Create a random string and also mark its `result` property as a secret,
// so it is not stored in plaintext in the stack's state.
const sessionSecretRandomString = new random.RandomPassword("random", {
 length: 32,
}, { additionalSecretOutputs: ["result"] });

Because we replaced those functions, we can remove the randstr and dotenv folders in the OAuth client example. Later, we’ll show how to pass the environment variables when we create the Fargate service.

Setting GitHub Scope

The default GitHub scope in Netlify’s example is read-only. When we call the github.New function, read and write access must be set to edit a target repository.

We set GITHUB_SCOPE to public_repo, granting read and write access to a public repository. The GITHUB_SCOPE value is read and parsed when the Fargate Service is created, which is set in the Github.New function in the OAuth server. We can also set GITHUB_SCOPE to a list of scope values.

githubScope := os.Getenv("GITHUB_SCOPE")
 if githubScope == "" {
 goth.UseProviders(
 github.New(
 os.Getenv("GITHUB_KEY"), os.Getenv("GITHUB_SECRET"),
 // concatenate with the host name
 fmt.Sprintf("%s/callback/github", host),
 "public_repo",
 ),
 bitbucket.New(
 os.Getenv("BITBUCKET_KEY"), os.Getenv("BITBUCKET_SECRET"),
 fmt.Sprintf("%s/callback//bitbucket", host),
 ),
 gitlabProvider,
 )
 } else {
 scopeArray := strings.Split(githubScope, ",")
 goth.UseProviders(
 github.New(
 os.Getenv("GITHUB_KEY"), os.Getenv("GITHUB_SECRET"),
 // concatenate with the host name
 fmt.Sprintf("%s/callback/github", host),
 scopeArray...,
 ),
 bitbucket.New(
 os.Getenv("BITBUCKET_KEY"), os.Getenv("BITBUCKET_SECRET"),
 fmt.Sprintf("%s/callback//bitbucket", host),
 ),
 gitlabProvider,
 )
 }

Creating the OAuth Server Image

To build our OAuth server container, replace the working directory in the Dockerfile with the path to your repository. Setting the working directory copies the application when Docker builds the image.

WORKDIR /go/src/github.com/pulumi/aws-ts-netlify-cms-and-oauth/cms-oauth

Infrastructure

Deploying to Fargate

We’ll use the Hello Fargate Example as a template to deploy the OAuth server to AWS Fargate. This example, written in Typescript, creates an ECS cluster, an Application Load Balancer (alb) with a listener, and a Fargate Service. We’ll change the script to deploy our OAuth server Docker image.

We set the alb port to 443 to use HTTPS, which is more secure when passing the CMS access token.

// Define an ec2 application load balancer alb to distribute incomming application traffic across multiple targets, such as EC2 instances, in multiple Availability Zones.
const alb = new awsx.elasticloadbalancingv2.ApplicationLoadBalancer(
 "net-lb", { external: true, securityGroups: cluster.securityGroups });

// alb need a listener to listen to 443 the standard port for the HTTPS traffic, certificate is using the certificate we created above
const web = alb.createListener("web", {
 port: 443,
 external: true,
 protocol: "HTTPS",
 certificateArn: certificate.arn
});

We create a target group for the load balancers to distribute traffic. The target group port defaults port 80 if it’s not set.

let inputTargetGroupPort: pulumi.Input<number> = cmsStackConfig.targetGroupPort!;

if (inputTargetGroupPort === undefined) {
 inputTargetGroupPort = 80;
}

// when Listener Rule is satisfied then traffic is route to this target group.
const tg = alb.createTargetGroup("oauth-tg", {
 port: inputTargetGroupPort,
 loadBalancer: alb
});

We can also set the target group port as an environment variable with pulumi config set cms-OAuth:targetGroupPort. The load balancer implements a listener and a rule to forward all requests to the target group.

// when the request are forwarded then every requests are send to the target group we created
new awsx.lb.ListenerRule("oauth-listener-rule", web, {
 actions: [{
 type: "forward",
 targetGroupArn: tg.targetGroup.arn,
 }],
 conditions: [{
 field: "path-pattern",
 values: "/*", //wildcard says every request would be send
 }],
});

Environment Variables and Pulumi Stack Configuration

The Fargate service requires several environment variables to start. Here are the variables we need to configure:

HOST is the URL for the OAuth Server, and we set it by running pulumi config set pulumi-website-cms:targetDomain <OAuth server URL>
SESSION_SECRET is generated by using Pulumi’s random package
GITHUB_SECRET and GITHUB_TOKEN are provided by GitHub when the GitHub OAuth application is registered. Netlify provides instructions for obtaining these credentials
The targetDomain is the URL of our CMS
GITHUB_SCOPE sets the type of access the CMS has for the target repository

We can set the github key and secret in the stack configuration. The –secret flag encrypts your GitHub secret. It is good practice to use the –secret flag, because appending the secret in the command line stores it in the command history. The –secret flag lets you enter the secret at the prompt, which hides the value.

$ pulumi config set netlify-cms-oauth-provider-infrastructure:githubKey {{YOUR_GITHUB_KEY}}
$ pulumi config set --secret netlify-cms-oauth-provider-infrastructure:githubSecret
$ {{YOUR_GITHUB_SECRET}}

When the Fargate service is created, it uses the environment variables set in the Pulumi project file and generates the session secret.

// Create a Fargate service task that can scale out.
const appService = new awsx.ecs.FargateService("app-svc", {
 cluster,
 taskDefinitionArgs: {
 container: {
 image: img,
 memory: 128 /*MB*/,
 portMappings: [ tg ],
 environment: [
 {
 name: "HOST",
 // The target domain which would concatenate with callbacks in main.go
 value: pulumi.interpolate `https://${cmsStackConfig.targetDomain}`
 },
 {
 name: "SESSION_SECRET",
 value: sessionSecretRandomString.result
 },
 {
 name: "GITHUB_KEY",
 value: cmsStackConfig.githubKey
 },
 {
 name: "GITHUB_SECRET",
 value: cmsStackConfig.githubSecret
 },
 {
 name: "TARGET_PORT",
 value: pulumi.interpolate `${inputTargetGroupPort}`
 },
 {
 name: "GITHUB_SCOPE",
 value: inputGithubScope
 },
 ]
 },
 },
 desiredCount: 1,
});

The environment variables are passed to the OAuth server in main.go. For example, the callback URL is the targetDomain with callback and the repository provider appended, e.g., https://<targetDomain>/callback/github/. You can also specify a different repository provider, such as Bitbucket or GitLab.

 github.New(
 os.Getenv("GITHUB_KEY"), os.Getenv("GITHUB_SECRET"),
 // concatenate with the host name
 fmt.Sprintf("%s/callback/github", host),
 "public_repo",
 )

We also pass TARGET_PORT to targetGroupPort, which is the port used to serve the application. We can check if this variable is set in main.go if we need to test locally without Pulumi.

targetGroupPort := os.Getenv("TARGET_PORT")
 if targetGroupPort == "" {
 targetGroupPort = "80"
 }
 listenPort := ":" + targetGroupPort
 fmt.Print("Started running on", listenPort, "\n")
 // listen on port 80 where we created the target group
 fmt.Println(http.ListenAndServe(listenPort, nil))

Create a Certificate for the OAuth Server

We use the same process for creating a certificate that’s documented in our previous post, but we use only the code for creating the certificate and discard the rest. The certificate used with the load balancer must be set to the same region as the fully qualified domain name (FQDN). The domain name in the certificate must be validated.

// Get a east provider
const eastRegion = new aws.Provider("east", {
 profile: aws.config.profile,
 region: "us-east-1", // Per AWS, ACM certificate must be in the us-east-1 region.
});

// Creating a Certificate for the given domain.
const certificate = new aws.acm.Certificate("certificate", {
 domainName: cmsStackConfig.targetDomain,
 validationMethod: "DNS",
}, { provider: eastRegion });

// Split given domain in the configuration to domain and subdomain
const domainParts = getDomainAndSubdomain(cmsStackConfig.targetDomain);

// Get the zone of the given domain
const hostedZoneId = aws.route53.getZone({ name: domainParts.parentDomain }, { async: true }).then(zone => zone.zoneId);

// The temporation record for the validation domain has 10 to be live
const tenMinutes = 60 * 10;

/**
 * Create a DNS record to prove that we _own_ the domain we're requesting a certificate for.
 * See https://docs.aws.amazon.com/acm/latest/userguide/gs-acm-validate-dns.html for more info.
 */
const certificateValidationDomain = new aws.route53.Record(`${cmsStackConfig.targetDomain}-validation`, {
 name: certificate.domainValidationOptions[0].resourceRecordName,
 zoneId: hostedZoneId,
 type: certificate.domainValidationOptions[0].resourceRecordType,
 records: [certificate.domainValidationOptions[0].resourceRecordValue],
 ttl: tenMinutes,
});

We configure the Alias Record with the application load balancer settings.

// Creates a new Route53 DNS record pointing the domain to the CloudFront distribution.
function createAliasRecord(
 targetDomain: string, lb: awsx.elasticloadbalancingv2.ApplicationLoadBalancer): aws.route53.Record {
 const domainParts = getDomainAndSubdomain(targetDomain);
 const hostedZoneId = aws.route53.getZone({ name: domainParts.parentDomain }, { async: true }).then(zone => zone.zoneId);
 return new aws.route53.Record(
 targetDomain,
 {
 name: domainParts.subdomain,
 zoneId: hostedZoneId,
 type: "A",
 aliases: [
 {
 name: lb.loadBalancer.dnsName,
 zoneId: lb.loadBalancer.zoneId,
 evaluateTargetHealth: true,
 },
 ],
 });
}
// Create the aliasRecord with targetdomain and application load balancer
const aRecord = createAliasRecord(cmsStackConfig.targetDomain, alb);

GitHub Workflow (Optional)

We can automate our build with GitHub Workflow. The build-and-deploy.yml file in cms-oauth/.github/workflows/ is used by GitHub Actions to build our application. Note that the workflow uses the repository secret set in GitHub.

Finishing Up

Our last task is to update the CMS configuration file. We combine the CMS and CMS OAuth Server deployment by setting site_domain and URL in ./public/config.yml of the CMS folder.

backend:
 name: github
 # Replace this with the GitHub repo you want to make change
 repo: github-username/target-github-repo
 # This site_domain and base_url are placeholders, replace with the correct one.
 site_domain: https://some-cms-domain.com # CMS's domain
 base_url: https://some-oauth-domain.com # OAuth Server's domain

The site_domain is the CMS URL, and the base_url is the OAuth Server URL. They are the same as the targetDomain variable we set in the stack configuration.

Up and Running

Congratulations on building and deploying both CMS web application and the OAuth client-server! If everything works perfectly, you should see the Login with GitHub button and click it to redirect to the GitHub login page.

People with access can now use the CMS.

You can find the project code for the CMS and the OAuth Client Server on GitHub.

Special Thanks to:

Tony Alves @talves for providing the template to separate CMS as a stand-alone React App
Igor Kuznetsov @igk1972 for providing go code example for providing the OAuth Client-Server source code
Paul Stack @stack72 for developing Pulumi example of deploying Dockerized App using ECS Fargate and Pulumi example of deploying the static website on AWS
Everyone at Pulumi who helped me out with this project.

Creating a Python AWS Application Using Flask and Redis

Vova Ivanov — Thu, 13 Aug 2020 15:06:26 -0700

Meet Vova Ivanov—one of the Pulumi summer interns. He’ll be writing about his experiences learning Pulumi while modernizing a web app and its underlying infrastructure.

I’ve recently started developing with Pulumi, and have begun to explore its inner workings and mechanisms. I decided to construct a production-level application and to document each step that I take and my progress as I go along.

This blog post recreates the existing Typescript voting app example step by step in Python with Flask as the frontend and Redis as the backend. In future blog posts, we will explore how to change the front and backends, how to upgrade the app with additional AWS services, and migrating from one cloud provider to another.

The first step is to clone the Typescript voting app example. We’ll use a sparse clone to copy only the aws-ts-voting-app directory.

$ mkdir examples && cd examples
$ git init
$ git remote add origin -f https://github.com/pulumi/examples/
$ git config core.sparseCheckout true
$ echo aws-ts-voting-app >> .git/info/sparse-checkout
$ git pull origin master

The next step is to create a new directory and initialize a Pulumi project with pulumi new aws-python.

$ mkdir aws-py-voting-app && cd aws-py-voting-app
$ pulumi new aws-python

Since we won’t be modifying the application’s frontend, we simply copy the frontend/ folder from the example into our directory.

$ cp -r aws-ts-voting-app/frontend/ aws-py-voting-app/frontend

The requirements.txt file lists the libraries that the project depends on. We will need to add the following:

pulumi-docker>=2.0.0,<3.0.0

The biggest difference between our project and the example is the code describing the cloud infrastructure we want to create. Instead of an index.ts file, the project uses a __main__.py file. The first few lines of the file will indicate which libraries to import and describe a pair of configuration options used by the application.

import json
import pulumi
import pulumi_aws as aws
import pulumi_docker as docker

config = pulumi.Config()
redis_password = config.require("redis-password")
redis_port = 6379

After setting up the imports and configurations, we create an Elastic Container Service Cluster. A Cluster represents a group of tasks and services that work together for a certain purpose. In this instance, the purpose is to provide users with a voting application.

app_cluster = aws.ecs.Cluster("app-cluster")

To allow different tasks within our cluster to communicate, we create a Virtual Private Cloud and an associated subnet.

app_vpc = aws.ec2.Vpc("app-vpc",
 cidr_block="172.31.0.0/16",
 enable_dns_hostnames=True)

app_vpc_subnet = aws.ec2.Subnet("app-vpc-subnet",
 cidr_block="172.31.32.0/20",
 vpc_id=app_vpc)

A gateway and routing table are needed to allow the VPC to communicate with the Internet. Once created, we associate the routing table with our VPC.

app_gateway = aws.ec2.InternetGateway("app-gateway",
 vpc_id=app_vpc.id)

app_routetable = aws.ec2.RouteTable("app-routetable",
 routes=[
 {
 "cidr_block": "0.0.0.0/0",
 "gateway_id": app_gateway.id,
 }
 ],
 vpc_id=app_vpc.id)

app_routetable_association = aws.ec2.MainRouteTableAssociation("app_routetable_association",
 route_table_id=app_routetable.id,
 vpc_id=app_vpc)

To control traffic flow between applications running inside our VPC, we create a security group.

app_security_group = aws.ec2.SecurityGroup("security-group",
 vpc_id=app_vpc.id,
 description="Enables HTTP access",
 ingress=[{
 'protocol': 'tcp',
 'from_port': 0,
 'to_port': 65535,
 'cidr_blocks': ['0.0.0.0/0'],
 }],
 egress=[{
 'protocol': '-1',
 'from_port': 0,
 'to_port': 0,
 'cidr_blocks': ['0.0.0.0/0'],
 }])

To start our services, we need to create an Identity and Access Management (IAM) role, and attach execution permissions to it.

app_exec_role = aws.iam.Role("app-exec-role",
 assume_role_policy="""{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Action": "sts:AssumeRole",
 "Principal": {
 "Service": "ecs-tasks.amazonaws.com"
 },
 "Effect": "Allow",
 "Sid": ""
 }]
 }""")

exec_policy_attachment = aws.iam.RolePolicyAttachment("app-exec-policy", role=app_exec_role.name,
 policy_arn=aws.iam.ManagedPolicy.AMAZON_ECS_TASK_EXECUTION_ROLE_POLICY)

Likewise, our ECS service will need to have a task role to manage it, along with its own set of permissions.

app_task_role = aws.iam.Role("app-task-role",
 assume_role_policy="""{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Action": "sts:AssumeRole",
 "Principal": {
 "Service": "ecs-tasks.amazonaws.com"
 },
 "Effect": "Allow",
 "Sid": ""
 }]
 }""")

task_policy_attachment = aws.iam.RolePolicyAttachment("app-access-policy", role=app_task_role.name,
 policy_arn=aws.iam.ManagedPolicy.AMAZON_ECS_FULL_ACCESS)

An Elastic Container Registry Repository stores the application Docker images that we want to run. The life cycle policy automatically removes the oldest untagged image that we uploaded.

app_ecr_repo = aws.ecr.Repository("app-ecr-repo",
 image_tag_mutability="MUTABLE")

app_lifecycle_policy = aws.ecr.LifecyclePolicy("app-lifecycle-policy",
 repository=app_ecr_repo.name,
 policy="""{
 "rules": [
 {
 "rulePriority": 10,
 "description": "Remove untagged images",
 "selection": {
 "tagStatus": "untagged",
 "countType": "imageCountMoreThan",
 "countNumber": 1
 },
 "action": {
 "type": "expire"
 }
 }
 ]
 }""")

With the basic infrastructure in place, we can begin to set up the application itself. First, we will start with the Redis backend.

We need to create a target group, a balancer, and a listener to allow Redis to communicate with the other services in our VPC, w

redis_targetgroup = aws.lb.TargetGroup("redis-targetgroup",
 port=redis_port,
 protocol="TCP",
 target_type="ip",
 stickiness= {
 "enabled": False,
 "type": "lb_cookie",
 },
 vpc_id=app_vpc.id)

redis_balancer = aws.lb.LoadBalancer("redis-balancer",
 load_balancer_type="network",
 internal= False,
 security_groups=[],
 subnets=[app_vpc_subnet.id])

redis_listener = aws.lb.Listener("redis-listener",
 load_balancer_arn=redis_balancer.arn,
 port=redis_port,
 protocol="TCP",
 default_actions=[{
 "type": "forward",
 "target_group_arn": redis_targetgroup.arn
 }])

Every service running within ECS requires a task definition, specifying what hardware, internet, and container settings it will use.

redis_task_definition = aws.ecs.TaskDefinition("redis-task-definition",
 family="redis-task-definition-family",
 cpu="256",
 memory="512",
 network_mode="awsvpc",
 requires_compatibilities=["FARGATE"],
 execution_role_arn=app_exec_role.arn,
 task_role_arn=app_task_role.arn,
 container_definitions=json.dumps([{
 "name": "redis-container",
 "image": "redis:alpine",
 "memory": 512,
 "essential": True,
 "portMappings": [{
 "containerPort": redis_port,
 "hostPort": redis_port,
 "protocol": "tcp"
 }],
 "command": ["redis-server", "--requirepass", redis_password],
 }]))

Now that we have our task definition blueprint ready, we can launch a service using it. An endpoint is created to hold information needed to connect the backend with the frontend.

redis_service = aws.ecs.Service("redis-service",
 cluster=app_cluster.arn,
 desired_count=1,
 launch_type="FARGATE",
 task_definition=redis_task_definition.arn,
 wait_for_steady_state=False,
 network_configuration={
 "assign_public_ip": "true",
 "subnets": [app_vpc_subnet.id],
 "security_groups": [app_security_group.id]
 },
 load_balancers=[{
 "target_group_arn": redis_targetgroup.arn,
 "container_name": "redis-container",
 "container_port": redis_port,
 }],
 opts=pulumi.ResourceOptions(depends_on=[redis_listener]),
)

redis_endpoint = {"host": redis_balancer.dns_name, "port": redis_port}

The Redis backend is complete, and now all that’s left is to create the Flask frontend.

A similar set of a target group, balancer, and listener is created for the frontend. Unlike Redis, Flask will serve as the webpage and, as such, needs to receive its data from port 80.

flask_targetgroup = aws.lb.TargetGroup("flask-targetgroup",
 port=80,
 protocol="TCP",
 target_type="ip",
 stickiness= {
 "enabled": False,
 "type": "lb_cookie",
 },
 vpc_id=app_vpc.id)

flask_balancer = aws.lb.LoadBalancer("flask-balancer",
 load_balancer_type="network",
 internal=False,
 security_groups=[],
 subnets=[app_vpc_subnet.id])

flask_listener = aws.lb.Listener("flask-listener",
 load_balancer_arn=flask_balancer.arn,
 port=80,
 protocol="TCP",
 default_actions=[{
 "type": "forward",
 "target_group_arn": flask_targetgroup.arn
 }])

The application is built into a Docker image and pushed to our ECR repository we created earlier.

def get_registry_info(rid):
 creds = aws.ecr.get_credentials(registry_id=rid)
 decoded = base64.b64decode(creds.authorization_token).decode()
 parts = decoded.split(':')
 if len(parts) != 2:
 raise Exception("Invalid credentials")
 return docker.ImageRegistry(creds.proxy_endpoint, parts[0], parts[1])

app_registry = app_ecr_repo.registry_id.apply(get_registry_info)

flask_image = docker.Image("flask-dockerimage",
 image_name=app_ecr_repo.repository_url,
 build="./frontend",
 skip_push=False,
 registry=app_registry
)

A task definition is created for the frontend; this time, using the flask image that we built. The redis endpoint is passed in as a group of environment variables, informing our app of our database’s host and port values.

flask_task_definition = aws.ecs.TaskDefinition("flask-task-definition",
 family="frontend-task-definition-family",
 cpu="256",
 memory="512",
 network_mode="awsvpc",
 requires_compatibilities=["FARGATE"],
 execution_role_arn=app_exec_role.arn,
 task_role_arn=app_task_role.arn,
 container_definitions=pulumi.Output.all(flask_image.image_name, redis_endpoint).apply(lambda args: json.dumps([{
 "name": "flask-container",
 "image": args[0],
 "memory": 512,
 "essential": True,
 "portMappings": [{
 "containerPort": 80,
 "hostPort": 80,
 "protocol": "tcp"
 }],
 "environment": [
 { "name": "REDIS", "value": args[1]["host"] },
 { "name": "REDIS_PORT", "value": str(args[1]["port"]) },
 { "name": "REDIS_PWD", "value": redis_password },
 ],
 }])))

The service for our Flask frontend is launched. It can receive requests from the Internet, process them, and change the Redis database accordingly.

flask_service = aws.ecs.Service("flask-service",
 cluster=app_cluster.arn,
 desired_count=1,
 launch_type="FARGATE",
 task_definition=flask_task_definition.arn,
 wait_for_steady_state=False,
 network_configuration={
 "assign_public_ip": "true",
 "subnets": [app_vpc_subnet.id],
 "security_groups": [app_security_group.id]
 },
 load_balancers=[{
 "target_group_arn": flask_targetgroup.arn,
 "container_name": "flask-container",
 "container_port": 80,
 }],
 opts=pulumi.ResourceOptions(depends_on=[flask_listener]),
)

To connect to our application, we export the DNS name of the Flask balancer, and open it in a browser window.

pulumi.export("app-url", flask_balancer.dns_name)

In this example, I showed how straightforward it is to convert infrastructure code defined in TypeScript to one defined in Python. Teams often choose to standardize on one language and Pulumi expands that choice, allowing them to standardize on one language for the frontend, the backend, and the infrastructure.

Next week, I’ll show how to modernize the application by changing the frontend from Flask to Django, and swapping out the database from Redis to Amazon RDS—using Pulumi to stand up new resources as I go.

The full code for the blog post can be found on github.

Managing AWS Containers with Pulumi

Sophia Parafina — Mon, 06 Jan 2020 00:00:00 +0000

Some of the code in this post is out of date. See the AWS guides for an updated overview and examples.

The Amazon Web Services (AWS) Cloud ecosystem is large and vibrant, so vast and vibrant that at times, it can be challenging to know where best to start! In the case of containers, Abby Fuller tweeted a descriptive summary about using AWS container services.

It looks straightforward, but it’s easy to get lost in the details when configuring multiple services to get your application running on AWS. Although AWS has extensive documentation and a point and click interface to these services, replicating then can be challenging, and it isn’t conducive to understanding the architecture. However, using infrastructure as code lets you see the details and results in a reproducible deployment that you can expand on. Let’s implement the diagram using Pulumi to deploy your container infrastructure with code in AWS.

Store your images

The AWS Elastic Container Registry (ECR) is a container registry that supports private container registries. ECR makes it easy to store, build, manage, and deploy container images and eliminates the need to operate your registry or use public registries, all in a highly available and scalable architecture.

In this example, we use Pulumi’s Crosswalk for AWS. Crosswalk for AWS is a collection of frequent tasks and best practices that simplify deploying infrastructure on AWS. We first declare a repository; this creates the AWS Container Repository resource. The last line of the code exports the URL so that we can access the repository after we have updated our Pulumi stack. After we’ve run pulumi up, the repository is ready to go, and you can use the Docker CLI to push, pull, and manage images.

import * as awsx from "@pulumi/awsx";

// Create a repository, as before.
const repo = new awsx.ecr.Repository("my-repo", {
 forceDelete: true,
});

// And publish its URL, so we can push to it if we'd like.
export const url = repo.repository.repositoryUrl;

Choose your scheduler

There are two services for running containers in the AWS Cloud: Elastic Cloud Service (ECS), or Elastic Kubernetes Service (EKS). ECS is a proprietary service for running containers in the AWS Cloud. In contrast, EKS is a managed Kubernetes service for containers using a Kubernetes control plane. It has three master nodes distributed across three availability zones to provide high availability.

ECS is designed to work with other AWS services and provides more straightforward configuration and integration with them while providing high availability. In contrast, EKS is a managed Kubernetes service, and because Kubernetes is open-source, your infrastructure is portable to other cloud providers. Also, Kubernetes can provide fine-grain control over deployed services. The choice of which scheduler to use depends on your requirements.

You can create either an ECS or EKS cluster using Pulumi. If you wish to use ECS, @pulumi/aws provides all the primitives needed to build infrastructure on AWS.

import * as awsx from "@pulumi/aws";

// Create an ECS cluster explicitly, and give it a name tag.
const cluster = new aws.ecs.Cluster("custom", {
 tags: {
 "Name": "my-custom-ecs-cluster",
 },
});

If you wish to use EKS, use the Pulumi EKS library (@pulumi/eks) for Kubernetes primitives on AWS.

import * as eks from "@pulumi/eks";

// Create an EKS cluster with the default configuration.
const cluster = new eks.Cluster("my-cluster");

Choose where/how to run, Fargate or EC2

Fargate is an AWS service that runs containers. It is suited to running small workloads or batch workloads with occasional bursts that require scaling quickly. AWS EC2 is ideal for large workloads requiring many CPU cores and gigabytes of memory.

Fargate

In this example, we create a load balancer open on port 80, spin up two instances of our container, and publish the endpoint URL.

import * as pulumi from "@pulumi/pulumi";

const lb = new awsx.lb.ApplicationLoadBalancer("nginx");

const service = new awsx.ecs.FargateService("service", {
 cluster: cluster.arn,
 assignPublicIp: true,
 taskDefinitionArgs: {
 containers: {
 nginx: {
 image: image.imageUri,
 essential: true,
 portMappings: [{
 containerPort: 80,
 targetGroup: lb.defaultTargetGroup,
 }],
 }
 }
 },
 desiredCount: 2,
});

// Export the load balancer's address so that it's easy to access.
export const appURL = pulumi.interpolate`http://${loadbalancer.loadBalancer.dnsName}`;

EC2 with Kubernetes

With Kubernetes, we create a deployment for the application and a service to make the application accessible via port 80. We publish both application endpoint to make it accessible and cluster config for use with kubectl CLI.

// Deploy a small canary service (NGINX), to test that the cluster is working.
const appName = "my-app";
const appLabels = { appClass: appName };
const deployment = new k8s.apps.v1.Deployment(`${appName}-dep`, {
 metadata: { labels: appLabels },
 spec: {
 replicas: 2,
 selector: { matchLabels: appLabels },
 template: {
 metadata: { labels: appLabels },
 spec: {
 containers: [{
 name: appName,
 image: "nginx",
 ports: [{ name: "http", containerPort: 80 }]
 }],
 }
 }
 },
}, { provider: cluster.provider });
const service = new k8s.core.v1.Service(`${appName}-svc`, {
 metadata: { labels: appLabels },
 spec: {
 type: "LoadBalancer",
 ports: [{ port: 80, targetPort: "http" }],
 selector: appLabels,
 },
}, { provider: cluster.provider });

// Export the URL for the load balanced service.
export const url = service.status.loadBalancer.ingress[0].hostname;

// Export the cluster's kubeconfig.
export const kubeconfig = cluster.kubeconfig;

Build!

Let’s put it all together for both ECS and EKS. To get started, follow the Pulumi AWS documentation to install Pulumi, install AWS Client set your AWS environment variables, and create a new project in Typescript.

$ mkdir get-started && cd get-started
$ pulumi new aws-typescript

In this example, we build the application in code using the awsx.ecr.Image resource. This resource uses Docker to build the image locally and push it to our repository. Make sure that you have Docker installed and running.

The container application for this example is an HTML page in NGINX. Make a ./app directory in your Pulumi project and add the Dockerfile below.

Dockerfile:

FROM nginx
COPY content /usr/share/nginx/html

Create a ./content director in the ./app directory and add the HTML file below.

<html>
 <head><meta charset="UTF-8">
 <title>Hello, Pulumi!</title>
 </head>
 <body>
 <p>Hello, containers!</p>
 <p>Made with ❤️ with <a href="https://pulumi.com">Pulumi</a></p>
 </body>
</html>

Now that setup is complete you project directory should look similar to this:

get-started/
├── Pulumi.dev.yaml
├── Pulumi.yaml
├── app
│   ├── Dockerfile
│   └── content
│   └── index.html
├── index.ts
├── node_modules
├── package-lock.json
├── package.json
└── tsconfig.json

ECS

Now we’re ready to start building our infrastructure. Replace the generated index.ts file with the example below. The example shows how to deploy containers on ECS using Fargate.

import * as awsx from "@pulumi/awsx";

// Create a repository
const repo = new awsx.ecr.Repository("my-repo", {
 forceDelete: true,
});

// Build an image from the "./app" directory
// and publish it to our ECR repository.
export const image = new awsx.ecr.Image("image", {
 repositoryUrl: repo.url,
 path: "./app",
});

// Create an ECS cluster explicitly, and give it a name tag.
const cluster = new aws.ecs.Cluster("custom", {
 tags: {
 "Name": "my-custom-ecs-cluster",
 },
});

// load balancer on port 80
const lb = new awsx.lb.ApplicationListener("nginx", { port: 80 });
const service = new awsx.ecs.FargateService("nginx", {
 cluster,
 taskDefinitionArgs: {
 containers: {
 nginx: {
 image: image.imageUri,
 memory: 512,
 portMappings: [ lb ],
 },
 },
 },
 desiredCount: 2,
});

// Publish the load balancer address so it's accessible.
export const appURL = lb.endpoint.hostname;

// Publish the repository URL, so we can push to it.
export const repoURL = repo.repository.repositoryUrl;

To deploy the application, run pulumi up. You’ll see a preview of the resources to be deployed. Accept the deployment by selecting yes, and when finished, your program lists the URLs for the application and the repository.

Outputs:
 appURL : "nginx-0296a2a-996897041.us-east-1.elb.amazonaws.com"
 image : "153052954103.dkr.ecr.us-east-1.amazonaws.com/my-repo-41f69f4:37c8add0fbd01474806c9c2319e62a857737de9e1460b09f935133860a069f4e"
 repoURL: "153052954103.dkr.ecr.us-east-1.amazonaws.com/my-repo-41f69f4"

Test to see if your application is working using either a browser or curl to see the web page.

$ curl nginx-0296a2a-996897041.us-east-1.elb.amazonaws.com
<html>
 <head><meta charset="UTF-8">
 <title>Hello, Pulumi!</title>
 </head>
 <body>
 <p>Hello, containers!</p>
 <p>Made with ❤️ with <a href="https://pulumi.com">Pulumi</a></p>
 </body>
</html>

EKS

The EKS example uses the @pulumi/eks and @pulumi/kubernetes node modules. Add the eks and kubernetes modules to the project.

$ node install “@plulumi/eks”
$ node install “@plulumi/kubernetes”

Like the ECS example, the EKS example builds the application image locally and pushes it into our custom repository. The application deployment has two replicas in the configuration. The service uses the EKS cluster, my-cluster, we declared to create a load balancer with port 80 open.

import * as awsx from "@pulumi/awsx";
import * as eks from "@pulumi/eks";
import * as k8s from "@pulumi/kubernetes";

// Create a repository, as before.
const repo = new awsx.ecr.Repository("my-repo", {
 forceDelete: true,
});

// Build an image from the "./app" directory. and publish it to our ECR repository.
export const image = new awsx.ecr.Image("image", {
 repositoryUrl: repo.url,
 path: "./app",
});

// Create an EKS cluster with the default configuration.
const cluster = new eks.Cluster("my-cluster")

// Deploy a small canary service (NGINX), to test that the cluster is working.
const appName = "my-app";
const appLabels = { appClass: appName };
const deployment = new k8s.apps.v1.Deployment(`${appName}-dep`, {
 metadata: { labels: appLabels },
 spec: {
 replicas: 2,
 selector: { matchLabels: appLabels },
 template: {
 metadata: { labels: appLabels },
 spec: {
 containers: [{
 name: appName,
 image: image.imageUri,
 ports: [{ name: "http", containerPort: 80 }]
 }],
 }
 }
 },
}, { provider: cluster.provider });
const service = new k8s.core.v1.Service(`${appName}-svc`, {
 metadata: { labels: appLabels },
 spec: {
 type: "LoadBalancer",
 ports: [{ port: 80, targetPort: "http" }],
 selector: appLabels,
 },
}, { provider: cluster.provider });

// Export the cluster's kubeconfig.
export const kubeconfig = cluster.kubeconfig;

// Publish the repository URL.
export const repoURL = repo.repository.repositoryUrl;

// Publish the URL for the load balanced service.
export const appURL = service.status.loadBalancer.ingress[0].hostname;

As with the ECS example, run pulumi up to deploy the application. To check the deployment, use either curl or a browser to see the page.

Want to know more?

Although these examples are simple, they demonstrate the basic building blocks for building, storing, and managing AWS Containers. They also show how to create ECS or EKS clusters for deploying apps. To get started with AWS and Pulumi check out the AWS Guide for core services and the Crosswalk for AWS guide for convenience APIs that simplify deploying infrastructure as code. For a deeper dive into managing containers on AWS, check out our blogs on ECS vs Fargate vs EKS: The Lowdown on Containers in AWS and How to Scale Your Amazon EKS Cluster: EC2, Managed Node Groups, and Fargate.

ECS vs Fargate vs EKS: The Lowdown on Containers in AWS

Joe Duffy — Thu, 20 Jun 2019 00:00:00 +0000

Some of the code in this post is out of date. See the AWS guides for an updated overview and examples.

An updated breakdown of when to prefer Fargate and when EC2 can be found in Fargate vs EC2

Amazon offers multiple solutions for running containers in AWS, through its managed Elastic Container Service (ECS). This includes three major approaches: ECS managed automatically with Fargate, ECS backed by EC2 instances, and Elastic Kubernetes Service (EKS), delivering the full power of Kubernetes. It's not always easy to choose between these, so in this article we provide some basic guidance on the tradeoffs you'll encounter when choosing.

The Options: ECS vs Fargate vs EKS

At Pulumi, we work with customers to deploy AWS applications all the time, from early development to scaling production environments worldwide, using infrastructure as code and often continuous delivery. During this, we’ve run across a breadth of container workloads and requirements, and developed best practices, which this article distills into some basic advice. As with any advice, the right answer depends on your scenario’s specific requirements.

Here are the three major options, roughly speaking from easiest to hardest:

Option 1 - ECS Fargate

Amazon ECS with Fargate launch type allows you to deploy and scale your containerized workloads, automatically, without needing to manage clusters of EC2 instances by hand. You focus on your Dockerized applications, with minimal extra declarative metadata about the requirements of those applications, and not the details of the underlying compute itself. One could say this is “serverless containers.” Of course, there are servers running your containers, however the ECS control plane makes informed decisions on your behalf about when to add capacity, where, and by how much.

To run a workload in ECS, you use a standard Dockerfile to author, build, and publish your container image, and then run that image inside your ECS cluster. Tasks and services are authored using ECS definitions, which specify the container image to run, instance count, and any CPU, memory, networking, and disk requirements your workload has.

Using Pulumi, these definitions can be easily expressed declaratively in code, like so:

import * as awsx from "@pulumi/awsx";

// Create a load balancer on port 80 and spin up two instances of Nginx.
const lb = new awsx.lb.ApplicationListener("nginx", { port: 80 });
const nginx = new awsx.ecs.FargateService("nginx", {
 taskDefinitionArgs: {
 containers: {
 nginx: {
 image: "nginx",
 memory: 128,
 portMappings: [ lb ],
 },
 },
 },
 desiredCount: 2,
});

// Export the load balancer's address so that it's easy to access.
export const url = lb.endpoint.hostname;

This example runs 2 load balanced instances of an NGINX web server using ECS Fargate.

ECS Fargate is often a great choice when getting started, when you don’t have stringent requirements around cost, and/or if you’re a small to medium sized organization who prefers simpler solutions. Many teams are successful running it scale too, however Fargate doesn’t give you as much control over persistent storage, privileged containers, custom scheduling, or special VM types like GPUs or FPGAs. For those, EC2 backed ECS is a better choice.

Option 2 - ECS Backed by EC2 Instances

Amazon ECS with EC2 launch type gives you similar capabilities to ECS managed by Fargate – in that you can deploy containerized workloads, declaratively specify their requirements, and let ECS do the placement and overall scheduling of them – but with full control over the compute environment those containers run in. That includes control over the precise numbers of servers, AMIs they are running, and their auto-scaling policies.

To use an EC2 backed cluster, you will need to create a cluster and configure the EC2 launch and autoscaling configurations, and then simply use the awsx.ecs.EC2Service class to create your service instead. Other than these differences, the resulting code would look very similar to Option 1 - ECS Fargate above.

For sophisticated AWS organizations who already know how to fine tune and run compute at scale efficiently and cost effectively, managing your ECS cluster’s EC2 instances explicitly is often a good choice. The good news is that you can easily start with Fargate, and then over time, shift to managing the EC2 compute by hand if you prefer. Managing your cluster amounts to managing fleets of EC2 instances, CloudWatch logging, and standard AWS services.

Option 3 - Elastic Kubernetes Service (EKS)

Amazon Elastic Kubernetes Service (EKS) offers a very different approach to running containerized workloads than ECS Fargate or EC2. EKS is a fully managed offering that runs Kubernetes inside of your AWS account, making it easier to operate Kubernetes, in addition to integrating with many AWS services like ELB for load balancing, IAM for cluster authentication, and CloudWatch for logging and diagnostics. Using EKS, you don’t need to explicitly provision, manage, and upgrade the Kubernetes control plane’s configuration, compute, and storage.

Kubernetes is an open standard for running containers across a variety of public and private cloud environments. Most public cloud vendors now offer an EKS-like managed offering, including Azure (AKS), Google Cloud (GKE), and Digital Ocean, and the availability of on-premises private cloud configurations enables hybrid cloud for large organizations. Kubernetes in inherently more complicated to operate than the equivalent ECS solutions mentioned earlier, because it requires becoming an expert in Kubernetes, which requires more specialization in its tools and management practices. But the potential payoff is larger for organizations wanting to go “all in” on cloud native technologies to facilitate either multi-cloud or hybrid computing – challenges most large enterprises are facing today.

In the EKS case, as with ECS, we continue using standard Dockerfiles, however we use the Kubernetes object model to configure and run workloads. Instead of task definitions, you use Kubernetes objects like Namespaces, Pods, Deployments, and Services. This full API is available to specify in code using Pulumi, like so:

import * as eks from "@pulumi/eks";
import * as k8s from "@pulumi/kubernetes";

// Create an EKS cluster with the default configuration.
const cluster = new eks.Cluster("my-cluster");

// Deploy a small canary service (NGINX), to test that the cluster is working.
const appName = "my-app";
const appLabels = { appClass: appName };
const deployment = new k8s.apps.v1.Deployment(`${appName}-dep`, {
 metadata: { labels: appLabels },
 spec: {
 replicas: 2,
 selector: { matchLabels: appLabels },
 template: {
 metadata: { labels: appLabels },
 spec: {
 containers: [{
 name: appName,
 image: "nginx",
 ports: [{ name: "http", containerPort: 80 }]
 }],
 }
 }
 },
}, { provider: cluster.provider });
const service = new k8s.core.v1.Service(`${appName}-svc`, {
 metadata: { labels: appLabels },
 spec: {
 type: "LoadBalancer",
 ports: [{ port: 80, targetPort: "http" }],
 selector: appLabels,
 },
}, { provider: cluster.provider });

// Export the URL for the load balanced service.
export const url = service.status.loadBalancer.ingress[0].hostname;

// Export the cluster's kubeconfig.
export const kubeconfig = cluster.kubeconfig;

Like the ECS example earlier, this spins up 2 load balanced instances of the NGINX web server.

You will find that the entirety of Kubernetes is more feature rich and offers more control over scheduling and scaling your workloads, but with that richness also comes added complexity that will increase training, and time to ramp up, in addition to ongoing maintenance costs. To learn more about Pulumi’s support for Kubernetes, go here.

Private ECR Container Registries

In all of these cases, Amazon Elastic Container Registry (ECR) lets you publish Docker container images to a privately hosted repository inside your AWS account. The benefit of this is that images are stored within your AWS account, where access is governed by IAM and, where they are stored closer to your workloads (maximizing push/pull efficiency). This is an alternative approach to, say, publishing images to the Docker Hub which entails a different authentication and pushing and pulling images over the Internet, where performance can be an issue.

Next Steps

Pulumi Crosswalk for AWS supports three main options for running containers in AWS – ECS Fargate, ECS with EC2 instances, and EKS – each of which integrates deeply with other AWS services like IAM, ELB, and CloudWatch. It’s possible to use any of these services without Pulumi, but there are many benefits to Pulumi’s infrastructure as code.

In short, to get your containers up on AWS, in an easy yet production-ready way, get started for free with Pulumi for AWS today!

After getting started, here are some additional resources to go deeper on specific topics: