Pulumi Blog: Vova Ivanov

Switching the application stack from PERN to MERN

Vova Ivanov — Fri, 18 Sep 2020 00:00:00 +0000

This post is outdated and contains references to a pre-release version of Pulumi Crosswalk (@pulumi/awsx). For updated AWSx documentation and examples, see the AWS Guides.

In this blog post, we return to the PERN application we previously migrated to Kubernetes and replace the PostgreSQL database with MongoDB. Although it might seem like a difficult task initially, the straightforward design of Pulumi and Kubernetes allows us to easily transition the application form a PERN stack to a MERN one.

The main difference between PostgreSQL and MongoDB is data is stored. PostgreSQL is a relational database that allows users to create complex tables and relations between values. In contrast, MongoDB is a key-value database that stores data in simple collections instead of a relational model. Because the application takes the form of three distinct components–the client, server, and database–almost all of the changes will occur in the database component. In fact, we can reuse most of the existing code and will be able to change the database without additional hassle.

The first step to switching the stack is to clone the PERN Kubernetes example. We’ll use a sparse clone to copy only the aws-ts-k8s-voting-app directory.

$ mkdir examples && cd examples
$ git init
$ git remote add origin -f https://github.com/pulumi/examples/
$ git config core.sparseCheckout true
$ echo aws-ts-k8s-voting-app >> .git/info/sparse-checkout
$ git pull origin master

The next step is to create a new directory and initialize a Pulumi project with pulumi new aws-typescript.

$ mkdir aws-ts-k8s-mern-voting-app && cd aws-ts-k8s-mern-voting-app
$ pulumi new aws-typescript

The project will require several configuration variables, which we set using pulumi config set. The variables are used to configure the MongoDB user account, initializing the database schema and table, and its region.

$ pulumi config set sql-user-name <NAME>
$ pulumi config set sql-user-password <PASSWORD> --secret
$ pulumi config set aws:region <REGION>

The package.json file lists the libraries used by the project. We will add the following to the dependencies section:

 "@pulumi/eks": "^0.20.0",
 "@pulumi/cloud-aws": "^0.19.0"

As most of our project components are nearly identical to those in the PERN Kubernetes application, we can copy many of them into our directory.

$ cd ..
$ cp -r aws-ts-k8s-voting-app/clientside/ aws-ts-k8s-mern-voting-app/clientside
$ cp -r aws-ts-k8s-voting-app/serverside/ aws-ts-k8s-mern-voting-app/serverside
$ cp -r aws-ts-k8s-voting-app/databaseside/ aws-ts-k8s-mern-voting-app/databaseside

$ cd aws-ts-k8s-voting-app

Hosting MongoDB will require a rework of the database Docker container and startup scripts. Like before, the outer databaseside folder holds our Dockerfile, while the inner database folder contains code.

FROM ubuntu:18.04

WORKDIR /

EXPOSE 5432

RUN apt update && \
 apt install -y mongodb

ADD database /database

CMD [ "/database/startDatabase.sh" ]

Our setup doesn’t require any files, and so the only thing we need to put in the database folder is a startDatabase.sh script that launches the database, configures users, and creates a table.

#!/bin/bash
set -exu
FILE=/persistentVolume/mongod.lock

if test -f "$FILE"; then
 echo "/persistentVolume already contains MongoDB, no need to initialize database."
 mongod --fork --dbpath /persistentVolume/ --bind_ip_all --logpath /persistentVolume/mongoLogs
else
 echo "/persistentVolume is empty, and we need to initialize the MongoDB database."
 mongod --fork --dbpath /persistentVolume/ --bind_ip_all --logpath /persistentVolume/mongoLogs
 echo "use $DATABASE_NAME
 db.choices.insert({ _id: 0, text: \"Tabs\", vote_count: 0 })
 db.choices.insert({ _id: 1, text: \"Spaces\", vote_count: 0 })
 db.createUser(
 {
 user: \"$USER_NAME\",
 pwd: \"$USER_PASSWORD\",
 roles: [ { role: \"readWrite\", db: \"$DATABASE_NAME\" } ]
 }
 )" | mongo
fi

while true; do
 sleep 3600
done

With the database in place, all that is left is to make some small MongoDB-specific changes to how the client and server components send queries.

The server component will be changed to use a model.js file, which acts as a blueprint for the database’s choices.

const mongoose = require("mongoose");

let Choice = new mongoose.Schema({
 _id: Number,
 text: {
 type: String
 },
 vote_count: {
 type: Number
 },
});

module.exports = mongoose.model("choice", Choice);

Likewise, the db.js file will be modified to create a specialized to our database.

const mongoose = require("mongoose");

const url = "mongodb://"
 + process.env["MONGODB_ADDRESS"]
 + ":" + process.env["MONGODB_PORT"]
 + "/" + process.env["DATABASE_NAME"]

mongoose.connect(url, {
 useNewUrlParser: true,
 user: process.env["USER_NAME"],
 pass: process.env["USER_PASSWORD"]
});

const connection = mongoose.connection;
module.exports = connection;

And finally, the index.js file holding the main code of our server will be modified to send queries to the MongoDB database.

const express = require("express");
const app = express();
const cors = require("cors");
const connection = require("./db");

app.use(cors());
app.use(express.json());

connection.once("open", function() {
 console.log("Connection opened to database!");
})

let Choice = require("./model");

app.get("/voting", async (request, response) => {
 console.log("Get all request");
 Choice.find(function(error, result) {
 if (error) {
 console.log(error);
 } else {
 response.json(result);
 }
 });
});

app.post("/voting/:id", async (request, response) => {
 const { id } = request.params;
 console.log("Casting vote for: " + id);

 Choice.findById(id, function(error, result) {
 if (!result) {
 response.status(404).send("Choice not found");
 } else {
 result.vote_count = result.vote_count + 1;
 result.save().then(result => {
 response.json("Vote successfully cast");
 }).catch(error => {
 response.status(400).send("Failed to cast vote");
 });
 }
 });
});

app.listen(5000, async () => {
 console.log("server has started on port 5000");
});

The VotingComponent.js file in the client will be slightly modified to reference choices by their MongoDB _id parameter.

import React, { Fragment, useEffect, useState } from "react";
const fullServerUrl = "http://" + window.SERVER_URL + ":5000/voting";

const ListChoices = () => {
 const [choices, setChoices] = useState([]);

 const getChoices = async () => {...};

 useEffect(() => {...}, []);

 return (
 <Fragment>
 <h1 className="text-center mt-5">Pulumi Voting App</h1>
 {" "}
 <table className="table mt-5 text-center">
 <thead>
 <tr>
 <th>Choice</th>
 <th>Votes</th>
 </tr>
 </thead>
 <tbody>
 {choices.map(choice => (
 <tr key={choice._id}>
 <td>{choice.text}</td>
 <td>{choice.vote_count}</td>
 <td>
 <button
 className="btn btn-success"
 onClick={() => castVote(choice._id)}
 >
 Vote
 </button>
 </td>
 </tr>
 ))}
 </tbody>
 </table>
 </Fragment>
 );
};

async function castVote(id) {...}

export default ListChoices;

Now that, server, client, and database have been fully converted to use MongoDB, we can make the last remaining change in the infrastructure by updating the database deployment and service.

import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";
import * as eks from "@pulumi/eks";
import * as k8s from "@pulumi/kubernetes";
import * as pulumi from "@pulumi/pulumi";

const config = new pulumi.Config();
const sqlUserName = config.require("sql-user-name");
const sqlUserPassword = config.requireSecret("sql-user-password");
const region = aws.config.region;

const eksCluster = new eks.Cluster(...);

const ebsVolume = new aws.ebs.Volume(...);

const databaseAppName = "database-side-service";
const databaseAppLabels = { appClass: databaseAppName };
const databaseDeployment = new k8s.apps.v1.Deployment(databaseAppName, {
 metadata: { name: databaseAppName, labels: databaseAppLabels },
 spec: {
 replicas: 1,
 selector: { matchLabels: databaseAppLabels },
 template: {
 metadata: { labels: databaseAppLabels },
 spec: {
 containers: [{
 name: databaseAppName,
 image: awsx.ecr.buildAndPushImage("database-side-service", "./databaseside").image(),
 ports: [{ name: "http", containerPort: 27017 }],
 env: [
 { name: "DATABASE_NAME", value: databaseName },
 { name: "USER_NAME", value: sqlUserName },
 { name: "USER_PASSWORD", value: sqlUserPassword },

 ],
 volumeMounts: [{
 name: "persistent-volume",
 mountPath: "/persistentVolume"
 }],
 resources: {
 limits: {
 memory: "1Gi",
 cpu: "1000m"
 }
 }
 }],
 volumes: [{
 name: "persistent-volume",
 awsElasticBlockStore: {
 volumeID: ebsVolume.id,
 },
 }],
 affinity: {
 nodeAffinity: {
 requiredDuringSchedulingIgnoredDuringExecution: {
 nodeSelectorTerms: [{
 matchExpressions: [{
 key: "failure-domain.beta.kubernetes.io/zone",
 operator: "In",
 values: [ebsVolume.availabilityZone],
 }]
 }]
 }
 }
 }
 }
 },
 strategy: {
 type: "Recreate"
 }
 }}, {
 deleteBeforeReplace: true,
 provider: eksCluster.provider,
});

const databasesideListener = new k8s.core.v1.Service("database-side-listener", {
 metadata: { labels: databaseDeployment.metadata.labels },
 spec: {
 type: "ClusterIP",
 ports: [{ port: 27017, targetPort: "http" }],
 selector: databaseAppLabels,
 publishNotReadyAddresses: false,
 }}, {
 provider: eksCluster.provider,
 }
);

const mongodbAddress = databasesideListener.spec.clusterIP;

const serverAppName = "server";
const serverAppLabels = { appClass: serverAppName };
const serverDeployment = new k8s.apps.v1.Deployment("server-side-service", {
 metadata: { labels: serverAppLabels },
 spec: {
 replicas: 2,
 selector: { matchLabels: serverAppLabels },
 template: {
 metadata: { labels: serverAppLabels },
 spec: {
 containers: [{
 name: serverAppName,
 image: awsx.ecr.buildAndPushImage("server-side-service", "./serverside").image(),
 ports: [{ name: "http", containerPort: 5000 }],
 env: [
 { name: "USER_NAME", value: sqlUserName },
 { name: "USER_PASSWORD", value: sqlUserPassword },
 { name: "MONGODB_ADDRESS", value: mongodbAddress },
 { name: "MONGODB_PORT", value: String(27017) },
 { name: "DATABASE_NAME", value: databaseName },
 ],
 resources: {
 limits: {
 memory: "500Mi",
 cpu: "500m"
 }
 }
 }],
 }
 }
 }}, {
 provider: eksCluster.provider,
});

const serversideListener = new k8s.core.v1.Service(...);

const clientAppName = ...;
const clientAppLabels = { ... };
const clientDeployment = new k8s.apps.v1.Deployment(...);

const clientsideListener = new k8s.core.v1.Service(...);

To give us a simple way to connect to our application, we export the clientside listener’s address. We can now open a browser window with the URL and port number to view our application.

export const kubeConfig = eksCluster.kubeconfig;
export const URL = clientsideListener.status.loadBalancer.ingress[0].hostname;

In this example, we showed how to change a PERN stack-based application to use MERN, and demonstrated the ease with which a designer can swap out different components. The isolated nature of containerized applications allows for a great deal of modularity. It can significantly reduce the amount of time spent renovating previously written code to fit any added new components.

The MERN application code is on Github.

Migrating a cloud application to Kubernetes

Vova Ivanov — Mon, 14 Sep 2020 00:00:00 +0000

This post is outdated and contains references to a pre-release version of Pulumi Crosswalk (@pulumi/awsx). For updated AWSx documentation and examples, see the AWS Guides.

In this blog post, we will explore and demonstrate the advantages of Kubernetes by converting and deploying our PERN application to Amazon EKS. With the help of Pulumi, the process becomes greatly simplified and allows us to focus more on the big picture of designing our cloud architecture.

Kubernetes offers several distinct advantages over individually deploying Docker containers to the cloud. Firstly, it’s a robust system for minimizing downtime and rolling out updates. If a pod or node is shut down, a new one is automatically allocated to take its place. Secondly, Kubernetes can scale and distribute your infrastructure. By placing containers into pods, Kubernetes allows you to design the right cloud setup to fit your target effortlessly. And lastly, Kubernetes runs on many cloud providers, letting you choose your organization’s best cloud service provider. Since AWS, Azure, and GCP all support deploying Kubernetes, you have the option of switching if the need arises.

The first step to creating the app is to clone the Typescript PERN example. We’ll use a sparse clone to copy only the aws-ts-pern-voting-app directory.

$ mkdir examples && cd examples
$ git init
$ git remote add origin -f https://github.com/pulumi/examples/
$ git config core.sparseCheckout true
$ echo aws-ts-pern-voting-app >> .git/info/sparse-checkout
$ git pull origin master

The next step is to create a new directory and initialize a Pulumi project with pulumi new aws-typescript.

$ mkdir aws-ts-k8s-voting-app && cd aws-ts-k8s-voting-app
$ pulumi new aws-typescript

The project will require several configuration variables, which we set using pulumi config set. The variables are used to configure the PostgreSQL admin account, a user account for initializing the schema and table, and its region.

$ pulumi config set sql-admin-name <NAME>
$ pulumi config set sql-admin-password <PASSWORD> --secret
$ pulumi config set sql-user-name <NAME>
$ pulumi config set sql-user-password <PASSWORD> --secret
$ pulumi config set aws:region <REGION>

The package.json file lists the libraries used by the project. We will add the following to the dependencies section:

"@pulumi/eks": "^0.20.0",
"@pulumi/cloud-aws": "^0.19.0"

Some of the components of our Kubernetes project are nearly identical to those in the PERN application. We can copy many of these files from the example into our directory.

$ cd ..
$ cp -r aws-ts-pern-voting-app/clientside/ aws-ts-k8s-voting-app/clientside
$ cp -r aws-ts-pern-voting-app/serverside/ aws-ts-k8s-voting-app/serverside
$ cd aws-ts-k8s-voting-app

To prepare our application for Kubernetes, we create a PostgreSQL Docker container for the application. Although this is slightly more complex than directly asking AWS to provision an RDS instance, the underlying mechanics are the same for either approach.

The first step is to create two directories:

$ mkdir databaseside && cd databaseside
$ mkdir database

Like the PERN app, the outer databaseside folder holds our Dockerfile, while the inner database folder contains code.

FROM ubuntu:18.04

WORKDIR /

EXPOSE 5432

RUN apt update && \
 apt install -y postgresql

ADD database /database

CMD [ "/database/startDatabase.sh" ]

Our setup doesn’t require any files, and so the only thing we need to put in the database folder is a startDatabase.sh script that launches the database, configures users, and creates a table.

#!/bin/bash
set -exu
FILE=/persistentVolume/postgresqlDb/postgresql.conf
chown postgres:postgres /persistentVolume

if test -f "$FILE"; then
 echo "/persistentVolume already contains postgresqlDb, no need to initialize database."
else
 echo "/persistentVolume is empty, and we need to initialize the postgresql database."
 cd /persistentVolume
 su postgres -c "/usr/lib/postgresql/10/bin/initdb -D /persistentVolume/postgresqlDb"

 echo "host all all 0.0.0.0/0 md5" >> /persistentVolume/postgresqlDb/pg_hba.conf
 echo "host all all ::/0 md5" >> /persistentVolume/postgresqlDb/pg_hba.conf
 echo "listen_addresses='*'" >> /persistentVolume/postgresqlDb/postgresql.conf

 su postgres -c "/usr/lib/postgresql/10/bin/pg_ctl -D /persistentVolume/postgresqlDb --wait -l logfile start"

 set +x
 echo "psql -U postgres -c \"CREATE ROLE $ADMIN_NAME LOGIN SUPERUSER PASSWORD *********;\""
 psql -U postgres -c "CREATE ROLE $ADMIN_NAME LOGIN SUPERUSER PASSWORD '$ADMIN_PASSWORD';"
 echo "psql -U postgres -c \"CREATE USER $USER_NAME WITH PASSWORD *********;\""
 psql -U postgres -c "CREATE USER $USER_NAME WITH PASSWORD '$USER_PASSWORD';"
 echo "psql -U postgres -c \"ALTER ROLE postgres WITH NOLOGIN;\""
 psql -U postgres -c "ALTER ROLE postgres WITH NOLOGIN;"
 set -x

 psql -U $ADMIN_NAME -d postgres -c "CREATE DATABASE $DATABASE_NAME;"
 psql -U $ADMIN_NAME -d $DATABASE_NAME -c "
 CREATE SCHEMA voting_app;
 CREATE TABLE voting_app.choice(
 choice_id SERIAL PRIMARY KEY,
 text VARCHAR(255) NOT NULL,
 vote_count INTEGER NOT NULL
 );
 GRANT USAGE ON SCHEMA voting_app TO $USER_NAME;
 GRANT SELECT, UPDATE ON ALL TABLES IN SCHEMA voting_app TO $USER_NAME;
 INSERT INTO voting_app.choice (text, vote_count) VALUES('Tabs', 0);
 INSERT INTO voting_app.choice (text, vote_count) VALUES('Spaces', 0);
 "
 su postgres -c "/usr/lib/postgresql/10/bin/pg_ctl -D /persistentVolume/postgresqlDb --wait -l logfile stop"
fi

su postgres -c "/usr/lib/postgresql/10/bin/postgres -D /persistentVolume/postgresqlDb"

Because we don’t have any SSL certificates to authenticate and encrypt internet traffic, the way the server connects to the database needs to be modified to include an ssl: false pool parameter.

const Pool = require("pg").Pool;

const pool = new Pool({
 user: process.env["USER_NAME"],
 password: process.env["USER_PASSWORD"],
 host: process.env["POSTGRES_ADDRESS"],
 port: process.env["POSTGRES_PORT"],
 database: process.env["DATABASE_NAME"],
 ssl: false,
});

module.exports = pool;

With our server, client, and database in place, we can focus on the infrastructure. We start with importing libraries and describing the application’s configuration options.

import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";
import * as eks from "@pulumi/eks";
import * as k8s from "@pulumi/kubernetes";
import * as pulumi from "@pulumi/pulumi";

const config = new pulumi.Config();
const sqlAdminName = config.require("sql-admin-name");
const sqlAdminPassword = config.requireSecret("sql-admin-password");
const sqlUserName = config.require("sql-user-name");
const sqlUserPassword = config.requireSecret("sql-user-password");
const awsConfig = new pulumi.Config("aws");
const region = aws.config.region;

The central part of our application is an AWS Elastic Kubernetes Cluster. It holds our application and dictates how many nodes should be allocated for the processes running inside the cluster.

const eksCluster = new eks.Cluster("eksCluster", {
 name: "eksCluster",
 instanceType: "t2.medium",
 desiredCapacity: 3,
 minSize: 2,
 maxSize: 4,
 deployDashboard: false,
 enabledClusterLogTypes: [
 "api",
 "audit",
 "authenticator",
 ],
});

Kubernetes nodes use ephemeral storage. When Pods shut down or get restarted, everything is erased. To create a permanent storage system suitable for our database, we need a Persistent Volume instance.

const ebsVolume = new aws.ebs.Volume("storage-volume", {
 size: 1,
 type: "gp2",
 availabilityZone: region + "a",
 encrypted: true,
 }, {
 protect: false,
});

Now that the infrastructure for our database is ready, we can launch a PostgreSQL server on the cloud in the form of a Deployment. Deployments offer a multitude of customization options ranging from attaching volumes, setting the memory and cpu requirements, to specifying which deployments can and cannot be run on the same node.

const databaseAppName = "database-side-service";
const databaseAppLabels = { appClass: databaseAppName };
const databaseDeployment = new k8s.apps.v1.Deployment(databaseAppName, {
 metadata: { name: databaseAppName, labels: databaseAppLabels },
 spec: {
 replicas: 1,
 selector: { matchLabels: databaseAppLabels },
 template: {
 metadata: { labels: databaseAppLabels },
 spec: {
 containers: [{
 name: databaseAppName,
 image: awsx.ecr.buildAndPushImage("database-side-service", "./databaseside").image(),
 ports: [{ name: "http", containerPort: 5432 }],
 env: [
 { name: "ADMIN_NAME", value: sqlAdminName },
 { name: "ADMIN_PASSWORD", value: sqlAdminPassword },
 ],
 volumeMounts: [{
 name: "persistent-volume",
 mountPath: "/persistentVolume"
 }],
 resources: {
 limits: {
 memory: "1Gi",
 cpu: "1000m"
 }
 }
 }],
 volumes: [{
 name: "persistent-volume",
 awsElasticBlockStore: {
 volumeID: ebsVolume.id,
 },
 }],
 affinity: {
 nodeAffinity: {
 requiredDuringSchedulingIgnoredDuringExecution: {
 nodeSelectorTerms: [{
 matchExpressions: [{
 key: "failure-domain.beta.kubernetes.io/zone",
 operator: "In",
 values: [ebsVolume.availabilityZone],
 }]
 }]
 }
 }
 }
 }
 },
 strategy: {
 type: "Recreate"
 }
 }}, {
 deleteBeforeReplace: true,
 provider: eksCluster.provider,
});

In order to have a a way for the pods in cluster to communicate with our database, we create a Kubernetes service for it. It is best practice to keep databases and other similar components entirely within the cluster, and not allow them to be accessed directly the internet.

const databasesideListener = new k8s.core.v1.Service("database-side-listener", {
 metadata: { labels: databaseDeployment.metadata.labels },
 spec: {
 type: "ClusterIP",
 ports: [{ port: 5432, targetPort: "http" }],
 selector: databaseAppLabels,
 publishNotReadyAddresses: false,
 }}, {
 provider: eksCluster.provider,
 }
);

const postgresqlAddress = databasesideListener.spec.clusterIP;
});

Now, all that remains is deploying the client and server containers to Kubernetes. Since the PERN application already deployed them as docker containers, it doesn’t take much effort to reconfigure it for Kubernetes. Unlike the database, these components can safely be open to the Internet using a service with a Load Balancer type.

We’ll first set up the server deployment:

const serverAppName = "server";
const serverAppLabels = { appClass: serverAppName };
const serverDeployment = new k8s.apps.v1.Deployment("server-side-service", {
 metadata: { labels: serverAppLabels },
 spec: {
 replicas: 2,
 selector: { matchLabels: serverAppLabels },
 template: {
 metadata: { labels: serverAppLabels },
 spec: {
 containers: [{
 name: serverAppName,
 image: awsx.ecr.buildAndPushImage("server-side-service", "./serverside").image(),
 ports: [{ name: "http", containerPort: 5000 }],
 env: [
 { name: "USER_NAME", value: sqlUserName },
 { name: "USER_PASSWORD", value: sqlUserPassword },
 { name: "POSTGRES_ADDRESS", value: postgresqlAddress },
 { name: "POSTGRES_PORT", value: String(5432) },
 { name: "DATABASE_NAME", value: postgresDatabase.name },
 ],
 resources: {
 limits: {
 memory: "500Mi",
 cpu: "500m"
 }
 }
 }],
 }
 }
 }}, {
 provider: eksCluster.provider,
});

const serversideListener = new k8s.core.v1.Service("server-side-listener", {
 metadata: { labels: serverDeployment.metadata.labels },
 spec: {
 type: "LoadBalancer",
 ports: [{ port: 5000, targetPort: "http" }],
 selector: serverAppLabels,
 publishNotReadyAddresses: false,
 }}, {
 provider: eksCluster.provider,
 }
);

And lastly, the client deployment.

const clientAppName = "client";
const clientAppLabels = { appClass: clientAppName };
const clientDeployment = new k8s.apps.v1.Deployment("client-side-service", {
 metadata: { labels: clientAppLabels },
 spec: {
 replicas: 2,
 selector: { matchLabels: clientAppLabels },
 template: {
 metadata: { labels: clientAppLabels },
 spec: {
 containers: [{
 name: clientAppName,
 image: awsx.ecr.buildAndPushImage("client-side-service", "./clientside").image(),
 ports: [{ name: "http", containerPort: 3000 }],
 env: [
 { name: "SERVER_HOSTNAME", value: serversideListener.status.loadBalancer.ingress[0].hostname },
 ],
 resources: {
 limits: {
 memory: "500Mi",
 cpu: "500m"
 }
 },
 }],
 }
 }
 }}, {
 provider: eksCluster.provider,
});

const clientsideListener = new k8s.core.v1.Service("client-side-listener", {
 metadata: { labels: clientDeployment.metadata.labels },
 spec: {
 type: "LoadBalancer",
 ports: [{ port: 3000, targetPort: "http" }],
 selector: clientAppLabels,
 publishNotReadyAddresses: false,
 }}, {
 provider: eksCluster.provider,
 }
);

To give us a simple way to connect to our application, we export the clientside listener’s address. We can now open a browser window with the URL and port number to view our application.

export const kubeConfig = eksCluster.kubeconfig;
export const URL = clientsideListener.status.loadBalancer.ingress[0].hostname;

In this example, we demonstrated the benefits Kubernetes offers and showed how to use Pulumi to deploy an application to Kubernetes hosted on AWS. Although it might seem unnecessary to use such a system for only a single voting app, the advantages of running your project in the form of containerized services become apparent as your application grows in size.

Next week, we’ll explore applications using the MERN stack: MongoDB, Express, React, and NodeJS.

The blog post’s code can be found on Github.

Deploying a PERN stack application to AWS

Vova Ivanov — Fri, 04 Sep 2020 00:00:00 +0000

Some of the code in this post is out of date. See the AWS guides for an updated overview and examples.

In this blog post, we will explore PERN stack applications and deploy one to AWS. PERN is an acronym for PostgreSQL, Express, React, and Node. A PERN stack application is a project that uses PostgreSQL, Express as an application framework, React as a user interface framework, and runs on Node. We will also use Pulumi Crosswalk to reduce the amount of code and provide a quick and straightforward path for deploying the application.

PERN projects have four distinct tiers:

a database that keeps track of our data,
a stateless server that receives commands to manage the database,
a clientside server that contains and renders the user interface code,
and the users who run the code in their browsers and send requests to the stateless server.

An advantage of infrastructure as code is that the application and infrastructure deployment can use the same language. As React and the other components use NodeJS, we’ll use it for our infrastructure by writing it in TypeScript. The first step is to create a new directory and initialize a Pulumi project with pulumi new aws-typescript.

$ mkdir aws-ts-pern-voting-app && cd aws-ts-pern-voting-app
$ pulumi new aws-typescript

This tutorial was written for the aws-ts-pern-voting-app example but will work with any other PERN stack application. The example uses two folders to hold the client and server tiers and a Dockerfile that builds images that run as containers in AWS.

To deploy our PERN stack, the project requires several configuration variables, which we set using pulumi config set. The variables are used to configure the PostgreSQL admin account, a user account for initializing the schema and table, and the database’s region.

$ pulumi config set sql-admin-name <NAME>
$ pulumi config set sql-admin-password <PASSWORD> --secret
$ pulumi config set sql-user-name <NAME>
$ pulumi config set sql-user-password <PASSWORD> --secret
$ pulumi config set aws:region <REGION>

The package.json file lists the libraries used by the project. We will add the following to the dependencies section:

"@pulumi/cloud-aws": "^0.19.0",
"@pulumi/postgresql": "^2.3.0",
"pg": "^8.3.3"

Our project uses a Dynamic Provider to help create tables and Schemas. It offers the same features as our MySQL provider, but for PostgreSQL.

import * as pulumi from "@pulumi/pulumi";
import * as crypto from "crypto";

export interface SchemaInputs {
 creatorName: pulumi.Input<string>
 creatorPassword: pulumi.Input<string>
 serverAddress: pulumi.Input<string>
 databaseName: pulumi.Input<string>
 creationScript: pulumi.Input<string>
 deletionScript: pulumi.Input<string>
 postgresUserName: pulumi.Input<string>
}

export class SchemaProvider implements pulumi.dynamic.ResourceProvider {
 async create(args: SchemaInputs): Promise<pulumi.dynamic.CreateResult> {
 const { Pool } = require("pg");
 const pool = new Pool({
 user: args.creatorName,
 password: args.creatorPassword,
 host: args.serverAddress,
 port: 2000,
 database: args.databaseName
 });
 const scriptExecuted = await pool.query(args.creationScript);

 await pool.end();
 return {id: "postgresqlSchema-" + crypto.randomBytes(16).toString('hex'), outs: args};
 }

 async delete(id: string, args: SchemaInputs): Promise<void> {
 const { Pool } = require("pg");
 const pool = new Pool({
 user: args.creatorName,
 password: args.creatorPassword,
 host: args.serverAddress,
 port: 2000,
 database: args.databaseName
 });
 const scriptExecuted = await pool.query(args.deletionScript);
 await pool.end();
 }

 async diff(id: string, oldArgs: SchemaInputs, newArgs: SchemaInputs): Promise<pulumi.dynamic.DiffResult> {
 let changes: boolean = ((oldArgs.creatorName != newArgs.creatorName) ||
 (oldArgs.creatorPassword != newArgs.creatorPassword) || (oldArgs.serverAddress != newArgs.serverAddress) ||
 (oldArgs.databaseName != newArgs.databaseName) || (oldArgs.creationScript != newArgs.creationScript) ||
 (oldArgs.deletionScript != newArgs.deletionScript));

 let replaces: string[] = [];
 if (oldArgs.serverAddress != newArgs.serverAddress) { replaces.push("serverAddress") };
 if (oldArgs.databaseName != newArgs.databaseName) { replaces.push("databaseName") };
 if (oldArgs.creationScript != newArgs.creationScript) { replaces.push("creationScript") };

 return {
 changes: changes,
 replaces: replaces,
 stables: [],
 deleteBeforeReplace: true
 }
 }

 async update(id: string, oldArgs: SchemaInputs, newArgs: SchemaInputs): Promise<pulumi.dynamic.UpdateResult> {
 return { outs: newArgs };
 }
}

export class Schema extends pulumi.dynamic.Resource {
 public readonly creatorName!: pulumi.Output<string>;
 public readonly creatorPassword!: pulumi.Output<string>;
 public readonly serverAddress!: pulumi.Output<string>;
 public readonly databaseName!: pulumi.Output<string>;
 public readonly creationScript!: pulumi.Output<string>;
 public readonly deletionScript!: pulumi.Output<string>;
 constructor(name: string, args: SchemaInputs) {
 super(new SchemaProvider, name, args);
 }
}

With the Dynamic provider configured, we can focus on the main index.ts file. We start with importing libraries and describing the application’s configuration options.

import * as aws from "@pulumi/aws";
import * as awsx from "@pulumi/awsx";
import * as pulumi from "@pulumi/pulumi";
import * as postgresql from "@pulumi/postgresql";
import { Schema } from "./PostgreSqlDynamicProvider";
import { table } from "console";

const config = new pulumi.Config();
const sql_admin_name = config.require("sql-admin-name");
const sql_admin_password = config.requireSecret("sql-admin-password");
const sql_user_name = config.require("sql-user-name");
const sql_user_password = config.requireSecret("sql-user-password");
const awsConfig = new pulumi.Config("aws");
const availabilityZone = awsConfig.get("region");

Apart from language differences, the code for creating the VPC, subnets, and tables is almost identical to our previous projects. Since we will be using Crosswalk, we can leave out a few components that will be automatically created later.

const appVpc = new aws.ec2.Vpc("app-vpc", {
 cidrBlock: "172.31.0.0/16",
 enableDnsHostnames: true
});

const appGateway = new aws.ec2.InternetGateway("app-gateway", {
 vpcId: appVpc.id
});

const appRoutetable = new aws.ec2.RouteTable("app-routetable", {
 routes: [
 {
 cidrBlock: "0.0.0.0/0",
 gatewayId: appGateway.id,
 }
 ],
 vpcId: appVpc.id
});

const appRoutetableAssociation = new aws.ec2.MainRouteTableAssociation("app-routetable-association", {
 routeTableId: appRoutetable.id,
 vpcId: appVpc.id
});

const rdsSecurityGroup = new aws.ec2.SecurityGroup("rds-security-group", {
 vpcId: appVpc.id,
 description: "Enables HTTP access",
 ingress: [{
 protocol: 'tcp',
 fromPort: 0,
 toPort: 65535,
 cidrBlocks: ['0.0.0.0/0'],
 }],
 egress: [{
 protocol: '-1',
 fromPort: 0,
 toPort: 0,
 cidrBlocks: ['0.0.0.0/0'],
 }]
});

const firstRdsSubnet = new aws.ec2.Subnet("first-rds-subnet", {
 vpcId: appVpc.id,
 cidrBlock: "172.31.0.0/20",
 availabilityZone: availabilityZone + "a"
});

const secondRdsSubnet = new aws.ec2.Subnet("second-rds-subnet", {
 vpcId: appVpc.id,
 cidrBlock: "172.31.128.0/20",
 availabilityZone: availabilityZone + "b"
});

We create a SubnetGroup and an RDS instance. The latter uses PostgreSQL instead of MySQL, and is assigned 2000.

const rdsSubnetGroup = new aws.rds.SubnetGroup("rds-subnet-group", {
 subnetIds: [firstRdsSubnet.id, secondRdsSubnet.id]
});

const postgresqlRdsServer = new aws.rds.Instance("postgresql-rds-server", {
 engine: "postgres",
 username: sql_admin_name,
 password: sql_admin_password,
 instanceClass: "db.t2.micro",
 allocatedStorage: 20,
 skipFinalSnapshot: true,
 publiclyAccessible: true,
 port: 2000,
 dbSubnetGroupName: rdsSubnetGroup.name,
 vpcSecurityGroupIds: [rdsSecurityGroup.id]
});

Pulumi offers additional tools to make handling PostgreSQL easier.

const postgresqlProvider = new postgresql.Provider("postgresql-provider", {
 host: postgresqlRdsServer.address,
 port: postgresqlRdsServer.port,
 username: sql_admin_name,
 password: sql_admin_password,
 superuser: false
});

We initialize the example database and create a user to manage it.

const postgresDatabase = new postgresql.Database("postgresql-database", {
 name: "votes"}, {
 provider: postgresqlProvider
});

const postgresUser = new postgresql.Role("postgres-standard-role", {
 name: sql_user_name,
 password: sql_user_password,
 superuser: false,
 login: true,
 connectionLimit: -1}, {
 provider: postgresqlProvider
});

We use the Dynamic provider to create a schema and a table, grant permissions for our user to edit and select it, and populate the table with two initial voting options.

const creation_script = `
 CREATE SCHEMA voting_app;
 CREATE TABLE voting_app.choice(
 choice_id SERIAL PRIMARY KEY,
 text VARCHAR(255) NOT NULL,
 vote_count INTEGER NOT NULL
 );
 GRANT USAGE ON SCHEMA voting_app TO ${sql_user_name};
 GRANT SELECT, UPDATE ON ALL TABLES IN SCHEMA voting_app TO ${sql_user_name};
 INSERT INTO voting_app.choice (text, vote_count) VALUES('Tabs', 0);
 INSERT INTO voting_app.choice (text, vote_count) VALUES('Spaces', 0);
`;

const deletion_script = "DROP SCHEMA IF EXISTS voting_app CASCADE";

const postgresqlVotesTable = new Schema("postgresql-votes-schema", {
 creatorName: sql_admin_name,
 creatorPassword: sql_admin_password,
 serverAddress: postgresqlRdsServer.address,
 databaseName: postgresDatabase.name,
 creationScript: creation_script,
 deletionScript: deletion_script,
 postgresUserName: postgresUser.name
});

With the basic infrastructure and provider completed, we can write the application deployment code to ECS. We will use Pulumi Crosswalk, a collection of libraries that makes common infrastructure-as-code tasks in AWS easier and more secure by automatically using well-architected best practices.

We’ll first set up the server. The Network Listener is assigned the same port as the server, which is, in our case, port 5000. A set of environment variables representing our PostgreSQL connection credentials are passed directly to the awsx.ecs.FargateService. With AWS Crosswalk, what would have been over 150 lines of code is reduced to just under 20 lines of code.

const serversideListener = new awsx.elasticloadbalancingv2.NetworkListener("server-side-listener", { port: 5000 });
const serversideService = new awsx.ecs.FargateService("server-side-service", {
 taskDefinitionArgs: {
 containers: {
 serversideService: {
 image: awsx.ecs.Image.fromPath("server-side-service", "./serverside"),
 memory: 512,
 portMappings: [serversideListener],
 environment: [
 { name: "USER_NAME", value: sql_user_name },
 { name: "USER_PASSWORD", value: sql_user_password },
 { name: "RDS_ADDRESS", value: postgresqlRdsServer.address },
 { name: "RDS_PORT", value: String(2000) },
 { name: "DATABASE_NAME", value: postgresDatabase.name },
 ],
 },
 },
 },
});

The same is true for the client service, which is reduced to a short and easy to understand format. By default, React uses port 3000, but it can set to a different port. The SERVER_HOSTNAME environment variable is passed in when the container starts to generate a tiny configuration file at runtime called serverParams.js with the URL. This way, we do not have to rebuild the entire docker image should the server URL change.

const clientsideListener = new awsx.elasticloadbalancingv2.NetworkListener("client-side-listener", { port: 3000 });
const clientsideService = new awsx.ecs.FargateService("client-side-service", {
 taskDefinitionArgs: {
 containers: {
 clientsideService: {
 image: awsx.ecs.Image.fromPath("client-side-service", "./clientside"),
 memory: 512,
 portMappings: [clientsideListener],
 environment: [
 { name: "SERVER_HOSTNAME", value: serversideListener.endpoint.hostname },
 ],
 },
 },
 },
});

To make our PERN stack application available on the Internet, we export the clientside listener’s address. We can open a browser window with the URL and port to view our application.

export const URL = clientsideListener.endpoint.hostname;

In this example, I explained the basic principles behind PERN stack applications and showed how to create the infrastructure to deploy them on ECS. A flexible and optimized tool, Pulumi Crosswalk supports rapidly prototyping applications, scaling workloads, securing and integrating applications in existing infrastructure, and going to production in multiple complex environments.

Next week, I’ll demonstrate how to integrate applications with Kubernetes, and seamlessly deploy them to EKS using Pulumi.

The blog post’s code can be found on Github.

How to Deploy a Django Application to AWS

Vova Ivanov — Fri, 28 Aug 2020 00:00:00 +0000

In this blog post, we will finish swapping out the frontend and backend of our Python AWS application. Although Flask and Redis are different from Django and MySQL in many ways, the underlying infrastructure behind their deployment is nonetheless very similar, and can be effortlessly updated as we transition from one to the other.

We will be paying additional attention to security, and will be making use of Pulumi’s secret management tools to protect our passwords and private keys.

The first step is to create a new directory and initialize a Pulumi project with pulumi new aws-python.

$ mkdir aws-py-django-voting-app && cd aws-py-django-voting-app
$ pulumi new aws-python

Next, let’s create a folder to hold our application and start a Django project in it.

$ mkdir frontend && cd frontend
$ django-admin startproject mysite
$ cd mysite

This tutorial was written for the aws-py-django-voting-app example but will work with any other Django application. The most important file is ./mysite/settings.py, which we will modify to accept secrets and configuration parameters in the form of environment variables. A common mistake that programmers make is to submit files with important data to source code repositories. Even if it is a private repository, it is still not recommended to leave passwords and private keys in your files.

SECRET_KEY = os.environ['SECRET_KEY']

DATABASES = {
 'default': {
 'ENGINE': 'django.db.backends.mysql',
 'NAME': os.environ['DATABASE_NAME'],
 'USER': os.environ['USER_NAME'],
 'PASSWORD': os.environ['USER_PASSWORD'],
 'HOST': os.environ['DATABASE_ADDRESS'],
 'PORT': os.environ['DATABASE_PORT'],
 }
}

To perform database migrations as part of the deployment, we will create a setupDatabase.sh script inside the aws-py-django-voting-app/frontend/mysite folder. As a bonus, the script will create an admin account that we can use to log into our website. Make sure the file has exec permissions.

#!/bin/bash
set -exu
python3 /mysite/manage.py makemigrations
python3 /mysite/manage.py migrate
export DJANGO_SUPERUSER_PASSWORD=$DJANGO_PASSWORD
python3 /mysite/manage.py createsuperuser \
 --no-input \
 --username=$DJANGO_NAME \
 --email=$DJANGO_NAME@example.com

The next step is to go back into the aws-py-django-voting-app/frontend folder and containerize our application with Docker. First, we will list the libraries our application uses by creating a requirements.txt file with the following lines.

django==3.1
mysqlclient==1.4.0

And lastly, we can create the Dockerfile.

FROM ubuntu:18.04

WORKDIR /

EXPOSE 80

RUN apt-get update && \
 apt install -y gcc python3-dev python3-pip mysql-client-core-5.7 libmysqlclient-dev

ADD requirements.txt /

RUN pip3 install -r requirements.txt

ADD mysite /mysite

CMD [ "python3", "/mysite/manage.py", "runserver", "0.0.0.0:80" ]

Now that our Django application and Dockerfile are ready, we can return to the main aws-py-django-voting-app folder. The Pulumi project requires several configuration variables, which we set using pulumi config set. They are used to configure the MySQL admin account, a user account for initializing the table, and the Django website admin account. The private key that Django uses will also be passed in the same way.

$ pulumi config set sql-admin-name <NAME>
$ pulumi config set sql-admin-password <PASSWORD> --secret
$ pulumi config set sql-user-name <NAME>
$ pulumi config set sql-user-password <PASSWORD> --secret
$ pulumi config set django-admin-name <NAME>
$ pulumi config set django-admin-password <PASSWORD> --secret
$ pulumi config set django_secret_key = <VALUE> --secret

The requirements.txt file lists the libraries used by the project. We will need to add the following:

pulumi-docker>=2.0.0,<3.0.0
pulumi-mysql>=2.0.0,<3.0.0
mysql-connector-python>=1.0.0,<10.0.0

The first few lines of our __main__.py file indicate the libraries to import and describe the application’s configuration options.

import json
import base64
import pulumi
import pulumi_aws as aws
import pulumi_docker as docker
import pulumi_mysql as mysql

config = pulumi.Config()
sql_admin_name = config.require("sql-admin-name")
sql_admin_password = config.require_secret("sql-admin-password")
sql_user_name = config.require("sql-user-name")
sql_user_password = config.require_secret("sql-user-password")
availability_zone = pulumi.Config("aws").get("region")

django_admin_name = config.require("django-admin-name")
django_admin_password = config.require_secret("django-admin-password")

django_secret_key = config.require_secret("django-secret-key")

After setting up the imports and configurations, we create an ECS Cluster, VPC, Subnet, RDS instance, and several other items. All the components are identical to the ones in the first and second blog posts, and the full code can be seen in this example’s github repository.

app_cluster = aws.ecs.Cluster(...)

app_vpc = aws.ec2.Vpc(...)
app_vpc_subnet = aws.ec2.Subnet(...)

app_gateway = aws.ec2.InternetGateway(...)
app_routetable = aws.ec2.RouteTable(...)
app_security_group = aws.ec2.SecurityGroup(...)

app_exec_role = aws.iam.Role(...)
exec_policy_attachment = aws.iam.RolePolicyAttachment(...)
app_task_role = aws.iam.Role(...)
task_policy_attachment = aws.iam.RolePolicyAttachment(...)
task_policy_attachment = aws.iam.RolePolicyAttachment(...)

app_ecr_repo = aws.ecr.Repository(...)
app_lifecycle_policy = aws.ecr.LifecyclePolicy(...)

extra_rds_subnet = aws.ec2.Subnet(...)
app_database_subnetgroup = aws.rds.SubnetGroup(...)
mysql_rds_server = aws.rds.Instance(...)
mysql_provider = mysql.Provider(...)
mysql_database = mysql.Database(...)
mysql_user = mysql.User(...)
mysql_access_grant = mysql.Grant(...)

With the basic infrastructure and MySQL backend in place, all that’s left is to create the Django frontend.

A target group, balancer, and listener are created for the frontend.

django_targetgroup = aws.lb.TargetGroup("django-targetgroup",
 port=80,
 protocol="TCP",
 target_type="ip",
 stickiness= {
 "enabled": False,
 "type": "lb_cookie",
 },
 vpc_id=app_vpc.id)

django_balancer = aws.lb.LoadBalancer("django-balancer",
 load_balancer_type="network",
 internal=False,
 security_groups=[],
 subnets=[app_vpc_subnet.id])

django_listener = aws.lb.Listener("django-listener",
 load_balancer_arn=django_balancer.arn,
 port=80,
 protocol="TCP",
 default_actions=[{
 "type": "forward",
 "target_group_arn": django_targetgroup.arn
 }])

The application is built into a Docker image and pushed to our ECR repository we created earlier.

def get_registry_info(rid):
 creds = aws.ecr.get_credentials(registry_id=rid)
 decoded = base64.b64decode(creds.authorization_token).decode()
 parts = decoded.split(':')
 if len(parts) != 2:
 raise Exception("Invalid credentials")
 return docker.ImageRegistry(creds.proxy_endpoint, parts[0], parts[1])

app_registry = app_ecr_repo.registry_id.apply(get_registry_info)

django_image = docker.Image("django-dockerimage",
 image_name=app_ecr_repo.repository_url,
 build="./frontend",
 skip_push=False,
 registry=app_registry
)

To help with debugging our application, we will create a Cloudwatch instance to store all logs.

django_log_group = aws.cloudwatch.LogGroup("django-log-group",
 retention_in_days=1,
 name="django-log-group"
)

Our project is special because it uses two independent ECS services—one that sets up the MySQL database, and one that continuously runs and handles the website.

While it is possible to have a single ECS service set up both the database and the server, doing so presents the risk of exposing important information to the internet. To set up the database, the ECS service would need to know our MySQL admin credentials. However, since the service would also handle the website, it would mean that a successful attack on our Django website could allow the attacker to access the credentials for our backend.

First, let’s create the task definition that runs once and sets up the database.

django_database_task_definition = aws.ecs.TaskDefinition("django-database-task-definition",
 family="django_database_task_definition-family",
 cpu="256",
 memory="512",
 network_mode="awsvpc",
 requires_compatibilities=["FARGATE"],
 execution_role_arn=app_exec_role.arn,
 task_role_arn=app_task_role.arn,
 container_definitions=pulumi.Output.all(
 # We must do Output.all() to access Pulumi secrets
 django_image.image_name,
 django_secret_key,
 mysql_database.name,
 sql_admin_name,
 sql_admin_password,
 django_admin_name,
 django_admin_password,
 mysql_rds_server.address,
 mysql_rds_server.port).apply(lambda args: json.dumps([{
 "name": "django-container",
 "image": args[0],
 "memory": 512,
 "essential": True,
 "portMappings": [{
 "containerPort": 80,
 "hostPort": 80,
 "protocol": "tcp"
 }],
 "environment": [
 { "name": "SECRET_KEY", "value": args[1] },
 { "name": "DATABASE_NAME", "value": args[2] },
 { "name": "USER_NAME", "value": args[3] },
 { "name": "USER_PASSWORD", "value": args[4] },
 { "name": "DJANGO_NAME", "value": args[5] },
 { "name": "DJANGO_PASSWORD", "value": args[6] },
 { "name": "DATABASE_ADDRESS", "value": args[7] },
 { "name": "DATABASE_PORT", "value": str(int(args[8])) },
 ],
 "logConfiguration": {
 "logDriver": "awslogs",
 "options": {
 "awslogs-group": "django-log-group",
 "awslogs-region": "us-west-2",
 "awslogs-stream-prefix": "djangoApp-database",
 },
 },
 # We override the command in the Dockerfile with a new one
 "command": ["/mysite/setupDatabase.sh"]
 }])))

We can now launch our first service on Fargate.

django_database_service = aws.ecs.Service("django-database-service",
 cluster=app_cluster.arn,
 desired_count=1,
 launch_type="FARGATE",
 task_definition=django_database_task_definition.arn,
 wait_for_steady_state=False,
 network_configuration={
 "assign_public_ip": "true",
 "subnets": [app_vpc_subnet.id],
 "security_groups": [app_security_group.id]
 },
 load_balancers=[{
 "target_group_arn": django_targetgroup.arn,
 "container_name": "django-container",
 "container_port": 80,
 }],
 opts=pulumi.ResourceOptions(depends_on=[django_listener]),
)

Finally, let’s create the second task definition to run the website.

django_site_task_definition = aws.ecs.TaskDefinition("django-site-task-definition",
 family="django-site-task-definition-family",
 cpu="256",
 memory="512",
 network_mode="awsvpc",
 requires_compatibilities=["FARGATE"],
 execution_role_arn=app_exec_role.arn,
 task_role_arn=app_task_role.arn,
 container_definitions=pulumi.Output.all(
 django_image.image_name,
 django_secret_key,
 mysql_database.name,
 sql_user_name,
 sql_user_password,
 mysql_rds_server.address,
 mysql_rds_server.port).apply(lambda args: json.dumps([{
 "name": "django-container",
 "image": args[0],
 "memory": 512,
 "essential": True,
 "portMappings": [{
 "containerPort": 80,
 "hostPort": 80,
 "protocol": "tcp"
 }],
 "environment": [
 { "name": "SECRET_KEY", "value": args[1] },
 { "name": "DATABASE_NAME", "value": args[2] },
 { "name": "USER_NAME", "value": args[3] },
 { "name": "USER_PASSWORD", "value": args[4] },
 { "name": "DATABASE_ADDRESS", "value": args[5] },
 { "name": "DATABASE_PORT", "value": str(int(args[6])) },
 ],
 "logConfiguration": {
 "logDriver": "awslogs",
 "options": {
 "awslogs-group": "django-log-group",
 "awslogs-region": "us-west-2",
 "awslogs-stream-prefix": "djangoApp-site",
 },
 },
 }])))

Similarly, we will launch our second task definition on Fargate.

django_site_service = aws.ecs.Service("django-site-service",
 cluster=app_cluster.arn,
 desired_count=1,
 launch_type="FARGATE",
 task_definition=django_site_task_definition.arn,
 wait_for_steady_state=False,
 network_configuration={
 "assign_public_ip": "true",
 "subnets": [app_vpc_subnet.id],
 "security_groups": [app_security_group.id]
 },
 load_balancers=[{
 "target_group_arn": django_targetgroup.arn,
 "container_name": "django-container",
 "container_port": 80,
 }],
 opts=pulumi.ResourceOptions(depends_on=[django_listener]),

To connect to our application, we export the DNS name of the Django balancer and open it in a browser window.

pulumi.export("app-url", django_balancer.dns_name)

To add questions to the database, add “/admin/” to the end of the URL, and log in using your Django admin credentials.

The Administration panel allows you to manage the state of your application. Groups, users, choices, and questions can all be added easily.

Comparison between Flask and Django:

In this example, I described how to set up a basic Django voting application and deploy it to AWS. Although our Django and Flask applications use very different AWS services, we seamlessly replaced the previous infrastructure with what was needed.

Next week, we’ll explore PostgreSQL, Express, React, and Node.js, and use them to create a simple application.

The blog post’s full code and an in-depth explanation for each component are on Github.

Deploying a MySQL schema using Dynamic Providers

Vova Ivanov — Tue, 18 Aug 2020 00:00:00 +0000

In our previous post, we created a Python voting application using Flask and Redis. This blog post will explore creating a MySQL database and initializing it with a schema and data. What seems to be a simple step is much more interesting than it appears, because Pulumi’s MySQL provider does not support creating and populating tables. To do it, we will extend it with a Dynamic Provider.

The existing Pulumi MySQL provider allows us to create the MySQL server, database, and different users. However, it would be great to create the tables as part of the deployment, too, since database schema can be considered a type of infrastructure. Because creating tables requires admin credentials, deploying them along with the other infrastructure allows us to limit the sharing of admin credentials, and run the application under restricted permissions at all times.

A great advantage of Pulumi is its extensible and modular design. If support for something isn’t implemented, you can write it yourself easily. We will be writing a Dynamic Provider that connects to a MySQL server, initializes a table, and creates some starting data all during pulumi up.

The first step is to create a new directory and initialize a Pulumi project with pulumi new aws-python.

$ mkdir aws-py-dynamicresource && cd aws-py-dynamicresource
$ pulumi new aws-python

This project requires several configuration variables, which we set using pulumi config set. They are used to configure the MySQL admin account during deployment, and a user account for initializing the table.

$ pulumi config set sql-admin-name <NAME>
$ pulumi config set sql-admin-password <PASSWORD> --secret
$ pulumi config set sql-user-name <NAME>
$ pulumi config set sql-user-password <PASSWORD> --secret

The requirements.txt file lists the libraries used in the project. We will need to add the following:

pulumi-mysql>=1.0.0,<3.0.0
mysql-connector-python>=1.0.0,<10.0.0

We will write the code for our Dynamic Provider in a separate file to reduce clutter, for example, MySqlDynamicProvider.py. The first few lines of the file will indicate which libraries to import.

import mysql.connector as connector
from mysql.connector import errorcode
from pulumi import Input, Output, ResourceOptions
from pulumi.dynamic import *
from typing import Any, Optional
import binascii
import os

After setting up the imports, the next step is to write the code. The Dynamic Provider’s first component is a class with the arguments that the dynamic provider requires when created. These arguments are given a type Input[str] and automatically converted to regular str before being passed to the provider’s functions.

class SchemaInputs(object):
 creator_name: Input[str]
 creator_password: Input[str]
 server_address: Input[str]
 database_name: Input[str]
 creation_script: Input[str]
 deletion_script: Input[str]
 def __init__(self, creator_name, creator_password, server_address, database_name, creation_script, deletion_script):
 self.creator_name = creator_name
 self.creator_password = creator_password
 self.server_address = server_address
 self.database_name = database_name
 self.creation_script = creation_script
 self.deletion_script = deletion_script

The second step is to write the Dynamic Provider. The provider handles the create, read, update, and delete operations the resource needs. The create function instantiates a new resource and assigns it a unique ID. The delete function deletes an existing resource. The diff function determines if we can update the resource without entirely replacing it, and the update function performs the update.

class SchemaProvider(ResourceProvider):
 def create(self, args):
 connection = connector.connect(user=args["creator_name"],
 password=args["creator_password"],
 host=args["server_address"],
 database=args["database_name"])
 cursor = connection.cursor()
 cursor.execute(args["creation_script"])
 return CreateResult("schema-"+binascii.b2a_hex(os.urandom(16)).decode("utf-8"), outs=args)

 def delete(self, id, args):
 connection = connector.connect(user=args["creator_name"],
 password=args["creator_password"],
 host=args["server_address"],
 database=args["database_name"])
 cursor = connection.cursor()
 cursor.execute(args["deletion_script"])

 def diff(self, id, oldInputs, newInputs):
 replaces = []
 if (oldInputs["server_address"] != newInputs["server_address"]): replaces.append("server_address")
 if (oldInputs["database_name"] != newInputs["database_name"]): replaces.append("database_name")
 if (oldInputs["creation_script"] != newInputs["creation_script"]): replaces.append("creation_script")

 return DiffResult(
 changes=oldInputs != newInputs,
 replaces=replaces,
 stables=None,
 delete_before_replace=True)

 def update(self, id, oldInputs, newInputs):
 return UpdateResult(outs={**newInputs})

And lastly, we add the main Schema resource that we instantiate in our infrastructure code. We make all the inputs to our provider available as outputs, so that they can be accessed and exported.

class Schema(Resource):
 creator_name: Output[str]
 creator_password: Output[str]
 server_address: Output[str]
 database_name: Output[str]
 creation_script: Output[str]
 deletion_script: Output[str]
 def __init__(self, name: str, args: SchemaInputs, opts = None):
 super().__init__(SchemaProvider(), name, vars(args), opts)

With the dynamic provider finished, all that is left is to create it in our infrastructure. Like the previous post, the project uses a __main__.py file; the first few lines indicate the libraries to import and describe the application’s configuration options.

import json
import base64
import pulumi
import pulumi_aws as aws
import pulumi_mysql as mysql
from MySqlDynamicProvider import *

config = pulumi.Config()
admin_name = config.require("sql-admin-name")
admin_password = config.require_secret("sql-admin-password")
user_name = config.require("sql-user-name")
user_password = config.require_secret("sql-user-password")
availability_zone = pulumi.Config("aws").get("region")

To allow different tasks within our project to communicate, we create a Virtual Private Cloud and an associated subnet. Two subnets are required for the project, so the availability zone suffix is set to “a”.

app_vpc = aws.ec2.Vpc("app-vpc",
 cidr_block="172.31.0.0/16",
 enable_dns_hostnames=True)

app_vpc_subnet = aws.ec2.Subnet("app-vpc-subnet",
 cidr_block="172.31.0.0/20",
 availability_zone=availability_zone + "a",
 vpc_id=app_vpc)

A gateway and routing table are needed to allow the VPC to communicate with the Internet. Once created, we associate the routing table with our VPC.

app_gateway = aws.ec2.InternetGateway("app-gateway",
 vpc_id=app_vpc.id)

app_routetable = aws.ec2.RouteTable("app-routetable",
 routes=[
 {
 "cidr_block": "0.0.0.0/0",
 "gateway_id": app_gateway.id,
 }
 ],
 vpc_id=app_vpc.id)

app_routetable_association = aws.ec2.MainRouteTableAssociation("app_routetable_association",
 route_table_id=app_routetable.id,
 vpc_id=app_vpc)

To control traffic flow between applications running inside our VPC, we create a security group.

app_security_group = aws.ec2.SecurityGroup("security-group",
 vpc_id=app_vpc.id,
 description="Enables HTTP access",
 ingress=[{
 'protocol': 'tcp',
 'from_port': 0,
 'to_port': 65535,
 'cidr_blocks': ['0.0.0.0/0'],
 }],
 egress=[{
 'protocol': '-1',
 'from_port': 0,
 'to_port': 0,
 'cidr_blocks': ['0.0.0.0/0'],
 }])

Our MySQL database is created with an RDS instance. To create the instance, Amazon requires that it be given two subnets in different availability zones.

extra_rds_subnet = aws.ec2.Subnet("extra-rds-subnet",
 cidr_block="172.31.128.0/20",
 availability_zone=availability_zone + "b",
 vpc_id=app_vpc)

Both subnets are assigned to a SubnetGroup and belong to the RDS instance.

app_database_subnetgroup = aws.rds.SubnetGroup("app-database-subnetgroup",
 subnet_ids=[app_vpc_subnet.id, extra_rds_subnet.id])

mysql_rds_server = aws.rds.Instance("mysql-server",
 engine="mysql",
 username=admin_name,
 password=admin_password,
 instance_class="db.t2.micro",
 allocated_storage=20,
 skip_final_snapshot=True,
 publicly_accessible=True,
 db_subnet_group_name=app_database_subnetgroup.id,
 vpc_security_group_ids=[app_security_group.id])

Pulumi offers some additional tools to make handling MySQL easier.

mysql_provider = mysql.Provider("mysql-provider",
 endpoint=mysql_rds_server.endpoint,
 username=admin_name,
 password=admin_password)

We initialize the example database and create a user to manage it.

mysql_database = mysql.Database("mysql-database",
 name="votes-database",
 opts=pulumi.ResourceOptions(provider=mysql_provider))

mysql_user = mysql.User("mysql-standard-user",
 user=user_name,
 host="example.com",
 plaintext_password=user_password,
 opts=pulumi.ResourceOptions(provider=mysql_provider))

The user only needs the SELECT and UPDATE permissions to function.

mysql_access_grant = mysql.Grant("mysql-access-grant",
 user=mysql_user.user,
 host=mysql_user.host,
 database=mysql_database.name,
 privileges= ["SELECT", "UPDATE"],
 opts=pulumi.ResourceOptions(provider=mysql_provider))

Now, we use our Dynamic Provider. The provider takes creation_script as a parameter, connects to our MySQL server, and runs it during deployment.

creation_script = """
 CREATE TABLE votesTable (
 choice_id int(10) NOT NULL AUTO_INCREMENT,
 vote_count int(10) NOT NULL,
 PRIMARY KEY (choice_id)
 ) ENGINE=InnoDB;
 INSERT INTO votesTable(choice_id, vote_count) VALUES (0,0);
 INSERT INTO votesTable(choice_id, vote_count) VALUES (1,0);
 """

The deletion_script parameter drops the table and deletes stored data.

deletion_script = "DROP TABLE votesTable CASCADE"

When we create our resource, it behaves the same way as any other Pulumi resource but takes its arguments as a SchemaInputs object.

mysql_votes_table = Schema(name="mysql_votes_table",
 args=SchemaInputs(admin_name, admin_password, mysql_rds_server.address, mysql_database.name, creation_script, deletion_script))

We can export the ID of our new resource, and view it when deployed.

pulumi.export("dynamic-resource-id",mysql_votes_table.id)

In this example, I showed how straightforward it is to expand functionality with Pulumi by writing additional code. Dynamic Providers enable excellent flexibility in cloud architecture design and help break down barriers that would otherwise be challenging to overcome.

Next week, I’ll change the frontend from Flask to Django, and will show how to integrate it with our new MySQL server.

The blog post’s full code and an in-depth explanation for each component can be found on Github.

Creating a Python AWS Application Using Flask and Redis

Vova Ivanov — Thu, 13 Aug 2020 15:06:26 -0700

Meet Vova Ivanov—one of the Pulumi summer interns. He’ll be writing about his experiences learning Pulumi while modernizing a web app and its underlying infrastructure.

I’ve recently started developing with Pulumi, and have begun to explore its inner workings and mechanisms. I decided to construct a production-level application and to document each step that I take and my progress as I go along.

This blog post recreates the existing Typescript voting app example step by step in Python with Flask as the frontend and Redis as the backend. In future blog posts, we will explore how to change the front and backends, how to upgrade the app with additional AWS services, and migrating from one cloud provider to another.

The first step is to clone the Typescript voting app example. We’ll use a sparse clone to copy only the aws-ts-voting-app directory.

$ mkdir examples && cd examples
$ git init
$ git remote add origin -f https://github.com/pulumi/examples/
$ git config core.sparseCheckout true
$ echo aws-ts-voting-app >> .git/info/sparse-checkout
$ git pull origin master

The next step is to create a new directory and initialize a Pulumi project with pulumi new aws-python.

$ mkdir aws-py-voting-app && cd aws-py-voting-app
$ pulumi new aws-python

Since we won’t be modifying the application’s frontend, we simply copy the frontend/ folder from the example into our directory.

$ cp -r aws-ts-voting-app/frontend/ aws-py-voting-app/frontend

The requirements.txt file lists the libraries that the project depends on. We will need to add the following:

pulumi-docker>=2.0.0,<3.0.0

The biggest difference between our project and the example is the code describing the cloud infrastructure we want to create. Instead of an index.ts file, the project uses a __main__.py file. The first few lines of the file will indicate which libraries to import and describe a pair of configuration options used by the application.

import json
import pulumi
import pulumi_aws as aws
import pulumi_docker as docker

config = pulumi.Config()
redis_password = config.require("redis-password")
redis_port = 6379

After setting up the imports and configurations, we create an Elastic Container Service Cluster. A Cluster represents a group of tasks and services that work together for a certain purpose. In this instance, the purpose is to provide users with a voting application.

app_cluster = aws.ecs.Cluster("app-cluster")

To allow different tasks within our cluster to communicate, we create a Virtual Private Cloud and an associated subnet.

app_vpc = aws.ec2.Vpc("app-vpc",
 cidr_block="172.31.0.0/16",
 enable_dns_hostnames=True)

app_vpc_subnet = aws.ec2.Subnet("app-vpc-subnet",
 cidr_block="172.31.32.0/20",
 vpc_id=app_vpc)

A gateway and routing table are needed to allow the VPC to communicate with the Internet. Once created, we associate the routing table with our VPC.

app_gateway = aws.ec2.InternetGateway("app-gateway",
 vpc_id=app_vpc.id)

app_routetable = aws.ec2.RouteTable("app-routetable",
 routes=[
 {
 "cidr_block": "0.0.0.0/0",
 "gateway_id": app_gateway.id,
 }
 ],
 vpc_id=app_vpc.id)

app_routetable_association = aws.ec2.MainRouteTableAssociation("app_routetable_association",
 route_table_id=app_routetable.id,
 vpc_id=app_vpc)

To control traffic flow between applications running inside our VPC, we create a security group.

app_security_group = aws.ec2.SecurityGroup("security-group",
 vpc_id=app_vpc.id,
 description="Enables HTTP access",
 ingress=[{
 'protocol': 'tcp',
 'from_port': 0,
 'to_port': 65535,
 'cidr_blocks': ['0.0.0.0/0'],
 }],
 egress=[{
 'protocol': '-1',
 'from_port': 0,
 'to_port': 0,
 'cidr_blocks': ['0.0.0.0/0'],
 }])

To start our services, we need to create an Identity and Access Management (IAM) role, and attach execution permissions to it.

app_exec_role = aws.iam.Role("app-exec-role",
 assume_role_policy="""{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Action": "sts:AssumeRole",
 "Principal": {
 "Service": "ecs-tasks.amazonaws.com"
 },
 "Effect": "Allow",
 "Sid": ""
 }]
 }""")

exec_policy_attachment = aws.iam.RolePolicyAttachment("app-exec-policy", role=app_exec_role.name,
 policy_arn=aws.iam.ManagedPolicy.AMAZON_ECS_TASK_EXECUTION_ROLE_POLICY)

Likewise, our ECS service will need to have a task role to manage it, along with its own set of permissions.

app_task_role = aws.iam.Role("app-task-role",
 assume_role_policy="""{
 "Version": "2012-10-17",
 "Statement": [
 {
 "Action": "sts:AssumeRole",
 "Principal": {
 "Service": "ecs-tasks.amazonaws.com"
 },
 "Effect": "Allow",
 "Sid": ""
 }]
 }""")

task_policy_attachment = aws.iam.RolePolicyAttachment("app-access-policy", role=app_task_role.name,
 policy_arn=aws.iam.ManagedPolicy.AMAZON_ECS_FULL_ACCESS)

An Elastic Container Registry Repository stores the application Docker images that we want to run. The life cycle policy automatically removes the oldest untagged image that we uploaded.

app_ecr_repo = aws.ecr.Repository("app-ecr-repo",
 image_tag_mutability="MUTABLE")

app_lifecycle_policy = aws.ecr.LifecyclePolicy("app-lifecycle-policy",
 repository=app_ecr_repo.name,
 policy="""{
 "rules": [
 {
 "rulePriority": 10,
 "description": "Remove untagged images",
 "selection": {
 "tagStatus": "untagged",
 "countType": "imageCountMoreThan",
 "countNumber": 1
 },
 "action": {
 "type": "expire"
 }
 }
 ]
 }""")

With the basic infrastructure in place, we can begin to set up the application itself. First, we will start with the Redis backend.

We need to create a target group, a balancer, and a listener to allow Redis to communicate with the other services in our VPC, w

redis_targetgroup = aws.lb.TargetGroup("redis-targetgroup",
 port=redis_port,
 protocol="TCP",
 target_type="ip",
 stickiness= {
 "enabled": False,
 "type": "lb_cookie",
 },
 vpc_id=app_vpc.id)

redis_balancer = aws.lb.LoadBalancer("redis-balancer",
 load_balancer_type="network",
 internal= False,
 security_groups=[],
 subnets=[app_vpc_subnet.id])

redis_listener = aws.lb.Listener("redis-listener",
 load_balancer_arn=redis_balancer.arn,
 port=redis_port,
 protocol="TCP",
 default_actions=[{
 "type": "forward",
 "target_group_arn": redis_targetgroup.arn
 }])

Every service running within ECS requires a task definition, specifying what hardware, internet, and container settings it will use.

redis_task_definition = aws.ecs.TaskDefinition("redis-task-definition",
 family="redis-task-definition-family",
 cpu="256",
 memory="512",
 network_mode="awsvpc",
 requires_compatibilities=["FARGATE"],
 execution_role_arn=app_exec_role.arn,
 task_role_arn=app_task_role.arn,
 container_definitions=json.dumps([{
 "name": "redis-container",
 "image": "redis:alpine",
 "memory": 512,
 "essential": True,
 "portMappings": [{
 "containerPort": redis_port,
 "hostPort": redis_port,
 "protocol": "tcp"
 }],
 "command": ["redis-server", "--requirepass", redis_password],
 }]))

Now that we have our task definition blueprint ready, we can launch a service using it. An endpoint is created to hold information needed to connect the backend with the frontend.

redis_service = aws.ecs.Service("redis-service",
 cluster=app_cluster.arn,
 desired_count=1,
 launch_type="FARGATE",
 task_definition=redis_task_definition.arn,
 wait_for_steady_state=False,
 network_configuration={
 "assign_public_ip": "true",
 "subnets": [app_vpc_subnet.id],
 "security_groups": [app_security_group.id]
 },
 load_balancers=[{
 "target_group_arn": redis_targetgroup.arn,
 "container_name": "redis-container",
 "container_port": redis_port,
 }],
 opts=pulumi.ResourceOptions(depends_on=[redis_listener]),
)

redis_endpoint = {"host": redis_balancer.dns_name, "port": redis_port}

The Redis backend is complete, and now all that’s left is to create the Flask frontend.

A similar set of a target group, balancer, and listener is created for the frontend. Unlike Redis, Flask will serve as the webpage and, as such, needs to receive its data from port 80.

flask_targetgroup = aws.lb.TargetGroup("flask-targetgroup",
 port=80,
 protocol="TCP",
 target_type="ip",
 stickiness= {
 "enabled": False,
 "type": "lb_cookie",
 },
 vpc_id=app_vpc.id)

flask_balancer = aws.lb.LoadBalancer("flask-balancer",
 load_balancer_type="network",
 internal=False,
 security_groups=[],
 subnets=[app_vpc_subnet.id])

flask_listener = aws.lb.Listener("flask-listener",
 load_balancer_arn=flask_balancer.arn,
 port=80,
 protocol="TCP",
 default_actions=[{
 "type": "forward",
 "target_group_arn": flask_targetgroup.arn
 }])

The application is built into a Docker image and pushed to our ECR repository we created earlier.

def get_registry_info(rid):
 creds = aws.ecr.get_credentials(registry_id=rid)
 decoded = base64.b64decode(creds.authorization_token).decode()
 parts = decoded.split(':')
 if len(parts) != 2:
 raise Exception("Invalid credentials")
 return docker.ImageRegistry(creds.proxy_endpoint, parts[0], parts[1])

app_registry = app_ecr_repo.registry_id.apply(get_registry_info)

flask_image = docker.Image("flask-dockerimage",
 image_name=app_ecr_repo.repository_url,
 build="./frontend",
 skip_push=False,
 registry=app_registry
)

A task definition is created for the frontend; this time, using the flask image that we built. The redis endpoint is passed in as a group of environment variables, informing our app of our database’s host and port values.

flask_task_definition = aws.ecs.TaskDefinition("flask-task-definition",
 family="frontend-task-definition-family",
 cpu="256",
 memory="512",
 network_mode="awsvpc",
 requires_compatibilities=["FARGATE"],
 execution_role_arn=app_exec_role.arn,
 task_role_arn=app_task_role.arn,
 container_definitions=pulumi.Output.all(flask_image.image_name, redis_endpoint).apply(lambda args: json.dumps([{
 "name": "flask-container",
 "image": args[0],
 "memory": 512,
 "essential": True,
 "portMappings": [{
 "containerPort": 80,
 "hostPort": 80,
 "protocol": "tcp"
 }],
 "environment": [
 { "name": "REDIS", "value": args[1]["host"] },
 { "name": "REDIS_PORT", "value": str(args[1]["port"]) },
 { "name": "REDIS_PWD", "value": redis_password },
 ],
 }])))

The service for our Flask frontend is launched. It can receive requests from the Internet, process them, and change the Redis database accordingly.

flask_service = aws.ecs.Service("flask-service",
 cluster=app_cluster.arn,
 desired_count=1,
 launch_type="FARGATE",
 task_definition=flask_task_definition.arn,
 wait_for_steady_state=False,
 network_configuration={
 "assign_public_ip": "true",
 "subnets": [app_vpc_subnet.id],
 "security_groups": [app_security_group.id]
 },
 load_balancers=[{
 "target_group_arn": flask_targetgroup.arn,
 "container_name": "flask-container",
 "container_port": 80,
 }],
 opts=pulumi.ResourceOptions(depends_on=[flask_listener]),
)

To connect to our application, we export the DNS name of the Flask balancer, and open it in a browser window.

pulumi.export("app-url", flask_balancer.dns_name)

In this example, I showed how straightforward it is to convert infrastructure code defined in TypeScript to one defined in Python. Teams often choose to standardize on one language and Pulumi expands that choice, allowing them to standardize on one language for the frontend, the backend, and the infrastructure.

Next week, I’ll show how to modernize the application by changing the frontend from Flask to Django, and swapping out the database from Redis to Amazon RDS—using Pulumi to stand up new resources as I go.

The full code for the blog post can be found on github.