Pulumi Blog: Getting Started with Kubernetes Series

Getting Started With Kubernetes: Day 2

Sophia Parafina — Mon, 10 Aug 2020 00:00:00 +0000

Your application made it out of the dev stage, passed the testing stage, and arrived in production. As a developer, you might think that it’s an ops problem now. However, DevOps is a collaborative effort between developers and operators to build and maintain applications using shared techniques and processes, often called “Day 2” activities.

Day 2 tasks fall into three categories. The first category is managing cluster components. It involves managing nodes, users and groups, ingress and egress, networking, storage, and operators. The second category is changing cluster components, including setting optimal resource quota, reclaiming unneeded resources, scaling, tuning and updating clusters, and managing CRDs. The third category of tasks is cluster monitoring and logging, which involves collecting and analyzing metrics, and monitoring the health of the cluster.

Although this subject is much deeper than we can cover in a single article, let’s take a look at how infrastructure as code can simplify some of these tasks.

Managing secrets

In addition to managing users and groups, managing secrets such as credentials is an important task. The Pulumi CLI lets you rotate the secrets provider for a stack. This is particularly important when people leave an organization or if you want to change to a different secrets provider, e.g., from Vault to AWS KMS. Also, we can copy secrets from one stack to another. A frequent use of this capability is to copy database account credentials from one stack to another. You can read more about managing secrets and the using the CLI to rotate providers on the Pulumi blog.

Updating clusters

When you first deploy an application in production, you make assumptions about the configuration. However, as the number of users increases or features are added, the cluster configuration will change. Moving to larger machines that can support more nodes is common; however, we want to make that change without disrupting the service. To accomplish the update, we want to create the new larger node group, migrate the application, and decommission the preceding node group. Pulumi shows you how to do this in a tutorial. A more descriptive blog post walks you through the project source code.

In addition to zero downtime cluster migrations, we can also control scaling. The following tutorial shows how to stage an application rollout from dev to testing controlled by performance collecting metrics. In this tutorial, we move a three replica canary to a ten replica staging deployment. Replicas are added based on a P90 response time, i.e., 90% of requests are processed within 2.75 seconds. The example uses Prometheus to collect the response times and a utility class to perform the health check. Check out the complete code on Github.

Pulumi also lets you keep an eye on your Kubernetes resources in real-time with Pulumi watch. Results are updated live as you make changes like adding replicas or exposing an application with a load balancer.

Monitoring

There are many logging and monitoring solutions available for Kubernetes. In the previous section, we used Prometheus for generating metrics. Pulumi supports logging and monitoring tools to assist with cluster and application management.

Kubernetes Cluster Services provide logging and monitoring at the cluster level or a subset of apps and workloads. Check out our tutorials for setting up logging and monitoring for Kubernetes on AWS, Azure, and GCP. The tutorial also shows how to configure DataDog to log and monitor applications. The example code is on Github.

Summary

Maintaining a Kubernetes cluster and modern applications can be varied and complex. However, infrastructure as code lets developers and operators use a common language to manage the cluster and resources. Pulumi has a rich toolset to help you accomplish Day 2 tasks, whether managing cluster components such as secrets, changing cluster components to scale applications, or logging and monitoring your cluster and resources.

Each article in this series is intended to be independent of each other. However, we build upon concepts introduced in previous articles. If some concepts or terminology are unfamiliar, I encourage reading the earlier articles:

Getting Started With Kubernetes: Networking

Sophia Parafina — Thu, 30 Jul 2020 00:00:00 +0000

In previous installments, we examined how to deploy applications. However, we only touched on how applications talk to each other inside and outside the cluster. Whether you are building a modern application or modernizing a legacy application, understanding how resources and components talk to each other is essential. In this installment, we’ll examine networking in Kubernetes.

Networking in a distributed environment can be complicated. Let’s use a hypothetical example of a web application deployed as microservices in Kubernetes. Containers must be able to communicate with each other; for example, we have a Java app running on Tomcat served by Nginx for the application’s frontend.

Pods must be able to connect to other pods. In our example, the frontend sends requests to a REST API connected to a database deployed as a StatefulSet in another Pod. Kubernetes assigns each Pod an IP address, which lets them send requests to each other.

However, as Pods are destroyed and created, the IP addresses will change. Kubernetes uses the Service object as an abstraction to define a logical set of Pods, labels and selectors to access them.

Our web application also needs to be available outside the cluster. We’ll examine how traffic receives and responds to requests outside the cluster.

Container to container networking

A Pod is a logical host, and each Pod is assigned a network namespace, which is a logical networking stack that provides isolation between network devices. Containers in a Pod have the same IP address assigned to the Pod’s networking namespace, which means that containers within a Pod can communicate using localhost. In our example application, the nginx container at port 80 is a reverse proxy for the Java application at port 8080. At this point, neither the Java application nor the reverse proxy is available outside the cluster. We could define a nodePort, a LoadBalancer, or Ingress to expose the pod outside of the cluster, but we’ll address that later.

What does this look like in code?

// Create the frontend Pod.
const pod = new k8s.core.v1.Pod("frontend", {
 spec: {
 restartPolicy: "OnFailure",
 containers: [
 {
 name: "nginx",
 image: "nginx",
 resources: {requests: {cpu: "50m", memory: "50Mi"}},
 ports: [{ containerPort: 80 }],
 },
 {
 name: "webapp",
 image: "java_app",
 resources: {requests: {cpu: "50m", memory: "50Mi"}},
 ports: [{ containerPort: 8080 }],
 command: [ "catalina.sh", "run"],
 }
 ],
 }
}, { provider: provider });

Pod to Pod networking

In our the web application example, the REST microservice and the database are deployed in separate pods. The frontend pod sends requests to the REST microservice, which sends requests to the database and returns data to the frontend application.

Pods can communicate with any Pod on any Node without a NAT (Network Address Translation). Pod to Pod networking is accomplished through a virtual ethernet device on the Node. Each Pod connects to a virtual ethernet device (called vethx) that acts as a tunnel between the Pod and the Node. A network bridge (called cbr0) connects the Pod’s network stack to the Node’s network by creating a single aggregate network from multiple networks.

A Pod can communicate with Pods on different Nodes. If the Pod isn’t found on the Node, the network bridge goes to the cluster level to map the IP address ranges to nodes. Once the Pod with the matching address range is found, the request is sent to the target Pod. In our example, the webapp sends a request to the database (10.0.20.1) on another node.

Pod to Service networking

Pods are ephemeral, Kubernetes add or destroy pods when scaling up or down, applications crash, and runtimes reboot. Pod IP addresses are dynamic; Services are built to ensure that network traffic reaches the correct Pod by managing the state of Pods.

Kubernetes assigns a virtual IP address called a clusterIP when a Service is created. Multiple Pods can be associated with a service, and Kubernetes load-balances requests to Pods associated with a Service. Kubernetes keeps track of changing IP addresses through these three methods: iptables, IPVS, and DNS.

Let’s take a look at iptables, which is a common case. Each node has a kube-proxy which maintains a table of IP addresses for nodes. The table keeps track of how to reach Pods inside or outside your cluster. Kube-proxy watches for the addition and removal of Service and Endpoint objects, which are the list of addresses (IP and port) of endpoints that implement a Service.

Iptables use fewer system resources because traffic is handled by Linux netfilter, a packet filtering framework inside the Linux kernel. This means that routing traffic doesn’t require switching between the user space where Kubernetes is running and makes use of the host system’s kernel network stack, which also makes it more reliable.

In iptables mode, if the first Pod selected fails to respond, the connection fails. You can use readiness probes to tell kube-proxy which pods are healthy as in this example.

There are other methods of Pod to Service networking, such as:

userspace with kube-proxy.
IP Virtual Server (IPVS), an in-cluster load balancer that uses more efficient hash tables that scale better than iptables.
DNS running as a Kubernetes service

We’ve reviewed how packets are routed between Pods and Services, but how does this work in practice? A Service is a logical collection of pods with a policy that defines access among them. Pods are grouped into a Service with labels and selectors. In our web application example, we can label Pods with “webapp” and declare the service’s selector.

Below is an example of how to create a service with code.

const frontendLabels = { app: "frontend" };
const frontendDeployment = new k8s.apps.v1.Deployment("frontend", {
 spec: {
 selector: { matchLabels: frontendLabels },
 replicas: 3,
 template: {
 metadata: { labels: frontendLabels },
 spec: {
 containers: [
 {
 name: "nginx",
 image: "nginx",
 resources: { requests: { cpu: "100m", memory: "100Mi" } },
 env: [{ name: "GET_HOSTS_FROM", value: "env" }],
 ports: [{ containerPort: 80 }],
 },
 {
 name: "webapp",
 image: "java_app",
 resources: { requests: { cpu: "100m", memory: "100Mi" } },
 env: [{ name: "GET_HOSTS_FROM", value: "env" }],
 ports: [{ containerPort: 8080 }],
 },

 ],
 },
 },
 },
});
const frontendService = new k8s.core.v1.Service("frontend", {
 metadata: {
 labels: frontendDeployment.metadata.labels,
 name: "frontend",
 },
 spec: {
 type: LoadBalancer,
 ports: [{ port: 80 },{ port:8080 }],
 selector: frontendDeployment.spec.template.metadata.labels,
 },
});

Egress and Ingress

Egress is moving data and requests from the cluster to the Internet. Kubernetes can accomplish egress with an Internet gateway that routes packets from inside the cluster by performing network address translation (NAT). NAT maps a Node’s internal IP address to an external IP address on the public Internet.

However, Pods have their IP address that virtual machines or Nodes can’t reach. If we trace a packet’s route, it starts from the Pod’s namespace and connects to root namespace of the Node via veth to the network bridge, cbr0. Iptables replaces the source IP of the Pod to the Node Ip address to get from the Node to the gateway. Changing the source IP is called a source NAT (SNAT), and it lets the packet travel from the Node to the Internet gateway.

Ingress is the process of routing Internet traffic to the Kubernetes cluster. There are two solutions for getting Internet traffic into your cluster: a Service LoadBalancer and an Ingress controller.

A Service load balancer is also called a Layer 4 LoadBalancer. The name refers to Layer 4, or the transport layer of the OSI network model. At Layer 4, the transport layer routes packets and resends them if they are not received. You can specify a LoadBalancer when creating a Service. The Service will advertise the IP address, and you can send traffic to LoadBalancer, which will route packets to the service.

Kubernetes can create a Service with an optional LoadBalancer. When the Service is created, it advertises the IP address for the load balancer. As an end-user, you can start directing traffic to the load balancer to begin communicating with your Service.

The load balancer is not aware of Pods or containers, so it sends packets to the Nodes in the cluster. The iptables rules in each Node will send the packets to Pod. The Pod’s response uses the Pod’s IP, but iptables rewrites the correct IP on the return with NAT.

The other way of routing packets to a pod from a gateway is with a Layer 7 Ingress Controller. Layer 7 is the application layer of the OSI network model. Layer 7 Ingress works on the HTTP/HTTPS portion of the network stack. Similar to a Layer 4 LoadBalancer, it is part of a Service. You must open a port in the Service with the NodePort Service type, and Kubernetes will allocate a port from a specified range. Traffic routed to the Node’s port will be sent to the service by iptables rules.

An Ingress object exposes a Node’s port to the Internet. An Ingress load balancer maps HTTP requests to Kubernetes Services. The Ingress method differs depending on how the Kubernetes provider implements it. HTTP load balancers, like Layer 4 network load balancers, only understand Node IPs (not Pod IPs), so traffic routing uses the internal load-balancing in iptables installed on each Node by kube-proxy.

Within an AWS environment, the ALB Ingress Controller provides Kubernetes Ingress using Amazon’s Layer 7 Application Load Balancer. The following diagram details the AWS components this controller creates. It also demonstrates the route Ingress traffic takes from the ALB to the Kubernetes cluster. This process is similar to a Layer 4 Load Balancer, but the significant difference is that the Application Load Balancer is HTTP aware and can perform host or path based routing.

Building on the previous example, we can expose the frontend by adding the following.

export const frontendIp: pulumi.Output<string>;
frontendIp = frontend.status.loadBalancer.ingress[0].ip;

Summary

The goal of this article is to provide an overview of the Kubernetes Networking model. The implementation of the Container Network Interface (CNI) varies from provider to provider. You can read more about CNI implementations on the Kubernetes documentation site.

Getting Started with Kubernetes: Stateful Applications

Sophia Parafina — Tue, 21 Jul 2020 00:00:00 +0000

This article is the fourth in a series using infrastructure as code to deploy applications with Kubernetes. This series walks you through:

Building a Kubernetes cluster on cloud providers
Basic application deployment
Advance application deployment and Helm charts
Stateful applications
Networking
“Day 2” activities such as migrating node groups.

In the previous post, we examined different methods for deploying applications. We worked through examples of a boilerplate deployment, to one using ComponentResources to automate deployment further, and deploying with Helm charts. In this installment, we’ll look at how to deploy stateful applications, such as databases, in Kubernetes. Unlike stateless applications, stateful apps require persistent storage, which presents scaling and availability challenges.

Stateful vs. Stateless

Stateless applications don’t require information from a previous transaction because they are single request and response. For example, the “Hello World” example that we deployed in an earlier article is a stateless application. The critical thing to remember is that in Kubernetes, any container with that application in any pod can respond to that request. Stateful applications can read or store information about previous transactions, and a transaction is executed within the context of a previous extraction. Email is an example of a stateful application; an email can be unread or read, or it can be grouped according to keywords or by content. In stateless applications, access to storage must be coordinated, either by sending all requests to the same Pod, or by synchronizing across multiple Pods.

Stateful sets

Stateful sets are the Kubernetes workload object for managing stateful applications. They are a specialized type of Deployments, where the Pods maintain a persistent identity during updates and rescheduling.

You would use a StatefulSet for:

stable and unique network identifiers
stable and persistent storage
applications that require ordered deployment and scaling
applications that require ordered and rolling updates

When a StatefulSet is created, each Pod is assigned a unique and persistent identifier. Pods are named numerically starting from 0, e.g., pod-0, pod-1, pod-n. StatefulSets use a Headless Service resource, which sets the clusterIP property to none. Pods in a StatefulSet will have an IP address mapped directly to them instead of a virtual IP address. This lets other Pods in the cluster directly communicate to an application, as in database connections.

StatefulSets can control the order of starting and stopping Pods. Some applications require a resource to be available before the Pod can be marked healthy. A Pod in a StatefulSet can be mounted to a dedicated persistent volume by declaring a PersistentVolumeClaim. If a Pod in a StatefulSet shuts down, the StatefulSet Controller will relaunch a replacement Pod with the same network identifier and reattach it to the persistent Volume.

StatefulSets have two update strategies. The OnDelete strategy does not automatically update the Pods, and they must be manually deleted for the controller to create new Pods. Rolling updates automatically deletes and creates a replacement for Pods in a StatefulSet. It creates replacement in the same order as they are deleted, which is from the highest to the smallest numbered Pod. A replacement Pod must be ‘Running and Ready’ before it replaces the preceding Pod.

Example

Let’s see how this works in practice with a MariaDB deployment in Kubernetes. The example begins with defining the cluster configuration and the MariaDB configuration using a ConfigMap. We’ll skip over this and go straight to the StatefulSet.

Headless Service

In the previous section, we described the Headless Service. The key parts are setting the ClusterIP to "None", which allocates an IP address and port so that applications can establish a database connection.

Deployment

Let’s look at the StatefulSet, step-by-step. The spec matches the StatefulSet to the Headless Service using a selector that matches the labels. We assign it a service name, specify a single replica, and a RollingUpdate update strategy. Note that this is the default update strategy, but it’s a good practice to declare it.

Pod scheduling

The next section declares variables for MariaDB, such as the serviceAccountName and the securityContext. However, of interest is the podAntiAffinity parameter, which ensures that the Pod will not be deployed on a Node with a Pod with the labels ‘mariadb’ and ‘example’. The weight is set for the preferred rule, where a node with a higher weight is preferred. We prevent another instance from being deployed on the same host by setting topologyKey: "kubernetes.io/hostname".

Putting it all together, we’ve told the Kubernetes Scheduler never to co-locate two Pods that have the app labels “mariadb” and “example” in the same domain (or Node) defined by setting topologyKey:”kubernetes.io/hostname”. These parameters control Pod scheduling to ensure that if a Pod is replaced, the new Pod is addressable by other resources in the application.

Configure the MariaDB container

The following section configures the MariaDB container. We’ll skip the part of the container configuration and look at healthchecks that impact the Pod’s scheduling. The livenessProbe is useful for detecting if a long-running application, such as a database, is broken and cannot recover except by a restart. In this example, we call the mysql init script as a status check. If the container fails the livenessProbe, the Kubernetes scheduler will kill the Pod and replace it. A readinessProbe is similar to a livenessProbe; if the probe fails, Kubernetes won’t send traffic to the Pod. Unlike a livenessProbe, the Pod remains running and is not terminated and replaced by the scheduler. Probe field or settings are described in the Kubernetes documentation.

We also configure two volumeMounts. The first volumeMount sets the MariaDB data path to the data Volume. The second volumeMount mounts the MariaDB configMap that was defined earlier.

Create a Volume for storage

The final section of code creates a Persistent Volume using a volumeClaimTemplate. This object requests a PVC (PersistentVolumeClaim) from the storage class dynamically. We set the accessModes to “ReadWriteOnce”, which means the volume can be mounted as read-write by a single node. Other accessModes are “ReadOnlyMany”, where the volume can be mounted read-only by many nodes, and “ReadWriteMany”, where the volume can be mounted as read-write by many nodes. We also set the size of the Volume to 8 gigabytes.

To summarize, we deployed MariaDB using a StatefulSet by creating a headlessService which assigns a persistent IP address to the Pod instead of using clusterIP and nodePorts to route traffic to the application. We then defined Node Affinity/AntiAffinity to set rules on which nodes are selected by the scheduler for creating Pods. Next, we configured the container and set livenessProbes and readinessProbes to tell the Scheduler if a Pod needs to be terminated and restarted or not to route traffic to the Pod until it is ready. We also configure volumeMounts and volumes to store configuration information and data for MariaDB.

Conclusion

Stateful applications present additional challenges when deployed in Kubernetes. Many applications require a stateful resource, such as a database or a component that maintains a login and session id. Stateful applications route traffic to a stable and persistent resource. Kubernetes accomplishes this with StatefulSets, which creates resources with a persistent id and unique address. We can configure scheduling and replacement of Pods, to allow resources to continue using a persistent id and address. We can also configure containers with health checks to signal if a Pod should be terminated or if it should continue to receive requests. We can also control which Node can access a PersistentVolume with a volumeClaimTemplate, and in our example, we limit access to only one Node. Make sure to try out the complete example on Github.

In the next series installment, we’ll examine Kubernetes networking. Until then, you can learn more about Kubernetes with these resources:

Watch educational content on Pulumi TV
Learn more about Pulumi’s support for Kubernetes
Practice Kubernetes Tutorials using Pulumi

Getting Started With Kubernetes: Advanced Deployment

Sophia Parafina — Thu, 09 Jul 2020 00:00:00 +0000

Welcome to the third article in a series using infrastructure as code to deploy applications with Kubernetes. In the previous post, we reviewed basic Kubernetes objects and abstractions used when deploying an application. We examined code examples across the cloud providers to show how to use infrastructure as code to deploy an application using Kubernetes objects. In this installment, we’ll progress from a simple deployment with just a single application container to a complex application with multiple containers and Pods.

This series walks you through:

Building a Kubernetes cluster on cloud providers
Basic application deployment
Advance application deployment and Helm charts
Stateful applications
Networking
“Day 2” activities such as migrating node groups.

This article reviews three types of application deployment scenarios with commonly used examples. The first scenario is a 12-factor microservices multi-tier application consisting of multiple frontend and backend services backed by databases and a messaging queue to handle requests. The second scenario takes the Kubernetes guestbook and turns the Service and Deployment into a single component, demonstrating another pattern for organizing your stack. The third scenario shows how to deploy a popular web application using a Helm, the package manager for Kubernetes.

In this article, we’ll assume that we already have a Kubernetes cluster available. You will have to set the context with kubectl to deploy the examples in this article. Because of Kubernetes’ OpenAPI, we can use the same Typescript application code on any compliant instance.

Sock Shop Microservices

We’ll use the Sock Shop microservices reference application. This is a polyglot application comprised of microservices written in Java, Go, and Node. This demo application adheres to the 12-factor app pattern by creating a microservice for each function. The application services will be deployed to pods, and they communicate with each other via REST calls over HTTP.

Let’s look at the Orders microservice, which is made up of a service worker and MongoDB for a backing service. Note that the code is organized by microservice with a separate Deployment and Service for each service and backing resource. The ordersDb configuration is straightforward, of note are the options in the container spec where we set securityContext and a emptyDir volume for storage. An emptyDir volume persists even if the container crashes, and the data in the volume is available if the container is restarted. The service definition for ordersDbService is also simple and opens port:27017 and ties the service back to the Deployment by the orders-db label.

const sockShopNs = new k8s.core.v1.Namespace("sock-shop", { metadata: { name: "sock-shop" } });

// --------------------------------------------------------------------------
// Orders microservice.
// --------------------------------------------------------------------------

const ordersDb = new k8s.apps.v1.Deployment("orders-db", {
 metadata: {
 name: "orders-db",
 labels: {
 name: "orders-db",
 },
 namespace: sockShopNs.metadata.name,
 },
 spec: {
 replicas: 1,
 template: {
 metadata: {
 labels: {
 name: "orders-db",
 },
 },
 spec: {
 containers: [
 {
 name: "orders-db",
 image: "mongo",
 ports: [
 {
 name: "mongo",
 containerPort: 27017,
 },
 ],
 securityContext: {
 capabilities: {
 drop: ["all"],
 add: ["CHOWN", "SETGID", "SETUID"],
 },
 readOnlyRootFilesystem: true,
 },
 volumeMounts: [
 {
 mountPath: "/tmp",
 name: "tmp-volume",
 },
 ],
 },
 ],
 volumes: [
 {
 name: "tmp-volume",
 emptyDir: {
 medium: "Memory",
 },
 },
 ],
 nodeSelector: {
 "beta.kubernetes.io/os": "linux",
 },
 },
 },
 },
});

const ordersDbService = new k8s.core.v1.Service("orders-db", {
 metadata: {
 name: "orders-db",
 labels: {
 name: "orders-db",
 },
 namespace: sockShopNs.metadata.name,
 },
 spec: {
 ports: [
 {
 port: 27017,
 targetPort: 27017,
 },
 ],
 selector: ordersDb.spec.template.metadata.labels,
 },
});

The orders Deployment is similar to the ordersDb Deployment, but it includes environmental variables, such as JAVA_OPTS to set heap memory size and garbage collection intervals, for the containerized application. The container runs as a non-root user and drops all capabilities that can run as a non-root user, except for NET_BIND_SERVICE, which lets the container bind to any port.

const orders = new k8s.apps.v1.Deployment("orders", {
 metadata: {
 name: "orders",
 labels: {
 name: "orders",
 },
 namespace: sockShopNs.metadata.name,
 },
 spec: {
 replicas: 1,
 template: {
 metadata: {
 labels: {
 name: "orders",
 },
 },
 spec: {
 containers: [
 {
 name: "orders",
 image: "weaveworksdemos/orders:0.4.7",
 env: [
 {
 name: "ZIPKIN",
 value: "zipkin.jaeger.svc.cluster.local",
 },
 {
 name: "JAVA_OPTS",
 value:
 "-Xms64m -Xmx128m -XX:PermSize=32m -XX:MaxPermSize=64m -XX:+UseG1GC -Djava.security.egd=file:/dev/urandom",
 },
 ],
 ports: [
 {
 containerPort: 80,
 },
 ],
 securityContext: {
 runAsNonRoot: true,
 runAsUser: 10001,
 capabilities: {
 drop: ["all"],
 add: ["NET_BIND_SERVICE"],
 },
 readOnlyRootFilesystem: true,
 },
 volumeMounts: [
 {
 mountPath: "/tmp",
 name: "tmp-volume",
 },
 ],
 },
 ],
 volumes: [
 {
 name: "tmp-volume",
 emptyDir: {
 medium: "Memory",
 },
 },
 ],
 nodeSelector: {
 "beta.kubernetes.io/os": "linux",
 },
 },
 },
 },
});

const ordersService = new k8s.core.v1.Service("orders", {
 metadata: {
 name: "orders",
 labels: {
 name: "orders",
 },
 namespace: sockShopNs.metadata.name,
 },
 spec: {
 ports: [
 {
 port: 80,
 targetPort: 80,
 },
 ],
 selector: orders.spec.template.metadata.labels,
 },
});

If we look at the orders-db deployment manifest, we can see that the code follows it closely. You can find the full Sock Shop code in the Pulumi examples repository.

Although we encapsulate the deployment in a single large script, we can improve that by creating components using classes to abstract the implementation details.

Deploying Guestbook as a Component

We can abstract objects, such as Deployments and Services, into components to create our resources. ComponentResource is Pulumi construct or resource that can aggregate other resources into a combined resource. Let’s take a look at how we implement the Guestbook with components.

The Kubernetes GuestBook is a canonical example of a stateless application. It consists of multiple PHP frontend services backed by Redis replicas and a lead Redis database to store guestbook entries. In the previous example, we created a Deployment and a Service for each microservice in the application. Although this is a valid pattern for deploying an application in Kubernetes, it doesn’t take full advantage of the capabilities of infrastructure as code.

When you examine index.ts, the main program, you’ll notice that it imports the k8sjs module. Note that each service is instantiated by calling the ServiceDeployment class and passing arguments such as the image, ports, and replicas. As you can see, this is a more efficient way to deploy services than coding individual Deployments and Services.

import * as pulumi from "@pulumi/pulumi";
import * as k8sjs from "./k8sjs";

const config = new pulumi.Config();

const redisLeader = new k8sjs.ServiceDeployment("redis-leader", {
 image: "redis",
 ports: [6379],
});

const redisReplica = new k8sjs.ServiceDeployment("redis-replica", {
 image: "pulumi/guestbook-redis-replica",
 ports: [6379],
});

const frontend = new k8sjs.ServiceDeployment("frontend", {
 replicas: 3,
 image: "pulumi/guestbook-php-redis",
 ports: [80],
 allocateIpAddress: true,
 isMinikube: config.getBoolean("isMinikube"),
});

export const frontendIp = frontend.ipAddress;

The ServiceDeployment class is defined in the k8sjs module using ComponentResource. We define deployment, service, and ipAddress as public read-only properties. The constructor takes three arguments:

name: The unique name of the resource.
args: Information passed to [initialize] method.
opts: A bag of options that control this resource’s behavior.

ServiceDeploymentArgs is the interface for the arguments used to initialize the method. Next, we instantiate the container using the container image property from the ServiceDeploymentArgs and configure resources, environmental variables, and ports. The deployment is instantiated with the properties you typically see in a deployment such labels, selectors, containers, and replicas. The same goes for Services. Finally, based on whether Kubernetes is running on cluster or Minikube, the ServiceDeployment class returns the clusterIP if it runs on Minikube or instantiates a load balancer if it is a cluster.

import * as k8s from "@pulumi/kubernetes";
import * as k8stypes from "@pulumi/kubernetes/types/input";
import * as pulumi from "@pulumi/pulumi";

/**
 * ServiceDeployment is an example abstraction that uses a class to fold together the common pattern of a
 * Kubernetes Deployment and its associated Service object.
 */
export class ServiceDeployment extends pulumi.ComponentResource {
 public readonly deployment: k8s.apps.v1.Deployment;
 public readonly service: k8s.core.v1.Service;
 public readonly ipAddress?: pulumi.Output<string>;

 constructor(name: string, args: ServiceDeploymentArgs, opts?: pulumi.ComponentResourceOptions) {
 super("k8sjs:service:ServiceDeployment", name, {}, opts);

 const labels = { app: name };
 const container: k8stypes.core.v1.Container = {
 name,
 image: args.image,
 resources: args.resources || { requests: { cpu: "100m", memory: "100Mi" } },
 env: [{ name: "GET_HOSTS_FROM", value: "dns" }],
 ports: args.ports && args.ports.map(p => ({ containerPort: p })),
 };
 this.deployment = new k8s.apps.v1.Deployment(name, {
 spec: {
 selector: { matchLabels: labels },
 replicas: args.replicas || 1,
 template: {
 metadata: { labels: labels },
 spec: { containers: [ container ] },
 },
 },
 }, { parent: this });

 this.service = new k8s.core.v1.Service(name, {
 metadata: {
 name: name,
 labels: this.deployment.metadata.labels,
 },
 spec: {
 ports: args.ports && args.ports.map(p => ({ port: p, targetPort: p })),
 selector: this.deployment.spec.template.metadata.labels,
 // Minikube does not implement services of type `LoadBalancer`; require the user to specify if we're
 // running on minikube, and if so, create only services of type ClusterIP.
 type: args.allocateIpAddress ? (args.isMinikube ? "ClusterIP" : "LoadBalancer") : undefined,
 },
 }, { parent: this });

 if (args.allocateIpAddress) {
 this.ipAddress = args.isMinikube ?
 this.service.spec.clusterIP :
 this.service.status.loadBalancer.ingress[0].ip;
 }
 }
}

export interface ServiceDeploymentArgs {
 image: string;
 resources?: k8stypes.core.v1.ResourceRequirements;
 replicas?: number;
 ports?: number[];
 allocateIpAddress?: boolean;
 isMinikube?: boolean;
}

As this example demonstrates, components let you build higher-level objects from smaller resources. Using a programming language alleviates writing tedious boilerplate code and results in a more efficient and compact deployment.

Helm Charts

Helm is described as the package manager for Kubernetes. Applications are packaged in charts, a collection of files that declare and configure Kubernetes resources for an application. Helm Hub is a repository for charts with over 1000 applications ready for deploying in Kubernetes.

The code below is from the Kubernetes Typescript Helm Chart example. Deploying a Helm chart is uncomplicated; all you need to do is instantiate a Helm Chart object and provide a name and the appropriate properties.

import * as k8s from "@pulumi/kubernetes";
import * as pulumi from "@pulumi/pulumi";

// Minikube does not implement services of type `LoadBalancer`; require the user to specify if we're
// running on minikube, and if so, create only services of type ClusterIP.
const config = new pulumi.Config();
if (config.require("isMinikube") === "true") {
 throw new Error("This example does not yet support minikube");
}

// Deploy the latest version of the stable/wordpress chart.
const wordpress = new k8s.helm.v2.Chart("wpdev", {
 repo: "stable",
 version: "2.1.3",
 chart: "wordpress",
});

// Export the public IP for Wordpress.
const frontend = wordpress.getResourceProperty("v1/Service", "wpdev-wordpress", "status");
export const frontendIp = frontend.apply(status => status.loadBalancer.ingress[0].ip);

Choosing how to deploy

Using a Helm chart to deploy an application is the simplest solution if it’s a standard application. If you are deploying your application, you might start with a boilerplate code defining Deployments, Services, and other objects in a prototype and later refactor it using Components. These methods aren’t exclusive; for example, you could use a Helm chart to deploy an ELK stack to gather metrics from your application and display them using Kibana. As an application becomes more complex, e.g., every function is a microservice, it becomes more unwieldy and difficult to manage. Infrastructure as code can help reduce that complexity through abstracting resources into higher-level objects, applying software development practices to infrastructure, and enabling robust and reliable deployments.

Conclusion

This article demonstrated three ways to deploy an application in Kubernetes. The applications we examined are primarily stateless with no persistent storage or volumes. In the next installment, we’ll examine stateful services that require persistent storage such as a database. Until the next post, learn more about Kubernetes with these resources.

Watch educational content on Pulumi TV
Learn more about Pulumi’s support for Kubernetes
Practice Kubernetes Tutorials using Pulumi

Getting Started With Kubernetes: Application Basics

Sophia Parafina — Tue, 30 Jun 2020 00:00:00 +0000

Welcome to the second article in a series using infrastructure as code to deploy applications with Kubernetes. The series walks you through building a Kubernetes cluster on cloud providers, deploying applications, and “Day 2” activities such as migrating Node groups. In the previous article, we showed how to create a Kubernetes cluster for AWS, Azure, and GCP. In this installment, we’ll learn how to deploy an application using Kubernetes objects.

Kubernetes has a rich feature set and it is important to keep in mind that Kubernetes is not merely a container orchestrator. It is designed to keep applications highly available and to scale on demand. It accomplishes this by running multiple instances of application containers and replacing them as needed.

There are many moving parts to Kubernetes, so let’s review the objects used to deploy applications in Kubernetes.

Pods

A Pod is where applications run, and it is the smallest unit of execution for Kubernetes. A Pod hosts one or more containers that make up a complete application. For example, a Pod can have a container running an entire LAMP stack for a web application, or it can run a container with NGINX that connects to a database running in another Pod.

Each Pod has an IP address and can address other Pods in the same network. In the previous article, we deployed clusters in a VPC. Pods deployed in a Kubernetes cluster can communicate with each other using TCP/IP, UDP, or SCTP. Containers within a Pod share the same IP address and port space, and can communicate via localhost or IPC, such as POSIX shared memory or SystemV semaphores.

Pods have a specific lifecycle. They are created, assigned a UID, scheduled to run on a Node, and run until they either fail or are terminated. When a Node dies, Pods are deleted and replaced onto other nodes. Pods have five distinct states or phases:

Pending The Pod has been scheduled for creation, but one or more containers have not been created.
Running The Pod has been created in the Node, all containers have been created, and at least one container is running.
Succeeded All the containers have been successfully terminated and will not be restarted.
Failed All containers have been terminated, but at least one container has exited with a failed non-zero status.
Unknown The state of the Pod was not returned, possibly due to a communication error.

When deploying an application, you can see the status of the Pods.

Services

A Service routes traffic to a logical set of Pods. For example, your application may deploy a web application, a data store for queued requests, and an RDBMS for customer data in separate Pods. A Service groups Pods using Labels and LabelSelectors. Labels are key/value pairs that you can use to identify objects meaningfully. Using the example above, the web application may be labeled tier : frontend and the database would be labeled tier : backend. Labels aren’t unique, and objects can have the same label. A LabelSelector identifies a set of objects and is a way to group objects.

Pod IPs are not exposed to traffic outside of the cluster. Applications require a Service to receive traffic, and by setting a ServiceSpec, they can be addressable by specifying a type:

ClusterIP (default) -The Service is only addressable from within the cluster.
NodePort - In addition to a ClusterIP, the Service is also available outside the cluster using <NodeIP>:<NodePort>.
LoadBalancer - In addition to a Cluster IP and Node Port, the Service is also assigned a load balancer to with a fixed, external IP that routes to the Service.
ExternalName - Uses a CNAME record to internally expose an external Service to Kubernetes within the cluster.

Volumes

Containers are ephemeral, and storage in a container is temporary as well. As soon as a container stops, any changes are lost. The lifecycle of a Kubernetes Volume is tied to a Pod, and the data in the volume is available across container restarts. If a Pod is terminated, then the data will also be lost.

Kubernetes supports many types of volumes, but of interest are PersistentVolumes, which persist beyond the lifecycle of a Pod. PersistentVolumes are a resource that has been provisioned by a Storage Class. To use a PersistentVolume, we have to declare a PersistentVolumeClaim (PVC) which allows a user to access abstract storage resources. We can declare parameters such as size, access (read/write), or volume expansion for a PVC. Applications that require a PVC are typically data stores or databases where data must be preserved outside the scope of the application in a Pod.

Namespaces

One of the advantages of Kubernetes is multi-tenancy or the ability to host many virtual clusters in the same physical cluster. Namespaces are used to organize application clusters by providing scope for names. To this end, Namespaces must be unique, cannot be nested inside another namespace, and resources can only belong to one namespace, if they are namespaced.

Namespaces are typically used where there are multiple teams or projects. For example, you may have an engineering and a marketing namespace. However, it’s more efficient to use labels if the applications are only slightly different than in grouping environments such as dev, test, and prod.

Controllers

Controllers manage the state of the cluster by making changes that move the cluster to the desired state. They watch resources making sure that objects, such as Pods, meet the spec for the declared state. The kube-controller-manager in the control plane has a built-in set of controllers. Let’s take a look at two commonly used controllers, Deployments and ReplicaSets.

A Deployment is a higher-level object that manages declarative updates for Pods and ReplicaSets. But first, let’s look at ReplicaSets, which are a set of Pods running the same application at any given time. As such, it is often used to guarantee the availability of a specified number of identical Pods. A ReplicaSet creates and deletes Pods as needed to meet the criteria of the declared state, and uses Labels and Selectors to determine which Pods to manage.

A Deployment is a higher-level abstraction that manages ReplicaSets by updating Pods. Deployments are commonly used instead of directly using ReplicaSets because the Deployment controller manages updates to the desired state at a controlled rate. For example, you can specify a rolling update to limit the downtime of an application.

Examples

Now that we have the basic Kubernetes objects, let’s see how they are used in a Kubernetes deployment across cloud providers.

In this example, we’ll deploy Nginx. We’ll skip the cluster creation, which we covered in the previous installment. We set name to helloworld and use it to set the Namespace for the application and the Label (appLabel). The Deployment uses these parameters in the metadata section.

The spec section of a Deployment specifies the desired state; in this example, we want one Replica, and the selector must match appLabel. The Deployment spec includes the application container and claims port 80 on the Pod. To make our deployment available outside the cluster, we create a LoadBalancer service that routes requests to the container. As with the Deployment, we add the Labels and Namespace to the metadata. The Service spec sets the type to LoadBalance and opens port 80 externally. Note that the selector uses appLabels to acquire the Pods from our Deployment.

...
const name = "helloworld";

// Create a Kubernetes Namespace
const ns = new k8s.core.v1.Namespace(name, {}, { provider: cluster.provider });

// Export the Namespace name
export const namespaceName = ns.metadata.name;

// Create a NGINX Deployment
const appLabels = { appClass: name };
const deployment = new k8s.apps.v1.Deployment(name,
 {
 metadata: {
 namespace: namespaceName,
 labels: appLabels,
 },
 spec: {
 replicas: 1,
 selector: { matchLabels: appLabels },
 template: {
 metadata: {
 labels: appLabels,
 },
 spec: {
 containers: [
 {
 name: name,
 image: "nginx:latest",
 ports: [{ name: "http", containerPort: 80 }],
 },
 ],
 },
 },
 },
 },
 {
 provider: cluster.provider,
 },
);

// Export the Deployment name
export const deploymentName = deployment.metadata.name;

// Create a LoadBalancer Service for the NGINX Deployment
const service = new k8s.core.v1.Service(name,
 {
 metadata: {
 labels: appLabels,
 namespace: namespaceName,
 },
 spec: {
 type: "LoadBalancer",
 ports: [{ port: 80, targetPort: "http" }],
 selector: appLabels,
 },
 },
 {
 provider: cluster.provider,
 },
);

// Export the Service name and public LoadBalancer Endpoint
export const serviceName = service.metadata.name;
export const serviceHostname = service.status.loadBalancer.ingress[0].hostname;

We can compare this deployment to a YAML manifest that accomplishes the same thing. The YAML deployment is similar to the one above written in code. A manifest for the service is also required to deploy the application. You can use kubectl to apply the manifests.

apiVersion: apps/v1
kind: Deployment
metadata:
 Labels:
 namespace: helloworld
 labels: appLabels
spec:
 replicas: 1
 selector:
 matchLabels: appLabels
 template:
 metadata:
 labels: appLabels
 spec:
 containers:
 - name: helloworld
 image: nginx:latest
 ports:
 - containerPort: 80

Although this is a simple example, you can see where using a programming language has advantages over editing YAML files, e.g., you can edit the name, namespace, and labels just once. The full example is available on GitHub for you to try.

In this example, we’ll deploy Nginx. We’ll skip the cluster creation which we covered in the previous installment. We set name to helloworld and use it to set the Namespace for the application and the Label (appLabel). The Deployment uses these parameters in the metadata section.

The spec section of a Deployment specifies the desired state; in this example, we want one Replica, and the selector must match the appLabel. The Deployment spec includes the application container and claims port:80 on the Pod. To make our deployment available outside the cluster, we create a LoadBalancer service that routes requests to the container. As with the Deployment, we add the Labels and Namespace to the metadata. The Service spec sets the type to LoadBalance and opens port 80 externally. Note that the selector uses appLabels to acquire the Pods from our Deployment.

...
const name = "helloworld";

// Create a Kubernetes Namespace
const ns = new k8s.core.v1.Namespace(name, {}, { provider: cluster.provider });

// Export the Namespace name
export const namespaceName = ns.metadata.name;

const deployment = new k8s.apps.v1.Deployment(name,
 {
 metadata: {
 namespace: namespaceName,
 labels: appLabels,
 },
 spec: {
 replicas: 1,
 selector: { matchLabels: appLabels },
 template: {
 metadata: {
 labels: appLabels,
 },
 spec: {
 containers: [
 {
 name: name,
 image: "nginx:latest",
 ports: [{ name: "http", containerPort: 80 }],
 },
 ],
 },
 },
 },
 },
 {
 provider: cluster.provider,
 },
);

// Export the Deployment name
export const deploymentName = deployment.metadata.name;

// Create nginx service
const service = new k8s.core.v1.Service(name,
 {
 metadata: {
 labels: appLabels,
 namespace: namespaceName,
 },
 spec: {
 type: "LoadBalancer",
 ports: [{ port: 80, targetPort: "http" }],
 selector: appLabels,
 },
 },
 {
 provider: cluster.provider,
 },
);

// Export the Service name and public LoadBalancer Endpoint
export const serviceName = service.metadata.name;
export const serviceHostname = service.status.loadBalancer.ingress[0].ip;

apiVersion: apps/v1
kind: Deployment
metadata:
 Labels:
 namespace: helloworld
 labels: appLabels
spec:
 replicas: 1
 selector:
 matchLabels: appLabels
 template:
 metadata:
 labels: appLabels
 spec:
 containers:
 - name: helloworld
 image: nginx:latest
 ports:
 - containerPort: 80

Although this is a simple example, you can see where using a programming language has advantages over editing YAML files, e.g., you can edit the name, namespace, and labels just once.

The spec section of a Deployment specifies the desired state; in this example, we want one Replica and the selector must match the appLabel. The Deployment spec includes the application container and claims port:80 on the Pod. To make our deployment available outside the cluster, we create a LoadBalancer service that routes requests to the container. As with the Deployment, we add the Labels and Namespace to the metadata. The Service spec sets the type to LoadBalance and opens port 80 externally. Note that the selector uses appLabels to acquire the Pods from our Deployment.

...
const name = "helloworld";

// Create a Kubernetes Namespace
const ns = new k8s.core.v1.Namespace(name, {}, { provider: clusterProvider });

// Export the Namespace name
export const namespaceName = ns.metadata.name;

// Create a NGINX Deployment
const appLabels = { appClass: name };
const deployment = new k8s.apps.v1.Deployment(name,
 {
 metadata: {
 namespace: namespaceName,
 labels: appLabels,
 },
 spec: {
 replicas: 1,
 selector: { matchLabels: appLabels },
 template: {
 metadata: {
 labels: appLabels,
 },
 spec: {
 containers: [
 {
 name: name,
 image: "nginx:latest",
 ports: [{ name: "http", containerPort: 80 }],
 },
 ],
 },
 },
 },
 },
 {
 provider: clusterProvider,
 },
);

// Export the Deployment name
export const deploymentName = deployment.metadata.name;

// Create a LoadBalancer Service for the NGINX Deployment
const service = new k8s.core.v1.Service(name,
 {
 metadata: {
 labels: appLabels,
 namespace: namespaceName,
 },
 spec: {
 type: "LoadBalancer",
 ports: [{ port: 80, targetPort: "http" }],
 selector: appLabels,
 },
 },
 {
 provider: clusterProvider,
 },
);

// Export the Service name and public LoadBalancer endpoint
export const serviceName = service.metadata.name;
export const servicePublicIP = service.status.loadBalancer.ingress[0].ip;

apiVersion: apps/v1
kind: Deployment
metadata:
 Labels:
 namespace: helloworld
 labels: appLabels
spec:
 replicas: 1
 selector:
 matchLabels: appLabels
 template:
 metadata:
 labels: appLabels
 spec:
 containers:
 - name: helloworld
 image: nginx:latest
 ports:
 - containerPort: 80

Conclusion

This article reviews the Kubernetes objects and abstractions used to deploy applications. Deploying an application using Pulumi’s infrastructure as code looks similar to YAML manifests in how Deployments and Services are configured. The advantage of using a programming language to deploy an application is that you have the full range of software development tools available to you, such as defining variables, code completion, error checking, and versioning. In contrast, maintaining a set of YAML manifests can be an error-prone exercise in copying and pasting.

This is the second in a series of articles on using infrastructure as code for Kubernetes. In the next article, we’ll examine a complex application deployment and a deployment using a Helm chart. Until the next post, learn more about Kubernetes with these resources.

Watch educational content on Pulumi TV
Learn more about Pulumi’s support for Kubernetes
Practice Kubernetes Tutorials using Pulumi

Getting Started With Kubernetes: Clusters and Nodes

Sophia Parafina — Tue, 23 Jun 2020 00:00:00 +0000

Containers solved the problem of moving software from one environment to another because they encapsulate all the software dependencies. However, an orchestration platform is needed to manage containers at scale. Kubernetes is a popular open-source solution that uses declarative configuration to specify the desired state of the application. Configuring and deploying an application on Kubernetes is often accomplished with YAML files to define the state and command line tools to manage and control the Kubernetes API. This article demonstrates how to use infrastructure as code to create basic Kubernetes objects and higher-level abstractions that build upon the basic objects.

This is the first article of a multi-part series covering deploying Kubernetes and applications using infrastructure as code. We’ll take a top-down approach starting from clusters to high-level abstractions such as deployments. This article aims to provide a step-by-step example of deploying an application in Kubernetes using infrastructure as code. For this example, we’ll use Typescript for the programming language and provide sample code for AWS, Azure, GCP, and Kubernetes.

Clusters and Nodes

A cluster is formed with a control plane and a collection of nodes, the smallest unit of computing in Kubernetes. A node can be either a physical or virtual that contains the necessary components to run containers. A cluster has one or more nodes designated to the control plane, which controls the worker nodes where the application containers are deployed. A control plane node has:

the Kube-APIServer, which provides communication to the cluster
a Kube-Controller-Manager that provides governance for the cluster
etcd, which is the cluster state database
Kube-Scheduler, which schedules worker nodes based on events in etcd

To create a cluster for Kubernetes on a cloud provider, we need to create a VPC that will host the nodes required to deploy the Kubernetes cluster. Each cloud provider requires configuration specific to their implementation, and we’ll cover their particular requirements.

We declare a VPC to host our Kubernetes cluster and specify a public subnet, which is the gateway for the Kube-APIServer. We create the Kubernetes cluster with the VPC we declared and the VPC’s default public subnets. The desiredCapacity parameter sets the desired number of EC2 t2.medium nodes. We also export the kubeconfig file which we can use with kubectl to communicate with our Kubernetes cluster.

import * as awsx from "@pulumi/awsx";
import * as eks from "@pulumi/eks";

// Create a VPC for our cluster.
const vpc = new awsx.ec2.Vpc("vpc", { subnets: [{ type: "public" }] });

// Create the EKS cluster itself and a deployment of the Kubernetes dashboard.
const cluster = new eks.Cluster("cluster", {
 vpcId: vpc.id,
 subnetIds: vpc.publicSubnetIds,
 instanceType: "t2.medium",
 desiredCapacity: 2,
 minSize: 1,
 maxSize: 2,
});

// Export the cluster's kubeconfig.
export const kubeconfig = cluster.kubeconfig;

You can try out the code by following the complete tutorial.

In this example, we store the configuration variables in the Pulumi..yaml file, e.g.,

$ pulumi config set password [value]
$ pulumi config set location [value]
$ pulumi config set nodeCount [value]
$ pulumi config set nodeSize [value]
$ pulumi config set sshPublicKey [value]

We retrieve the values with config.ts for use in our main program.

config.ts

import * as azure from "@pulumi/azure";
import * as pulumi from "@pulumi/pulumi";

// Parse and export configuration variables for this stack.
const config = new pulumi.Config();
export const password = config.require("password");
export const location = config.get("location") || azure.Locations.EastUS;
export const nodeCount = config.getNumber("nodeCount") || 2;
export const nodeSize = config.get("nodeSize") || "Standard_D2_v2";
export const sshPublicKey = config.require("sshPublicKey");
export const resourceGroup = new azure.core.ResourceGroup("aks", { location });

In Azure, we have to create a service principal, which is an identity for accessing Azure resources. Note that in this example, the service principal password is stored as a secret in the project config file. Azure allocates a VPC when creating a Kubernetes cluster based on the values set in the config file. For example, the default number of nodes is two and uses Standard_D2_v2 virtual machine if these are not set in the config file. We also export the kubeconfig file which we can use with kubectl to communicate with our Kubernetes cluster.

cluster.ts

import * as azure from "@pulumi/azure";
import * as azuread from "@pulumi/azuread";
import * as k8s from "@pulumi/kubernetes";
import * as pulumi from "@pulumi/pulumi";
import * as config from "./config";

// Create the AD service principal for the K8s cluster.
const adApp = new azuread.Application("aks");
const adSp = new azuread.ServicePrincipal("aksSp", { applicationId: adApp.applicationId });
const adSpPassword = new azuread.ServicePrincipalPassword("aksSpPassword", {
 servicePrincipalId: adSp.id,
 value: config.password,
 endDate: "2099-01-01T00:00:00Z",
});

// Now allocate an AKS cluster.
export const k8sCluster = new azure.containerservice.KubernetesCluster("aksCluster", {
 resourceGroupName: config.resourceGroup.name,
 location: config.location,
 defaultNodePool: {
 name: "aksagentpool",
 nodeCount: config.nodeCount,
 vmSize: config.nodeSize,
 },
 dnsPrefix: `${pulumi.getStack()}-kube`,
 linuxProfile: {
 adminUsername: "aksuser",
 sshKey: {
 keyData: config.sshPublicKey,
 },
 },
 servicePrincipal: {
 clientId: adApp.applicationId,
 clientSecret: adSpPassword.value,
 },
});

// Expose a K8s provider instance using our custom cluster instance.
export const k8sProvider = new k8s.Provider("aksK8s", {
 kubeconfig: k8sCluster.kubeConfigRaw,
});

The code above is an abbreviated example of this AKS deployment on GitHub.

In this example, we store the configuration variables in the Pulumi..yaml file, e.g.,

$ pulumi config set password [value]
$ pulumi config set nodeCount [value]
$ pulumi config set nodeMachineType [value]
$ pulumi config set username [value]
$ pulumi config set masterVersion [value]

We retrieve the values with config.ts for use in our main program.

config.ts

import * as gcp from "@pulumi/gcp";
import { Config } from "@pulumi/pulumi";
import * as random from "@pulumi/random";

const config = new Config();

// nodeCount is the number of cluster nodes to provision. Defaults to 3 if unspecified.
export const nodeCount = config.getNumber("nodeCount") || 3;

// nodeMachineType is the machine type to use for cluster nodes. Defaults to n1-standard-1 if unspecified.
// See https://cloud.google.com/compute/docs/machine-types for more details on available machine types.
export const nodeMachineType = config.get("nodeMachineType") || "n1-standard-1";

// username is the admin username for the cluster.
export const username = config.get("username") || "admin";

// password is the password for the admin user in the cluster.
// If a password is not set, a strong random password will be generated.
export const password = config.get("password") || new random.RandomPassword(
 "password", { length: 20, special: true }).result;

// GKE master version
// Default to the latest available version.
export const masterVersion = config.get("masterVersion") ||
 gcp.container.getEngineVersions().then(it => it.latestMasterVersion);

We create the Kubernetes cluster using the variables we set in the configuration file. Note that for the nodes, we specify the oauthScopes, which are the Google API scopes available to all of the node VMs under the “default” service account. Because GKE uses gcloud to authenticate to the service, we have to create a kubeconfig file that uses gcloud. The kubeconfig file lets us communicate with our Kubernetes cluster.

import * as gcp from "@pulumi/gcp";
import * as k8s from "@pulumi/kubernetes";
import * as pulumi from "@pulumi/pulumi";
import { masterVersion, nodeCount, nodeMachineType, password, username } from "./config";

// Create the GKE cluster and export it.
export const k8sCluster = new gcp.container.Cluster("gke-cluster", {
 initialNodeCount: nodeCount,
 nodeVersion: masterVersion,
 minMasterVersion: masterVersion,
 masterAuth: { username, password },
 nodeConfig: {
 machineType: nodeMachineType,
 oauthScopes: [
 "https://www.googleapis.com/auth/compute",
 "https://www.googleapis.com/auth/devstorage.read_only",
 "https://www.googleapis.com/auth/logging.write",
 "https://www.googleapis.com/auth/monitoring",
 ],
 },
});

// Manufacture a GKE-style Kubeconfig. Note that this is slightly "different" because of the way GKE requires
// gcloud to be in the picture for cluster authentication (rather than using the client cert/key directly).
export const k8sConfig = pulumi.
 all([ k8sCluster.name, k8sCluster.endpoint, k8sCluster.masterAuth ]).
 apply(([ name, endpoint, auth ]) => {
 const context = `${gcp.config.project}_${gcp.config.zone}_${name}`;
 return `apiVersion: v1
clusters:
- cluster:
 certificate-authority-data: ${auth.clusterCaCertificate}
 server: https://${endpoint}
 name: ${context}
contexts:
- context:
 cluster: ${context}
 user: ${context}
 name: ${context}
current-context: ${context}
kind: Config
preferences: {}
users:
- name: ${context}
 user:
 auth-provider:
 config:
 cmd-args: config config-helper --format=json
 cmd-path: gcloud
 expiry-key: '{.credential.token_expiry}'
 token-key: '{.credential.access_token}'
 name: gcp
`;
 });

// Export a Kubernetes provider instance that uses our cluster from above.
export const k8sProvider = new k8s.Provider("gkeK8s", {
 kubeconfig: k8sConfig,
});

You can try out the code by following the complete tutorial.

Conclusion

This article covers how to set up a Kubernetes cluster on AWS, Azure, and GCP using Pulumi. Creating a cluster differs among cloud providers, but the process is generally the same. We defined configuration parameters such as node type, number of nodes, and passwords to instantiate the cluster then exported a kubeconfig file that we can use with kubectl.

This is the first in a series of articles on using infrastructure as code for Kubernetes. In the next article, we’ll cover basic Kubernetes objects such as pods, services, and volumes. We’ll also cover higher-level abstractions such as deployments and replicasets. Stay tuned! In the meantime, learn more about Pulumi:

Watch educational content on Pulumi TV
Learn more about Pulumi’s support for Kubernetes
Practice Kubernetes Tutorials using Pulumi