Pulumi Blog: Alex Clemmer

Introducing Pulumi Query for Kubernetes

Alex Clemmer — Wed, 20 Nov 2019 00:00:00 +0000

We often need answers to simple questions about Kubernetes resources. Questions like:

How many distinct versions of MySQL are running in my cluster?
Which Pods are scheduled on nodes with high memory pressure?
Which Pods are publicly exposed to the internet via a load-balanced Service?

Each of these questions would normally be answered by invoking kubectl multiple times to list resources of each type, and manually parsing the output to join it together into a single report.

Examples like these motivated us to build Pulumi Query for Kubernetes, which aims to greatly simplify scenarios like the ones mentioned above.

Pulumi Query is a tool and SDK for querying live Kubernetes resources. Pulumi Query supports both batch and streaming query modes. Batch queries run to completion, generating a fixed report. Streaming queries watch Kubernetes resources in real-time and take action when specific events occur. For example:

Post a slack notification when we run out of disk space.
Page someone when a rollout fails.

In this post, we’ll look at both batch and streaming query modes by showing examples of each. The Pulumi Query for Kubernetes SDK also contains a set of example queries and how to run them.

Batch queries

A simple example is a query program that finds all distinct versions of MySQL running in the default namespace of the active cluster of your $KUBECONFIG file.

The core primitive of batch queries is list. In this case, list("v1", "Pod") returns list of all Pods in the default namespace. The rest of the query “flattens” the list of Pods into a list of all container images, filters out all images that don’t contain the string “mysql”, and filters that to only distinct image names.

import * as kq from "@pulumi/query-kubernetes";

// Find all distinct versions of MySQL running in your cluster.
const mySqlVersions = kq
 .list("v1", "Pod")
 .flatMap(pod => pod.spec.containers)
 .map(container => container.image)
 .filter(imageName => imageName.includes("mysql"))
 .distinct();

mySqlVersions.forEach(console.log);

Query programs are run via the pulumi query command. Note that this feature is in beta, so to run it, you will need to prepend the command with the beta flag: PULUMI_EXPERIMENTAL=true pulumi query. The output looks like the following:

Batch query mode supports complex join operations via the join and groupJoin methods. This allows us to ask complex questions about how resources relate to each other, such as “find all Pods that reference some Secret.” For more examples, check out the examples in the SDK repository!

Streaming queries

The streaming query SDK continuously observes changes to any set of Kubernetes resources and takes action as a result.

The core primitive of the streaming SDK is watch. In the following example, we use watch to observe all Events related to a Deployment, which we then print to the console. Unlike list, which produces a set of live Kubernetes resources, watch publishes a stream of resource updates. Users get notified if a resource is ADDED, MODIFIED, or UPDATED.

Here is a simple example that watches for Events related to a deployment called alexbot

import * as kq from "@pulumi/query-kubernetes";

for await (const e of kq.watch("v1", "Event")) {
 const { apiVersion, kind, name } = e.object.involvedObject;
 if (
 apiVersion === "apps/v1" &&
 kind === "Deployment" &&
 name === "alexbot"
 ) {
 const { type, reason, message } = e.object;
 console.log(`${type} [${reason}] ${message}`);
 }
}

When run, we can see in real-time as the Deployment controller scales up the new ReplicaSet and scales down the old one.

For streaming queries over multiple resources, we expose a ResourceSet, which is a convenience data structure that simplifies the code. ResourceSet observes a sequence of resource updates published by watch, and uses those updates to keep track of which resources currently exist. It exposes an onUpdate function, which will run when this set is updated; i.e., whenever a resource is added, modified, or deleted from the set entirely.

In this case, when onUpdate is called, we run a query that filters the set of events down to those specifically having to do with a Deployment called alexbot. We can rewrite the previous query program to use ResourceSet in the following way:

import * as kq from "@pulumi/query-kubernetes";

new kq.ResourceSet([kq.watch("v1", "Event")]).onUpdate(([events]) => {
 events.filter(e => {
 const { apiVersion, kind, name } = e.involvedObject;
 return (
 apiVersion === "apps/v1" &&
 kind === "Deployment" &&
 name === "alexbot"
 );
 }).forEach(e => {
 const { type, reason, message } = e;
 console.log(`${type} [${reason}] ${message}`);
 });
});

Running this code with pulumi query results in an infinite stream of events telling us about this Deployment, i.e. when it’s rolling out, when it’s scaling down a ReplicaSet, and so on.

Conclusion

Pulumi Query provides powerful primitives for introspecting and reacting to changes in the state of Kubernetes resources. These primitives make it simpler to build tools that facilitate observability and helps to understand Kubernetes applications and to integrate event-based systems in response to them.

One of our hopes for Pulumi Query is that by building these simple integrations that it will lead to a groundswell of tooling that makes it easier to operate Kubernetes clusters by automating away tedious tasks that would normally be scripted. With this toolkit, teams can write linters, pre- and post-flight deployment checks, and to improve governance and security operations with minimal effort.

Using Helm and Pulumi to define cloud native infrastructure

Alex Clemmer — Wed, 31 Oct 2018 00:00:00 +0000

The Helm community is one of the brightest spots in the infrastructure ecosystem: collectively, it has accumulated person-decades of operational expertise to produce Kubernetes manifests that “just work.”

But for many users, it is not feasible to run everything in Kubernetes, and the community is just starting to develop answers to questions like: what happens when a Helm Chart needs to interface with, for example, a managed database like AWS RDS or Azure CosmosDB?

Pulumi is a cloud native development platform designed to be able to express any cloud native infrastructure as code in a natural, intentional manner using familiar languages. The most natural way to solve this challenge would be to stand up an instance of AWS RDS, populate a Kubernetes Secret with the connection details, and then simply let my application use these newly available resources. Pulumi gives users the primitives they need in order to achieve tasks like this most effectively.

How to connect a Kubernetes app with CosmosDB

In the following Pulumi program, we can manage both Azure and Kubernetes resources, including the interconnected dependencies between the two. Specifically, we:

Create an AKS cluster,
Create a MongoDB-flavored instance of Azure’s CosmosDB,
Create a Kubernetes Secret from the connection string exported by CosmosDB
Deploy a Node.js Helm Chart that references it.

If you have the Azure command line, try running the example with pulumi up!

import * as k8s from "@pulumi/kubernetes";
import * as azure from "@pulumi/azure";
import * as mongoHelpers from "./mongoHelpers";
import * as config from "./config";

// Create an AKS cluster.
import { k8sCluster, k8sProvider } from "./cluster";

// Create a MongoDB-flavored instance of CosmosDB.
const cosmosdb = new azure.cosmosdb.Account("cosmosDb", {
 kind: "MongoDB",
 resourceGroupName: config.resourceGroup.name,
 location: config.location,
 consistencyPolicy: {
 consistencyLevel: "BoundedStaleness",
 maxIntervalInSeconds: 10,
 maxStalenessPrefix: 200
 },
 offerType: "Standard",
 enableAutomaticFailover: true,
 geoLocations: [
 { location: config.location, failoverPriority: 0 },
 { location: config.failoverLocation, failoverPriority: 1 }
 ]);

// Create secret from MongoDB connection string.
const mongoConnStrings = new k8s.core.v1.Secret(
 "mongo-secrets",
 { data: mongoHelpers.parseConnString(cosmosdb.connectionStrings) },
 { provider: k8sProvider }
);

// Boot up nodejs Helm chart example using CosmosDB in place of in-cluster MongoDB.
const node = new k8s.helm.v2.Chart(
 "node",
 {
 repo: "bitnami",
 chart: "node",
 version: "4.0.1",
 values: {
 serviceType: "LoadBalancer",
 mongodb: { install: false },
 externaldb: { ssl: true, secretName: mongoConnStrings.metadata.apply(m => m.name) }
 }
 },
 { providers: { kubernetes: k8sProvider }, dependsOn: mongoConnStrings }
);

Pulumi supports deploying resources to all the major cloud vendors, as well as Kubernetes. Using the Pulumi programming model, it is possible to define and deploy apps and infrastructure with an arbitrary mix of resources from any combination of cloud providers. And, because Pulumi uses normal $KUBECONFIG files, it is compatible anywhere you would use Helm or kubectl.

The Pulumi CLI provides rich insight into the progress a deployment makes. When you run pulumi up, the CLI will provide detailed information about the progress we’re making as we try to deploy the Chart. To get a sense of what this looks like, take a look at the progress reported as we deploy the Wordpress Chart:

Toward Cloud Native Infrastructure as Code

As cloud native architectures mature, Pulumi can reduce the complexity in choosing and using the available services from cloud vendors, and combine those choices using a single, consistent programming model. This can make the best use of existing tools such as Helm, and also reduce the friction caused by multiple deployment tools and models across complex architectures.

Find out more about our Azure and Kubernetes support
Join the Slack community at https://slack.pulumi.com

How do Kubernetes Deployments work?

Alex Clemmer — Wed, 03 Oct 2018 00:00:00 +0000

This post is part 3 in a series on the Kubernetes API. Earlier, Part 1 focused on the lifecycle of a Pod and Part 2 focused on the lifecycle of a Service.

What is happening when a Deployment rolls out a change to your app? What does it actually do when a Pod crashes or is killed? What happens when a Pod is re-labled so that it’s not targeted by the Deployment?

Deployment is probably the most complex resource type in Kubernetes core. Deployment specifies how changes should be rolled out over ReplicaSets, which themselves specify how Pods should be replicated in a cluster.

In this post we continue our exploration of the Kubernetes API, cracking Deployment open using kubespy, a small tool we developed to observe Kubernetes resources in real-time.

Using kubespy trace, for example, we can observe at a high level what happens when Deployment rolls out a new version of an application:

But this post also comes with a twist, because in addition to being complex, Deployment is also the most dynamic resource in Kubernetes core. A Pod can crash or be killed. A node can disappear. A user can trigger a rollout. In each of these cases, some combination of the Deployment controller, the ReplicaSet controller, and the Kubelet have to cope.

In other words, understanding Deployment necessarily involves understanding how it does this coping. We therefore will also use kubespy trace to observe what happens when we kill and otherwise bother our Pods.

Following along

The kubespy repository comes with a simple example Kubernetes app, which is used in each of these examples. If you want to try it out for yourself the README contains detailed instructions.

You can use kubectl or pulumi — kubespy CLI itself is powered by the same code that underlies the core (OSS) Pulumi engine. If you like this, and would like to see information like it in CI/CD, we hope you’ll give it a shot! To get a flavor of what this looks like in practice, you might also check out my tweetstorm on the subject.

What happens during a rollout?

The gif in the introduction shows what happens when:

We deploy the example application.
Then, later, change the image tag to nginx:1.12-alpine and deploy again.

When kubespy trace deploy nginx is run, it will watch for changes to (1) the Deployment called nginx, (1) the ReplicaSets it controls, and (3) the Pods they control over time. The overall status is aggregated and printed to the console as they change.

From this gif, we can see 3 distinct phases:

First, the initial state: the Deployment is in a steady state. It controls a single ReplicaSet, which in turn controls three replicas of the application Pod. Each Pod is marked as available. We can see from Deployment’s status that the application is on revision 2, which means the application has been deployed twice — one initial deployment, and one update.

Second: the user submits a change to the Deployment, which triggers a rollout. The Deployment creates a new revision, revision 3. It creates a new ReplicaSet to represent this revision, and begins to start Pods with the new app version on it.

This gif is slowed down to show each of these individual changes — in the gif above, you can see these happen in very quick succession.

Third: the new version of the app comes online; the old ReplicaSet is killed. As the new Pods in revision 3 come online and are marked available, the Deployment controller begins killing off replicas from revision 2. Eventually it succeeds and the rollout is complete.

What happens if we kill a Pod?

Now that we understand the semantics of Deployment’s rollout, we can see what happens when we use kubectl delete pod <name> on one of the Pods that is controlled by our Deployment.

As expected, we can see that the ReplicaSet controller notices the Pod goes missing and spins up a new one. Note that it does not trigger a rollout, and does not increment the revision.

Notice also that the Pod that was destroyed hangs around for a few seconds, even though the new one has been booted up and the ReplicaSet has been marked available.

What happens if we add or remove labels from a Pod?

Try using kubectl edit to delete the labels on one of your Pods. kubespy trace will show you something the following:

Unlike the “killing a Pod” example above, the old Pod seems to disappear, replaced by a new Pod that, when booted up, causes the Deployment to be marked as available again.

What’s happening here? It turns out that if you remove the app labels for a Pod, the ReplicaSet controller notices, removes itself from .metadata.ownerRef, and then treats the Pod as if it’s been deleted, spinning up a new one immediately.

This is useful: if you notice one of your Pods is behaving strangely, you can simply take it out of rotation by removing those labels, so that you can pull it aside and test it.

Conclusion

The Kubernetes API is rich, packed with useful information about the state of your cluster. It’s remarkable how little it is directly used to make tools. Combine a little knowledge of Deployment’s .status field with the Kubernetes Watch API and a bit of terminal UI programming, you’re most of your way to kubespy trace deployment.

If you enjoyed this post, or are curious to see how this lifecycle is baked into the Pulumi CLI, please give us a shot! We’d love to hear your feedback.

kubespy trace: a real-time view into of a Kubernetes Service

Alex Clemmer — Wed, 26 Sep 2018 00:00:00 +0000

This post is part 3 in a series on the Kubernetes API. Earlier, Part 1 focused on the lifecycle of a Pod, and later Part 3 details how Kubernetes deployments work.

Why isn’t my Pod getting any traffic?

An experienced ops team running on GKE might assemble the following checklist to help answer this question:

Does a Service exist? Does that service have a .spec.selector that matches some number of Pods?
Are the Pods alive and has their readiness probe passed?
Did the Service create an Endpoints object that specifies one or more Pods to direct traffic to?
Is the Service reachable via DNS? When you kubectl ``exec into a Pod and you use curl to poke the Service hostname, do you get a response? (If not, does any Service have a DNS entry?)
Is the Service reachable via IP? When you SSH into a Node and you use curl to poke the Service IP, do you get a response?
Is kube-proxy up? Is it writing iptables rules? Is it proxying to the Service?

This question might have the highest complexity-to-sentence-length ratio of any question in the Kubernetes ecosystem. Unfortunately, it’s also a question that every user finds themselves asking at some point. And when they do, it usually means their app is down.

To help answer questions like this, we’ve been developing a small diagnostic tool, kubespy. In this post we’ll look at the new kubespy trace command, which is broadly aimed at automating questions 1, 2, 3, and providing “hints” about 4 and 5.

Below is a gif demonstrating the CLI experience. You can watch in real-time as the Service comes online, finds pods to target, and finally is allocated a public IP address:

What is kubespy, again?

kubespy is a simple, standalone diagnostic tool, meant to make it easy to introspect on Kubernetes resources in real time.

Before we begin, it’s worth noting that this kubespy actually re-packages the machinery we developed for Kubernetes support in Pulumi.

One of our major goals in this work was to make deploying an application to Kubernetes as simple as possible, by presenting a concise summary of this information in the CLI experience. See my tweetstorm on the subject, or try it out for yourself!

A real-time view of a Service’s life

The kubespy repository contains the simple trace example we use in this demo. The README contains detailed installation instructions, as well as explaining how to run the app (using either kubetl or pulumi though of course we hope you will try Pulumi).

Essentially: running kubespy trace service nginx will cause kubespy to sit and wait for you to deploy a Service called nginx. When you run this example, it will do just this: creating a Deployment which replicates an nginx Pod 3 times and exposes it publicly to the Internet with a Service, also called nginx.

Let’s break down the kubespy trace gif above to show that there are actually several distinct steps in the process of booting up a Service.

First: Service is created, the Service controller creates an Endpoints object of the same name. The Endpoints object is to specify which Pods get traffic — their IPs, which ports to direct traffic to, and so on. In this case, there are no Pods to target, which kubespy trace tells us:

Second: Pods that match the Service’s .spec.selector are created; their readiness probes immediately pass. The Endpoints object is updated to reflect this. As we will see below, if the Pods failed the readiness probes, kubespy trace would note this.

Third: Service is allocated a public IP address. The Service has .spec.type set to LoadBalancer, which on most cloud platforms means that a public IP address should be allocated for it.

Exercise: Other Service types, watching rollouts, deleting Services!

kubespy trace supports all the other Service types, including ExternalName and ClusterIP. Try both of those, and you’ll see slightly different output. Try them! It’s also worth watching what happens when a Service is deleted.

You can also use kubespy trace to watch an unhealthy deployment become healthy. In the following gif, we see a bunch of Pods that are failing readiness checks become healthy as a new version is rolled out:

Conclusions

Confession time. Last time we told you we’d dig more into the lifecycle of a Pod. And we will, at some point. But we ended up deciding that it would be easier to explain with a cohesive trace command.

And, while this is a good start, it is only the beginning. trace currently supports only Service. In our next post, we’ll extend trace to Deployment (or perhaps ReplicaSet), and from there, we will have enough tools to really dig into what is happening when you roll out your app.

In the mean time, if you enjoyed this post, or are curious to see how this lifecycle is baked into the Pulumi CLI, give it a spin! We’d love to have your feedback.

Kubespy, and the lifecycle of a Pod, in 4 images

Alex Clemmer — Tue, 18 Sep 2018 00:00:00 +0000

This post is the first part in a series on the Kubernetes API. Future installments include Part 2 focused on the lifecycle of a Service and Part 3 details how Kubernetes deployments work.

One of the most popular features of the recent v0.15.2 release of Pulumi is fine-grained status updates for Kubernetes resources. On the CLI they look like this:

But wait — how does this work exactly? What is actually happening when you deploy a Pod to a cluster? How does a Service go from uninitialized, to being allocated a public IP address? How often is my Deployment’s status changing?

To answer these questions and others like them, today we’re releasing a small tool, kubespy, purpose-built to help you answer questions like this, by displaying the changes made to a Kubernetes object in real time. For example, in the following gif, we’re running kubespy status v1 Pod nginx to watch the changes to a Pod’s status as it is booted up.

We will spend the rest of this short post using kubespy to take a closer look at what happens to a Pod when it’s deployed to the cluster.

What happens when you boot up a Pod?

kubespy comes with a simple example that deploys nginx using a naked Pod. (The example works with both the Pulumi CLI and kubectl, but we hope you’ll give Pulumi a shot!)

The essential gist is that kubespy will sit and watch for the example Pod to be created or changed; once you deploy the Pod (either pulumi up or kubectl apply), you will see something like the gif above.

If you do this yourself, you will see that there are 4 updates reported by kubespy, each of which individually gives us a fairly clear picture of what Kubernetes is doing internally to try to run the Pod:

First: Acknowledging the Pod definition is written to etcd. The API server receives the Pod definition and begins trying to schedule it.

Second: Scheduling the Pod. The scheduler successfully schedules the Pod to run on a node.

Third: Creating the Pod. The kubelet running on the node receives the Pod definition and begins “creating” the Pod. This involves pulling the container image, adding volume mounts, etc. At the end of all of this, it will attempt to run the container.

Fourth: Marking the Pod as Running. The kubelet has now resolved the image tag to a specific SHA, pulled it to the node, has successfully applied the required configuration, and has successfully started the nginx container. In this case, the Pod contains only one container to run, and once it’s initialized, the whole Pod gets marked running.

Exercises:

Try inducing an error and see what kubespy outputs! Make the image tag nonsense, add a volume that doesn’t exist, and so on.
Try booting up a service! See what kubespy displays under various configurations!

Conclusions, and the subtleties of the Pod API.

This post shows a fairly basic, happy-path use of Pods, and shows that we can kubespy status to see exactly what our containers are up to.

But wait — what does Running mean, precisely? If a Pod is Running, does that mean it’s healthy? If a Pod has multiple containers, and one crashes, does it still get marked Running? Are there any other settings that can change when a Pod is marked Running?

Unfortunately, though the Pod is the atomic compute abstraction of Kubernetes (or perhaps because of this), the answer to these questions turns out to be fairly subtle. But, because Pods are so important, we should endeavor to answer them anyway!

In the next post, we will kubespy to observe some of the subtleties of the Pod lifecycle, and especially the effect of restartPolicy on the semantics of the Pod object. We will also talk about some of the best practices we’ve accumulated, and describe how we solved the problem of repeatable Pod deployments in Pulumi.

If you enjoyed this post, or are curious to see how this lifecycle is baked into the Pulumi CLI, give us a shot! We’d love to have your feedback.

Simple, Reproducible Kubernetes Deployments

Alex Clemmer — Fri, 24 Aug 2018 00:00:00 +0000

Kubernetes is a powerful container orchestrator for cloud native applications that can run on any cloud – AWS, Azure, GCP – in addition to hybrid and on-premises environments. Its CLI, kubectl, offers basic built-in support for performing deployments, but intentionally stops short here. In particular, it doesn’t offer diffs and previews, the ability to know when a deployment has succeeded or failed, and why, and/or sophisticated deployment orchestration.

In this post, we’ll see how Pulumi, an open source cloud native development platform, can not only let you express Kubernetes programs in familiar programming languages, like TypeScript, instead of endless YAML templates, but also how Pulumi delivers simple and reproducible, yet powerful, Kubernetes deployment workflows.

Less YAML, More Robustness

Pulumi 0.15 introduced Kubernetes support to Pulumi. When we started working on this, we already loved being able to write in our favorite languages – with the benefits of IDEs, classes and functions, and reuse through packages – instead of YAML. But we felt just having great programming language alone wasn’t enough.

As we looked to how teams wanted to operationalize our Kubernetes support, we set our sights on one major area: delivering simple, reproducible application deployment workflows. When an app attempts to roll out, it should be obvious if it succeeded. If the rollout failed, on the other hand, errors should be clear and useful enough that most problems are simple to fix – and resuming where you left off after fixing should be similarly just as easy. Finally, all of this should be repeatable so that CI/CD is robust in production team environments.

Like many others, we’ve used kubectl apply to dump a pile of YAML into the API server. The command returns instantly, but what happens next? Is the app up? How does this tie back into CI/CD, so others on the team know what’s up with a deployment? Did something fail, or get stuck?

The vision in our heads, which we set out to build, was something more like this:

In this example, we can clearly see the entire set of resource objects being created, their ongoing status, and when and why something might have failed (in this case, a missing Docker image). This is a real screengrab of our CLI.

The service works in tandem with the CLI, so we always have a history and record of successful or failed deployments:

Let’s Deploy Some Code!

In this post, we’ll get a taste of this and several other aspects of the Pulumi workflow by deploying a very simple app – an nginx web server – to a Kubernetes cluster.

We’ll show how to use the Pulumi update command to get real-time information about a deployment’s progress. We’ll see how to use Pulumi’s notion of “stack outputs” to present information about successful deployments to the user – e.g. the public IP address allocated to an application after the load balancer spins up. Finally, we’ll see how to use Pulumi’s diff command to reason about the blast radius of an update to the application, before even attempting it.

If you’d like to follow along, you can find the code here. In the README, you’ll find instructions for the prerequisites – installing Pulumi, setting up a local or remote Kubernetes cluster, etc.

Application Config-as-Code

Our first task in our journey is to write a Kubernetes application that deploys nginx to the cluster and exposes it publicly to the Internet. Kubernetes ships out of the box with useful APIs for deploying applications, managing incremental rollouts of changes, and specifying how traffic is directed, all of which we will use in this example.

The code below as a slightly modified version of the “Hello World” Deployment example from the Kubernetes docs. We have chosen to write this Pulumi program in TypeScript, an excellent choice for a mix of dynamic productivity, static typing (meaning we will find errors sooner, often at compile time!), and provides instant access to the NPM ecosystem.

Note: It is also possible to directly deploy existing Kubernetes YAML objects, as a stepping stone. See the example here. This lets you reap all of the benefits of the deployment workflow called out in this post without needing to rewrite all of your Kubernetes YAML. It’s easy to then rewrite it piecemeal, one step at a time, after getting up and running.

This application has two major pieces:

The Deployment, which takes a template for an application, and then instantiates some user-specified number of copies (or replicas) of that template in the cluster.
The Service, which we will use to allocate a publicly-reachable IP address, and to direct traffic.

The code below from index.ts defines our application using these two APIs:

// nginx container, replicated 3 time.
const appName = "nginx";
const appLabels = { app: appName };
const nginx = new k8s.apps.v1beta1.Deployment(appName, {
 spec: {
 selector: { matchLabels: appLabels },
 replicas: 3,
 // The application template to replicate -- just the nginx container.
 template: {
 metadata: { labels: appLabels },
 spec: { containers: [{ name: appName, image: "nginx:1.15-alpine" }] }
 }
 }
});

// Allocate an IP to the nginx Deployment.
const frontend = new k8s.core.v1.Service(appName, {
 metadata: { labels: nginx.spec.apply(spec => spec.template.metadata.labels) },
 spec: {
 // Type `LoadBalancer` causes us to allocate a public IP address.
 type: "LoadBalancer",
 ports: [{ port: 80, targetPort: 80, protocol: "TCP" }],
 selector: appLabels
 }
});

This code creates 3 replicas of the nginx container, puts them behind a load balancer listening on port 80, and exposes it publicly to the Internet.

Right away, we notice:

Pulumi does not define a new API for Kubernetes; it simply uses the same schema as the upstream Deployment and Service APIs. In fact, it is generated from the Kubernetes OpenAPI specification, so it is always up-to-date and complete.
By using TypeScript, we have the ability to eliminate repetition and boilerplate with familiar programming language constructs, like we did with our appLabels variable that is reused multiple times. Another example is the use of a function to capture common patterns, as can be seen in our port of the ever-familiar guestbook application.

Deploying the Application

If we were deploying normal Kubernetes YAML, we’d run kubectl apply. This would return instantly, and it would be up to us to use ancillary commands (e.g. kubectl get service) to figure out if the application succeeded.

The Pulumi equivalent is to run pulumi up. This evaluates our program, figures out the diff and incremental update plan, shows us a preview of what it will do, and ultimately carries it out. Under the hood, this is using the Kubernetes Go client library, and – although we will still use kubectl to inspect and interact with cluster state, the entire deployment process is subsumed by pulumi.

Indeed, if we take our project and simply run pulumi up we’ll see something like the following:

Compared to kubectl apply, many things are happening:

We see that pulumi up blocks until all Kubernetes resources have completed initialization. In this case, that includes a Deployment and a Service.
We see intermediate status messages as Kubernetes resources make progress towards initialization. For example the Service called nginx alerts us when (1) it successfully found application containers to direct traffic to, and (2) that an IP address has been allocated to it. If things fail, we will see that too.
All deployment is coordinated with a state manager so that concurrent updates are handled correctly in a team setting, rather than potentially clobbering one another.
If you look closely, at the end of the gif loop, you’ll notice some green text that looks like this:
```
 ---outputs:---
frontendIp: "35.226.79.25"
```
This is the IP that was allocated to the Service, which in turn directs traffic to our nginx containers. If you run pulumi up yourself and open that IP address in your browser, easily available with the pulumi stack output frontendIp command, you will actually see the nginx landing page.

Using Stack Outputs

As a brief aside, you might have wondered: how did this IP address get here?

When kubectl apply completes, it is difficult to automatically obtain values like the IP address allocated to a Service. It can take many minutes for the IP address to be allocated, and most users simply resort to running kubectl get service repeatedly, until it appears. This is cumbersome and makes it hard to use reliably.

In Pulumi, such values are first-class citizens – called stack outputs. Because Pulumi has a notion of “done-ness”, and because pulumi up will block until all resources are finished initializing, it is trivial to “export” values from a fully-initialized resource. In this case, frontendIp simply contains the IP address obtained from the fully-initialized Service that directs traffic to the nginx containers – the IP address allocated to it.

We could take any value from any Kubernetes resource and put it in any variable – in this case, we just happen to have exported the IP address using a variable frontendIp. The code to do this export looks like this (at the bottom of index.ts):

export const frontendIp = frontend.status.apply(
 status => status.loadBalancer.ingress[0].ip);

Obtaining this value from the CLI is trivial using the pulumi stack output command. Here we use curl to get find the title of the nginx landing page that frontendIp points at.

$ curl -sL $(pulumi stack output frontendIp) | grep "<title>"
<title>Welcome to nginx!</title>

Updating our Application

Pulumi makes it easy to incrementally update our application too, after the initial creation. pulumi up just handles this automatically for us, by comparing its notion of a goal state to the cluster’s current state, and using that to devise a plan. Because of this, we get reliable preview diffs.

When running kubectl apply, it is often difficult to reason about what changes will be made to a running application, before actually carrying out the changes. This makes it difficult to predict the impact an update will have, including whether there would be potential downtime. Put simply, this can lead to accidents.

In contrast, in Pulumi the idea of a diff is a first-class concept, and is front and center in the workflow. To see this in action, try changing the container image from nginx:1.15-alpine to nginx:alpine-1.16-alpine in index.ts, and then run pulumi preview --diff. You will see something like this:

You can see several things happening here:

The diff captures the change in the nginx image.
It reports that the nginx Deployment resource will be updated in-place. Replaces may also be replaced or deleted, which would be evident from the diff also.

The intention here is to give users an intuition for the blast radius of the changes to a Kubernetes application.

Running diffs manually by hand isn’t necessary, despite being useful. Unless you explicitly bypass it, running pulumi up will always show you a preview first and confirm that you’d like to proceed before actually making any changes. And because Pulumi is always tracking old states, rollback afterwards is easy also.

This CLI workflow is great for the dev inner loop, however the Pulumi GitHub App also integrates these previews and diffs with your CI/CD system, by enlightening your GitHub Pull Requests with potential update impacts, while your team still has a chance to discuss changes inside the usual PR workflow, enabling powerful GitOps scenarios.

Conclusion

In this article, you’ve seen that you can express Kubernetes apps in real code, instead of YAML, and you’ve gotten a taste of the pulumi up deployment workflow, especially compared to kubectl.

Try it out now by heading over to our Getting Started page.

We are just getting started. In the coming weeks we will be sharing more around other deployment scenarios, including A/B traffic splitting, canaries guarded by Prometheus metrics, Kubernetes application code living alongside AWS infrastructure, triggering of cascading rollouts based on ConfigMap changes… And much, much, more.

If you have specific use cases you’d like to see us tackle, don’t hesitate to reach out, either on GitHub or in our Pulumi Community Slack. We’d love to hear from you!