Pulumi Blog: Sean Gilespie

Pulumi and Docker: Development to Production

Sean Gilespie — Wed, 15 May 2019 00:00:00 +0000

Here at Pulumi, we’re big fans of Docker: at this point there is little doubt that Docker has completely revolutionized the way that we think about deploying software. However, even in 2019, it’s quite difficult to get Docker containers to production. docker run is great, and we all love it, but unfortunately it’s quite a big leap from docker run to running your container in a production-ready environment.

We recently wrote a blog post about using AWS Fargate to run your docker containers with our open source packages. In this blog post we’re going to focus on another interesting aspect of Pulumi: being able to re-use your infrastructure code for both development and production!

It’s All Code: Composing Docker Containers

In addition to the number of cloud and infrastructure providers that Pulumi supports, Pulumi also supports defining Docker resources in code. Let’s look at this code snippet of Pulumi TypeScript code:

// This program encodes a complete application: a container running Redis Commander,
// and a container running Redis. Redis Commander connects directly with Redis.
//
// In order for the two containers to communicate directly, they must be placed in the
// same network. Before creating the containers themselves, we first create the network
// that will contain both of them.
const network = new docker.Network("net");
// Also before creating a container, we must first obtain a "RemoteImage", which is a reference to an external image
// that is downloaded to the local machine. In this case, we're referring to an image on Docker Hub.
const redisImage = new docker.RemoteImage("redis-image", {
name: "redis:latest",
keepLocally: true, // don't delete the image from the local machine when deleting this resource.
});
// We can create a container using a reference to the name of the image we just downloaded and a reference to the name
// of the network that this container should use.
const redisContainer = new docker.Container("redis", {
image: redisImage.name,
networksAdvanced: [{ name: network.name }],
restart: "on-failure",
});
// We do the same thing for the Redis Commander container.
const redisCommanderImage = new docker.RemoteImage("redis-commander-image", {
name: "rediscommander/redis-commander:latest",
keepLocally: true,
});
const redisCommanderContainer = new docker.Container("redis-commander", {
image: redisCommanderImage.name,
restart: "on-failure",
networksAdvanced: [{ name: network.name }],
envs: [
// Pulumi resources have "output" properties, which are properties that are set upon successful completion of
// the resource operation. These "outputs" are like promises - they can't be used directly but instead must be
// manipulated through use of the `apply` function.
//
// Here, we are using the name of the Redis commander to build an environment variable `REDIS_HOST` that points
// to the network identity of the Redis container we just created.
redisContainer.name.apply(name => `REDIS_HOST=${name}`),
],
// Finally, we expose some ports. Redis Commander listens on port 8081, so we'll map that to external
// port 3000 for easy consumption on the user machine.
ports: [{
internal: 8081,
external: 3000,
}]
});

Here we are looking at a complete Pulumi program that does a bunch of things with Docker:

We create a Docker network named net, which we’ll use to connect two containers together
We pull the redis:latest image from Docker Hub and create a container using it, named redis, which we attach to the network we just created
We pull the rediscommander/redis-commander:latest image from Docker Hub and also create a container using it, named redis-commander, which likewise is attached to the network and exposes some ports on the host machine.

This is fairly nontrivial, but we’ve done it in code right here. Using code we’ve replicated many of the features offered by tools like docker-compose, but without writing a single line of YAML. Running pulumi up with this code results in a working instance of Redis Commander listening on port 3000 on your machine. Nice and simple!

Moving to Production

Orchestrating Docker containers on your local machine while developing is great, but when it comes time to pushing containers to production, you can’t rely on Docker alone anymore. There are a number of excellent container orchestrators out there but there few are as successful as Kubernetes. The way that you launch production workloads in Kubernetes is significantly different than how you would do it using docker run on your own, so it’s often the case that you’d need to write another series of YAML documents to get your application into production.

With Pulumi, we’re just writing code. Since we’re just writing code, we can do exactly what we would do if we were writing application code: abstract!

// Since we're using a general-purpose programming language, we can now use that language to provide multiple
// implementations of the same abstraction. In this case, we define an abstract "Redis" component whose only property is
// that it has some network identity (host) that consumers can connect to and talk to a Redis connection.
export abstract class Redis extends pulumi.ComponentResource {
 public abstract readonly host: pulumi.Output<string>;

 public static create(name: string, args: RedisArgs, opts?: pulumi.ComponentResourceOptions): Redis {
 // Since we are using TypeScript and RedisArgs is the union of argument types for each resource. Since args.type
 // has type "amazon" | "docker" | "kubernetes", the compiler assists us in making sure that we don't
 // accidentally mix up argument types.
 switch (args.type) {
 case "amazon":
 return new AmazonRedis(name, args, opts);
 case "docker":
 return new DockerRedis(name, args, opts);
 case "kubernetes":
 return new KubernetesRedis(name, args, opts);
 }
 }

 constructor(ty: string, name: string, args: RedisArgs, opts?: pulumi.ComponentResourceOptions) {
 super(ty, name, args, opts);
 }
}

export type RedisArgs = DockerRedisArgs | KubernetesRedisArgs | AmazonRedisArgs;

We’ve departed a bit from our previous example, but we’re still writing code. In particular, what we’re doing here is defining a new, abstract, Redis component that takes the Redis container from the previous example and abstracts it away a little bit. This class we’ve created still represents Redis, but completely abstracts away where and how the Redis instance is running. In this case, it can run in any one of three different configurations:

Redis backed by AWS ElastiCache (“amazon”)
Redis running as a Docker container on your machine (“docker”)
Redis running as a Deployment on a Kubernetes cluster (“kubernetes”)

Now, you can use this Redis component completely in the abstract, like this:

const redis = Redis.create("redis", {
 type: "docker",
 network: network,
});

We’re requesting a “docker” container, but we could just as easily switch this out for “amazon” and “kubernetes”, and suddenly our Redis workload is transparently deployed in different ways, all without disrupting our development environment!

Pulumi: Infrastructure as Software

We often say on this blog that Pulumi is “infrastructure as code”, and that sentence is definitely true, but I prefer calling it “infrastructure as software” instead. The distinction is subtle, but in my mind Pulumi brings to the realm of infrastructure the variety of tools we already use for software engineering:

Abstraction, encapsulation, and code reuse for infrastructure and applications
Testing, both unit and integration
IDEs and tools for detecting errors extremely early in a developer’s inner loop, instead of at deployment time

Pulumi is open source, free to use, and works today with a variety of clouds and bring a little more software and less code into your infrastructure! If you’d like to see more about this particular code demo, check out my DockerCon EU 2018 talk where I dive into this in detail. The code for this post and the talk are in this repository.

Programming the Cloud with Python

Sean Gilespie — Thu, 04 Apr 2019 00:00:00 +0000

Across the industry, the popularity of Python is exploding. Amongst our own customers at Pulumi, who automate their infrastructure using Python, we’ve seen the same. Stack Overflow wrote about the astounding growth of Python:

The term “fastest-growing” can be hard to define precisely, but we make the case that Python has a solid claim to being the fastest-growing major programming language.

– David Robinson, Stack Overflow

Since Python is not a new language, what could be driving this incredible adoption curve?

Python is, and always has been, particularly amenable to the automation of previously-manual tasks. So much so, in fact, that books have been written on how to automate common tasks with Python. In recent years, Python’s penchant for task automation have led to its use at the foundation of tools now ubiquitous in the DevOps space: tools like Ansible, SaltStack, and OpenStack are all authored primarily in Python. It would not be a stretch to claim that Python is the language of automation.

At Pulumi we are passionate about automating cloud infrastructure. However, we see a major open opportunity to leverage our most powerful, fundamental tool - the programming language - to automate our infrastructure tasks. We love the idea of “infrastructure of code”, but we struggle with the industry’s definition of “code”; it’s hardly code at all, often consisting of YAML documents, DSLs, or other, domain-specific solutions to the task at hand. Since Python is such an incredible tool for task automation and since its use is rapidly pervading the industry, wouldn’t it make sense to use code, to use Python, to automate tasks as critical as the provisioning of our cloud infrastructure and delivery of our applications?

Delivery is not a detail, it is our job. Now is the time to apply our core skills to our own work. Now is the time to engineer our delivery. We divide our work between ourselves and computers: humans for decisions, and automation for tasks.

– The Software Defined Delivery Manifesto

That sounds great, but what is this “Pulumi” thing?

Pulumi is an infrastructure as code tool for provisioning cloud infrastructure and deploying applications. The key difference between Pulumi and other tools in this space (such as Ansible, Terraform, and AWS CloudFormation) is that Pulumi aggressively reclaims the word “code”: users of Pulumi write programs in general purpose programming languages that, when run, deploy and run applications in cloud environments, while still retaining all the known benefits of infrastructure as code that our industry has grown accustomed to (repeatability, resilience to failure, history and auditing, and so on).

Pulumi is a rebellion against YAML and domain-specific non-Turing-complete languages. We firmly believe that your infrastructure isn’t just code: your infrastructure is software. By applying the same engineering discipline to your infrastructure code that you apply to your application code, we can iterate quickly and with confidence, leaning on things such as automated tests, code analysis tools, and a common language. Pulumi does not generate YAML; it instead brings the full API surface area of your cloud environment into Python, powered by an infrastructure as code engine.

It is hard to overstate how liberating it is to write infrastructure as code with a full general-purpose programming language at your disposal. Below is a code snippet of a Pulumi program, written in Python, that deploys a static website to S3:

import mimetypes
import os

from pulumi import export, FileAsset
from pulumi_aws import s3

web_bucket = s3.Bucket('s3-website-bucket', website={
    "index_document": "index.html"
})

content_dir = "www"
for file in os.listdir(content_dir):
    filepath = os.path.join(content_dir, file)
    mime_type, _ = mimetypes.guess_type(filepath)
    obj = s3.BucketObject(file,
        bucket=web_bucket.id,
        source=FileAsset(filepath),
        content_type=mime_type)

There are only 18 lines of Python code here, but this program does a ton!

First, we create an AWS S3 Bucket that we’ll use to host our website from, setting the index document to be “index.html”.
Next, we iterate over all files in the “www” directory (using the standard Python for loop!
Finally, for every file in the “www” directory, we guess its MIME type using the Python standard library and then create an S3 BucketObject inside the website bucket, which uploads the file to S3.

This is a simple example, but it’s hard to overstate how awesome it is to be able to reach into Python’s deep standard library whenever we need to. In this example we’re using both the “os” and “mimetypes” packages in the Python standard library: the first to list the files in a particular directory and the second to guess a particular file’s MIME type based on its extension and contents, which we eventually pass directly to S3.

Remember, this code isn’t performing imperative commands — like, say, your cloud provider’s Python SDK or Boto would — and instead creates a declarative resource graph that the Pulumi engine understands how to act upon. This works for the first deployment, but more importantly, for all subsequent incremental updates too. If we were to change the contents of one of the files on disk, Pulumi would recognize that one of the files has changed and re-upload it, without changing the bucket or any of the other already existing bucket objects!

Where can I learn more?

Don’t take our word for it, though! We’d love for you to check it out and see for yourself how great it is to reclaim the “code” in “infrastructure as code”.

If you want to know more about Pulumi and the things it can do for you, check out the Getting Started page and Documentation for more information.

If you’d like to try the above example out for yourself, we’ve put the full code on GitHub. There are also dozens of other Python examples across AWS, Azure, GCP, Kubernetes and more you can explore.

Python is the language of automation today and for the future. Pulumi is the infrastructure as code automation tool of the future. Using them both together is an incredible way to automate a crucial job while staying entirely within a language you already know and love!

How to deploy Jenkins to Kubernetes with Pulumi

Sean Gilespie — Wed, 27 Jun 2018 00:00:00 +0000

In this blog post, we’ll see how to run Jenkins on Kubernetes with Pulumi. We abstract away the complexity of the configuration into a reusable component to enable it to be reused by team mates. Our example is based on adapting a Helm template.

Reusing the Jenkins component

Once the component is created, it can be reused with just a few lines of implementation code.

import * as pulumi from "@pulumi/pulumi";
import * as jenkins from "./jenkins";

const config = new pulumi.Config("jenkins");
const instance = new jenkins.Instance("jenkins", {
 name: "jenkins",
 credentials: {
 username: config.require("username"),
 password: config.require("password"),
 },
 resources: {
 memory: "512Mi",
 cpu: "100m",
 }
});

We could take the component code and prepare a fully reusable NPM package, but as this is just an example, we’ve simply maintained the code as an included class.

Running the App

Follow the steps in Pulumi Installation and Setup and Configuring Pulumi Kubernetes to get setup with Pulumi and Kubernetes.

Note: The code in this repo assumes you are deploying to a cluster that supports the LoadBalancer service type. This includes most cloud providers as well as Docker for Mac Edge w/ Kubernetes. If not (for example if you are targeting minikube or your own custom Kubernetes cluster), replace type: "LoadBalancer" with type: "ClusterIP" in jenkins.ts. See the Kubernetes Services docs for more details.

Clone the repo and navigate to the Jenkins example:

$ git clone https://github.com/pulumi/examples.git
$ cd kubernetes-ts-jenkins

Install dependencies:

$ npm install

Create a new stack:

$ pulumi stack init
Enter a stack name: kubernetes-ts-jenkins-dev

Create configuration keys for the root username and password for the Jenkins instance we are about to create:

$ pulumi config set jenkins:username <your desired username>
$ pulumi config set jenkins:password <your desired password> --secret

Preview the deployment of the application:

$ pulumi preview
Previewing update of stack 'kubernetes-ts-jenkins-dev'
Type Name Status Info
* global global unchanged
Previewing changes:
Type Name Status Info
* global global no change
+ pulumi:pulumi:Stack kubernetes-ts-jenkins-kubernetes-ts-jenkins-dev create
+ └─ jenkins:jenkins:Instance jenkins create
+ ├─ kubernetes:core:v1:Secret jenkins-secret create
+ ├─ kubernetes:core:v1:PersistentVolumeClaim jenkins-pvc create
+ ├─ kubernetes:core:v1:Service jenkins-service create
+ └─ kubernetes:extensions:v1beta1:Deployment jenkins-deploy create
info: 6 changes previewed:
+ 6 resources to create

Perform the deployment:

$ pulumi update --skip-preview
Updating stack 'kubernetes-ts-jenkins-dev'
Type Name Status Info
* global global unchanged
Performing changes:
Type Name Status Info
* global global unchanged
+ pulumi:pulumi:Stack kubernetes-ts-jenkins-kubernetes-ts-jenkins-dev created
+ └─ jenkins:jenkins:Instance jenkins created
+ ├─ kubernetes:core:v1:Secret jenkins-secret created
+ ├─ kubernetes:core:v1:PersistentVolumeClaim jenkins-pvc created
+ ├─ kubernetes:core:v1:Service jenkins-service created
+ └─ kubernetes:extensions:v1beta1:Deployment jenkins-deploy created
info: 6 changes performed:
+ 6 resources created
Update duration: 2m30.397621136s

The deployment is complete! Use kubectl to see the Service that we just deployed:

$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
jenkins LoadBalancer 10.43.241.251 35.230.56.127 80:30638/TCP,443:30204/TCP 3m

The Jenkins instance we just deployed is reachable through port 80 of the external IP address. You can now visit http://35.230.56.127/jenkins/login in a Web browser to begin the first-install flow for your new Jenkins instance. You can use the username and password that you saved in your Pulumi config to log in to your new Jenkins instance.

Note: If you are deploying to a cluster that does not support type: "LoadBalancer", and deployed the example using type: "ClusterIP" instead, run kubectl port-forward svc/jenkins 8080:80 to forward the cluster port to the local machine and access the service via localhost:8080.

When you’re ready to be done with Jenkins, you can destroy the instance:

$ pulumi destroy
Do you want to perform this destroy? yes
Destroying stack 'kubernetes-ts-jenkins-dev'
Performing changes:
Type Name Status Info
- pulumi:pulumi:Stack kubernetes-ts-jenkins-kubernetes-ts-jenkins-dev deleted
- └─ jenkins:jenkins:Instance jenkins deleted
- ├─ kubernetes:extensions:v1beta1:Deployment jenkins-deploy deleted
- ├─ kubernetes:core:v1:Service jenkins-service deleted
- ├─ kubernetes:core:v1:PersistentVolumeClaim jenkins-pvc deleted
- └─ kubernetes:core:v1:Secret jenkins-secret deleted
info: 6 changes performed:
- 6 resources deleted
Update duration: 18.202009282s

Next Steps

This example showed just a little of the potential of reusing components and code, instead of copy and pasting config, to improve productivity and comprehension for building cloud stacks.

Find out more:

Get the example code on GitHub
Read the docs on Kubernetes
See the tutorial on building components