Pulumi Blog: Albert Zhong

Reflections of a Pulumi Intern

Albert Zhong — Fri, 18 Sep 2020 00:00:00 +0000

Hi everyone! I’m Albert, a soon-to-be sophomore studying computer science at the University of Washington. Today marks my last day as a Pulumi intern, so I figure I’d reflect on my experiences up until this point.

Joining Pulumi

I heard of Pulumi for the first time when they visited my school’s career fair in January. As I rounded the corner into the CSE1 atrium, I saw a banner titled “Modern Infrastructure as Code,” with some lines of TypeScript of what appeared to be the creation of an S3 bucket. I found it interesting because it looked so simple and modular. In contrast, when I had played around with AWS in the past to deploy the occasional Django web app, it had always involved following a series of exact instructions (usually go to this dashboard, copy this YAML file, run this command, and pray it works).

I walked up to the banner’s adjacent booth and chatted with Sean, a software engineer at Pulumi. He explained how Pulumi was building an open-source platform that allows developers to manage cloud infrastructure using familiar programming languages. To be honest, I didn’t have too much experience with the cloud, but I did sympathize with the pain of wrangling with YAML files. The concept sounded compelling, so I told him I was interested in interning there. He gave me the Pulumi recruiter’s email, who I later emailed that day with my resume and a brief bit about my background. After a few email and phone exchanges, I came over to the Pulumi office for an in-person interview (which was unfortunately also the only time I ever saw it, due to COVID-19.

A couple of weeks later the recruiter called me and extended an offer for the summer intern role! I was incredibly excited about the opportunity to work at such an interesting startup, and especially one working with the cloud, an industry that seemingly dominates my hometown of Seattle. I accepted it a few weeks later since I believed Pulumi had the most fascinating technical challenges to work on relative to the other companies I applied to. From then until summer, I was nervous that my position would be canceled since the coronavirus was forcing many companies to reconsider their budgets. Thankfully Pulumi confirmed that it would shift to a virtual setting, and I started in late June.

The First Week

I had very little idea of what the internship would look like. In addition to it being virtual, it was my first software engineering gig, so I had no prior experience to base it off. I wasn’t sure what my role would be—would I just be assigned a single project and be told to work on it? Would I have a mentor? How does the process of submitting code look like? Fortunately, these questions were answered fairly quickly.

For the first week, the other interns and I converted Pulumi examples from one language to another. TypeScript seemed to be the most common language, so I converted a few from that language to some of the relatively less popular languages, such as Python and Go. I thought that this was an excellent way to help us interns ramp up since it allowed us to get some baseline experience with Pulumi and the various cloud providers and programming languages it supports. By Friday, I was pretty familiar with how to navigate the various Pulumi repositories and documentation, which was a huge benefit as I launched into my project.

My Project: `crd2pulumi`

On the following Monday, I joined a Zoom meeting with Levi and my mentor Justin. They had written up a preliminary design doc on generating SDK types for Kubernetes CustomResources, which would be my project for the summer. I had a minimal background with Kubernetes, so I couldn’t fully conceptualize what the end product would look like until a few days later.

You can define custom fields of a Kubernetes CustomResource using CustomResourceDefinitions (CRDs). Pulumi already allows you to create CustomResources using the apiextensions package. However, there’s no way to inherently know the exact fields of a CustomResource, since it’s set by some CRD. Therefore we have to set the CustomResource fields to some generic type, such as map[string]interface{} in Go or pulumi.Input<any> in TypeScript. This makes filling out the fields for complex CRDs such as cert-manager or Istio very cumbersome since they contain thousands of nested properties. Being able to leverage IDE type-checking and auto-complete makes the process much quicker and less error-prone.

Throughout the summer, I progressively built crd2pulumi, a CLI tool that generates a typed CustomResource from a CRD. I then leveraged this tool to generate libraries for various Kubernetes Operators, which you can find here. I still feel like there are additional features to build out in this area though, so I plan to submit a PR here and there even after today.

School vs. Work

Before joining Pulumi, all my programming experiences had come from either school projects or practicing coding challenge type problems. So far, these scenarios have been fairly atomic, in the sense that they are self-contained and started and finished in a relatively discrete chunk of time. A typical coding project at school is straightforward—I sit down, read the instructions, code everything, and don’t think about it until my grade comes back. Most problems (not relating to the difficulty of the course content itself) can usually be resolved by looking more closely at the instructions or consulting the Piazza board.

In contrast, real software development at Pulumi was far more multi-person, rich, and dynamic. It was challenging, but also far more rewarding. In the first couple of weeks, I was seemingly running into issues at all layers—with the pulumi/pulumi codegen, with the Go compiler, with the Pulumi CLI, with the Python SDKs, or with my Macbook overheating and crashing whenever I tried opening up the examples repository in VSCode. Getting stuff to “just work” often required fixing many nested sub-problems until the root problem could finally be resolved. After a while, I internalized a lot of general debugging heuristics, and developed better skills in squashing bugs and getting code to “just work.”

I thought that Pulumi gave me an excellent balance between guidance and autonomy. My co-workers gave me an outline of what my project should achieve, and some pointers on how which tools to use, but other than that it was up to me to implement it.

One aspect that I liked about Pulumi was that its main audience is, of course, developers, so it was easier to design crd2pulumi since I am a developer. Being able to directly interact with other programmers that use my tool and hear their feedback was immensely satisfying. I loved the energy that came from working with a fast-growing startup that provided real value to real people. It was exciting to come to Monday’s team meeting and see Pulumi’s user count continuously rise, and hear about the latest deal that the sales team had just closed.

Pros of the Virtual Life

I hugely appreciated the flexibility of virtual work-from-home. In conjunction with the two summer classes I was taking, this allowed me to occasionally optimize my schedule. For example, if I had stayed up late on Tuesday night submitting a problem set and I knew I had no meetings tomorrow, I could wake up later on Wednesday morning, but just work a few more hours in the evening to compensate. However, I tried my best to avoid this to keep in sync with the majority of Pulumi employees that are on from 9 am to 5 pm PST.

After a few days, I settled into a routine of:

Wake up around 9-10 am and drink some water
Fire up my Macbook, check Slack and emails, attend meetings if scheduled, read through the reviews to my PR, and write some code
Eat brunch around 1-2 pm (almost always a double-wrapped Chipotle burrito with brown rice and chicken)
Write some more code, fix some bugs, push some changes until 5-6 pm

I feel that the quality of the technical practice I received would have been the same, had the internship been in-person. I usually knew which feature or fix to accomplish, and could troubleshoot most technical errors along the way. If I did run into a bug with my code or had questions on how to implement some function, I could always message a co-worker or set up a meeting to resolve it pretty quickly. I also picked up on a lot of knowledge regarding Pulumi, our tech stack, and best practices by just listening in on meetings and browsing the company Slack channel.

Cons of the Virtual Life

For me, the largest drawback to work-from-home was diminished social opportunities. Everybody I met at Pulumi was so interesting, talented, and kind, so I’m disappointed that I couldn’t meet them in person. There’s no online replacement for actually seeing a person for 8 hours a day and 5 days a week. I feel that so much of life is disproportionately formed via one-off, chance encounters with people, so not being able to go down to a city such as Seattle and work in an office limited my exposure to the envelope of serendipity. Looking back, I should’ve set up more 1-1 meetings with people at Pulumi, just to chat with them and learn from their experiences over lunch.

Nevertheless, it was still enjoyable, and Pulumi tried its hardest to keep us engaged with remote social activities. For example, Lee set up a couple of virtual escape room games which were entertaining.

Wrapping Up

In my view, one of the best lessons I got was that now I know how much I don’t know, which is far better than not knowing how much I don’t know. I was exposed to many different areas of tech—Kubernetes, code generation, TypeScript, Go, infrastructure as code, AWS, GCP, and so much more. I’ve only begun to scratch the huge surface area of the cloud ecosystem, and I’ll always be grateful to the awesome people at Pulumi who helped me get started.

In two weeks I’ll start (also remote) fall quarter at college. If you want to chat, feel free to reach out to me via any of the links on my website. Thanks for reading!

Introducing crd2pulumi: Typed CustomResources for Kubernetes

Levi Blackstone — Wed, 12 Aug 2020 00:00:00 +0000

CustomResources in Kubernetes allow users to extend the API with their types. These types are defined using CustomResourceDefinitions (CRDs), which include an OpenAPI schema. This extensibility is quite useful but comes at the cost of complex YAML definitions. Our new crd2pulumi tool takes the pain out of managing CustomResources by generating types in the Pulumi-supported language of your choice!

Pulumi already supports the management of CRDs and their associated CustomResources using the apiextensions package. However, these SDK resources are untyped since every schema is, well, custom. While this is fine for simple CRDs, it quickly becomes unwieldy for real-world CRDs like cert-manager or Istio. These CRDs contain thousands of lines of complex YAML schemas, making it difficult to write CustomResources that adhere to those specs. Programming languages offer a better path forward. Instead of wrangling error-prone YAML definitions, using types in a programming language lets you use IDE type checking and autocomplete features!

Getting Started with crd2pulumi

Let’s test crd2pulumi on the example CronTab CRD specified in the Kubernetes Documentation.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
 # name must match the spec fields below, and be in the form: <plural>.<group>
 name: crontabs.stable.example.com
spec:
 # group name to use for REST API: /apis/<group>/<version>
 group: stable.example.com
 # list of versions supported by this CustomResourceDefinition
 versions:
 - name: v1
 # Each version can be enabled/disabled by Served flag.
 served: true
 # One and only one version must be marked as the storage version.
 storage: true
 schema:
 openAPIV3Schema:
 type: object
 properties:
 spec:
 type: object
 properties:
 cronSpec:
 type: string
 image:
 type: string
 replicas:
 type: integer
 # either Namespaced or Cluster
 scope: Namespaced
 names:
 # plural name to be used in the URL: /apis/<group>/<version>/<plural>
 plural: crontabs
 # singular name to be used as an alias on the CLI and for display
 singular: crontab
 # kind is normally the CamelCased singular type. Your resource manifests use this.
 kind: CronTab
 # shortNames allow shorter string to match your resource on the CLI
 shortNames:
 - ct

Copy this definition into a file called crontab.yaml and then run crd2pulumi to generate types for your language of choice.

$ crd2pulumi --nodejsPath ./crontabs crontabs.yaml

import * as crontabs from "./crontabs"
import * as pulumi from "@pulumi/pulumi"

// Register the CronTab CRD.
const cronTabDefinition = new crontabs.stable.CronTabDefinition("my-crontab-definition")

// Instantiate a CronTab resource.
const myCronTab = new crontabs.stable.v1.CronTab("my-new-cron-object",
{
 metadata: {
 name: "my-new-cron-object",
 },
 spec: {
 cronSpec: "* * * * */5",
 image: "my-awesome-cron-image",
 replicas: 3,
 }
})

$ crd2pulumi --pythonPath ./crontabs crontabs.yaml

import pulumi_kubernetes as k8s
import crontabs.pulumi_crds as crontabs


# Register the CronTab CRD.
crontab_definition = k8s.yaml.ConfigFile("my-crontab-definition", file="crontabs.yaml")

# Instantiate a CronTab resource.
crontab_instance = crontabs.stable.v1.CronTab(
 "my-new-cron-object",
 metadata=k8s.meta.v1.ObjectMetaArgs(
 name="my-new-cron-object"
 ),
 spec=crontabs.stable.v1.CronTabSpecArgs(
 cron_spec="* * * */5",
 image="my-awesome-cron-image",
 replicas=3,
 )
)

$ crd2pulumi --dotnetPath ./crontabs crontabs.yaml

using Pulumi;
using Pulumi.Kubernetes.Yaml;
using Pulumi.Kubernetes.Types.Inputs.Meta.V1;

class MyStack : Stack
{
 public MyStack()
 {
 // Register a CronTab CRD.
 var cronTabDefinition = new Pulumi.Kubernetes.Yaml.ConfigFile("my-crontab-definition",
 new ConfigFileArgs{
 File = "crontabs.yaml"
 }
 );

 // Instantiate a CronTab resource.
 var cronTabInstance = new Pulumi.Crds.Stable.V1.CronTab("my-new-cron-object",
 new Pulumi.Kubernetes.Types.Inputs.Stable.V1.CronTabArgs{
 Metadata = new ObjectMetaArgs{
 Name = "my-new-cron-object"
 },
 Spec = new Pulumi.Kubernetes.Types.Inputs.Stable.V1.CronTabSpecArgs{
 CronSpec = "* * * * */5",
 Image = "my-awesome-cron-image",
 Replicas = 3
 }
 });
 }
}

$ crd2pulumi --goPath ./crontabs crontab.yaml

package main

import (
 crontabsv1 "crds-go-final/crontabs/stable/v1"

 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v2/go/kubernetes/meta/v1"
 "github.com/pulumi/pulumi-kubernetes/sdk/v2/go/kubernetes/yaml"
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // Register the CronTab CRD.
 _, err := yaml.NewConfigFile(ctx, "my-crontab-definition",
 &yaml.ConfigFileArgs{
 File: "crontabs.yaml",
 },
 )
 if err != nil {
 return err
 }

 // Instantiate a CronTab resource.
 _, err = crontabsv1.NewCronTab(ctx, "my-new-cron-object", &crontabsv1.CronTabArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("my-new-cron-object"),
 },
 Spec: crontabsv1.CronTabSpecArgs{
 CronSpec: pulumi.String("* * * * */5"),
 Image: pulumi.String("my-awesome-cron-image"),
 Replicas: pulumi.IntPtr(3),
 },
 })
 if err != nil {
 return err
 }

 return nil
 })
}

As you can see, the v1.CronTab object is strongly-typed. So if you try to set replicas to a string or add an unsupported argument, your IDE will immediately warn you!

Cert Manager Example

Now let’s examine a real-world cert-manager example. In this case, the CRD is over 1200 lines of YAML, but crd2pulumi generates a nice interface so that we don’t have to worry about it. Here’s what it looks like to create a Certificate CustomResource using our new types.

$ crd2pulumi --nodejsPath ./certificates certificate.yaml

import * as certificates from "./certificates"

// Register the Certificate CRD.
new certificates.certmanager.CertificateDefinition("certificate");

// Instantiate a Certificate resource.
new certificates.certmanager.v1beta1.Certificate("example-cert", {
 metadata: {
 name: "example-com",
 },
 spec: {
 secretName: "example-com-tls",
 duration: "2160h",
 renewBefore: "360h",
 commonName: "example.com",
 dnsNames: [
 "example.com",
 "www.example.com",
 ],
 issuerRef: {
 name: "ca-issuer",
 kind: "Issuer",
 }
 }
});

$ crd2pulumi --pythonPath ./certificates certificate.yaml

import pulumi_kubernetes as k8s
import certmanager.pulumi_crds as certmanager


# Register the Certificate CRD.
_ = k8s.yaml.ConfigFile("my-certificate-definition", file="certificate.yaml")

# Instantiate a Certificate resource.
_ = certmanager.certmanager.v1beta1.Certificate(
 "example-cert",
 metadata=k8s.meta.v1.ObjectMetaArgs(
 name="example-com"
 ),
 spec=certmanager.certmanager.v1beta1.CertificateSpecArgs(
 secret_name="example-com-tls",
 duration="2160h",
 renew_before="360h",
 common_name="example.com",
 dns_names=[
 "example.com",
 "www.example.com"
 ],
 issuer_ref=certmanager.certmanager.v1beta1.CertificateSpecIssuerRefArgs(
 name="ca-issuer",
 kind="Issuer"
 )
 )
)

$ crd2pulumi --dotnetPath ./certificates certificate.yaml

using Pulumi;
using Pulumi.Kubernetes.Yaml;
using Pulumi.Kubernetes.Types.Inputs.Meta.V1;

class MyStack : Stack
{
 public MyStack()
 {
 // Register a Certificate CRD.
 var certificateDefinition = new Pulumi.Kubernetes.Yaml.ConfigFile("my-certificate-definition",
 new ConfigFileArgs{
 File = "certificate.yaml"
 }
 );

 // Instantiate a Certificate resource.
 var certificateInstance = new Pulumi.Crds.Certmanager.V1Beta1.Certificate("example-cert",
 new Pulumi.Kubernetes.Types.Inputs.Certmanager.V1Beta1.CertificateArgs{
 Metadata = new ObjectMetaArgs{
 Name = "example-com"
 },
 Spec = new Pulumi.Kubernetes.Types.Inputs.Certmanager.V1Beta1.CertificateSpecArgs{
 SecretName = "example-com-tls",
 Duration = "2160h",
 RenewBefore = "360h",
 CommonName = "example.com",
 DnsNames = {
 "example.com",
 "www.example.com"
 },
 IssuerRef = new Pulumi.Kubernetes.Types.Inputs.Certmanager.V1Beta1.CertificateSpecIssuerRefArgs{
 Name = "ca-issuer",
 Kind = "Issuer"
 }
 }
 }
 );
 }
}

$ crd2pulumi --goPath ./certificates certificate.yaml

package main

import (
 certv1b1 "crds-go-final/certificates/certmanager/v1beta1"

 metav1 "github.com/pulumi/pulumi-kubernetes/sdk/v2/go/kubernetes/meta/v1"
 "github.com/pulumi/pulumi-kubernetes/sdk/v2/go/kubernetes/yaml"
 "github.com/pulumi/pulumi/sdk/v2/go/pulumi"
)

func main() {
 pulumi.Run(func(ctx *pulumi.Context) error {
 // Register the Certificate CRD.
 _, err := yaml.NewConfigFile(ctx, "my-certificate-definition",
 &yaml.ConfigFileArgs{
 File: "certificate.yaml",
 },
 )
 if err != nil {
 return err
 }

 // Instantiate a Certificate resource.
 _, err = certv1b1.NewCertificate(ctx, "example-cert", &certv1b1.CertificateArgs{
 Metadata: &metav1.ObjectMetaArgs{
 Name: pulumi.String("example-com"),
 },
 Spec: certv1b1.CertificateSpecArgs{
 SecretName: pulumi.String("example-com-tls"),
 Duration: pulumi.String("2160h"),
 RenewBefore: pulumi.String("360h"),
 CommonName: pulumi.String("example.com"),
 DnsNames: pulumi.StringArray{
 pulumi.String("example.com"),
 pulumi.String("www.example.com"),
 },
 IssuerRef: certv1b1.CertificateSpecIssuerRefArgs{
 Name: pulumi.String("ca-issuer"),
 Kind: pulumi.String("Issuer"),
 },
 },
 })
 if err != nil {
 return err
 }

 return nil
 })
}

Kubernetes can be complex, but Pulumi gives you the tools you need to manage it successfully. With Pulumi superpowers at your fingertips, you can stop worrying about YAML indentation, and get back to solving the problems you care about!

Learn More

If you’d like to try crd2pulumi today, head to the release page and download the appropriate binary for your operating system.

If you’d like to learn about Pulumi and how to manage your infrastructure and Kubernetes through code, get started today. Pulumi is open source and free to use.

For further examples on how to use Pulumi to create Kubernetes clusters, or deploy workloads to a cluster, check out the rest of the Kubernetes tutorials.

As always, you can check out our code on GitHub, follow us on Twitter, subscribe to our YouTube channel, or join our Community Slack channel if you have any questions, need support, or just want to say hello.