Pulumi Blog: Jupyter

Data Science in the Cloud

Sophia Parafina — Thu, 18 Mar 2021 00:00:00 +0000

Data science has advanced because tools like Jupyter Notebook hide complexity by running high level code for the specific problem they are trying to solve. Increasing the level of abstraction lets a data scientist be more productive by reducing the effort to try multiple approaches to near zero, which encourages experimentation and better results.

Data scientists typically work locally, but they often store data for analyses and models in the cloud. There are clear advantages to using cloud resources for these tasks:

Data scientists generally don’t want to manage their storage and databases.
They need to be able to store large data sets cheaply.
They need large capacity swings available on-demand.

SDKs like AWS’ Python library, boto3, can create resources, but they still require domain expertise to manage and properly architect a solution. The Pulumi Automation API improves on raw SDKs by providing high-level abstractions for creating and managing cloud services, letting data scientists concentrate on analyses and models without being well-versed in cloud APIs.

How Pulumi Automation API Improves on Cloud SDKs

Pick any cloud provider, and you will find a cloud SDK in various languages, for example AWS SDKs, Azure SDKs, and Google Cloud SDKs. These SDKs directly access cloud resources, letting data scientists provision resources as required. The drawback to using these libraries is that the data scientist is also responsible for providing resources reliably and securely and performing modeling and analysis.

Pulumi reduces the burden of creating and maintaining repeatable and robust deployments by providing the latest language runtimes, concurrency management, support for diffing updates, and maintaining state with robust checkpoints. Your infrastructure will always be in a well-defined and recoverable state.

Automation API takes this one step further by providing an API overlay to Pulumi’s Infrastructure as Code engine. A data scientist can build infrastructure programmatically within Jupyter to automate building and creating infrastructure as functions and call them as needed. Let’s take a look at an example using a Jupyter notebook.

Infrastructure as a function

As a data scientist, you might want to share data, charts, or graphs with remote team members and make them available through a shared website. You want to keep iterating on your model but don’t want to update the site manually, or worse, let it get out of date.

This example is a simple overview of Automation API that demonstrates a basic deployment of a static website from within a Jupyter notebook. This example uses an inline program to define infrastructure within a function alongside your other code. The function called s3_static_site creates an s3 bucket, sets it up as a basic static website, and exports the URL. This is the equivalent of creating a Pulumi stack using the CLI.

We’ve added code to the example to upload files written to the www directory. Every time you want to update the website, you can write a file to the directory and call the function to upload it to the s3 bucket. Pulumi maintains state, so it will only upload new files when the function is called.

from pulumi_aws import s3

def s3_static_site():
 # Create a bucket and expose a website index document
 site_bucket = s3.Bucket("s3-website-bucket", website=s3.BucketWebsiteArgs(index_document="index.html"))
 index_content = """
 <html>
 <head><title>Hello S3</title><meta charset="UTF-8"></head>
 <body>
 <p>Hello, world!</p>
 <p>Made with ❤️ with <a href="https://pulumi.com">Pulumi</a></p>
 </body>
 </html>
 """

 # Write our index.html into the site bucket
 s3.BucketObject("index",
 bucket=site_bucket.id, # reference to the s3.Bucket object
 content=index_content,
 key="index.html", # set the key of the object
 content_type="text/html; charset=utf-8") # set the MIME type of the file

 # Set the access policy for the bucket so all objects are readable
 s3.BucketPolicy("bucket-policy", bucket=site_bucket.id, policy={
 "Version": "2012-10-17",
 "Statement": {
 "Effect": "Allow",
 "Principal": "*",
 "Action": ["s3:GetObject"],
 # Policy refers to bucket explicitly
 "Resource": [pulumi.Output.concat("arn:aws:s3:::", site_bucket.id, "/*")]
 },
 })

 # Export the website URL
 pulumi.export("website_url", site_bucket.website_endpoint)

We can write additional functions to deploy and destroy the S3 static website. We deploy the stack by calling the deploy_project function and pass it the s3_static_site function, the Pulumi AWS plugin, and the AWS region.

s3_site = deploy_project("my_first_project",
 s3_static_site,
 plugins=[("aws", "v3.23.0")],
 config={"aws:region": auto.ConfigValue(value="us-west-2")})

You can view the resources that the function created in two ways. The standard method is to open the AWS console to see your resources.

You can also view them with the Pulumi Service as a resource graph.

The Pulumi Service also displays updates to the static site and the changes made.

Now that the static site has been deployed, we want to get the site URL. The s3_static_site function exports the website url. Pulumi returns the s3_site as an object with outputs that we use to retrieve the URL. There are several ways to check if the site is working, but a simple way is to open the URL in a web browser.

import webbrowser

outputs = s3_site.outputs()

webbrowser.open(url)

Data Science Your Way

In this simple example, we demonstrated how to create a static website from an S3 bucket, write content into the bucket, and set the access policy as a Python function. We called the function, which created a stack for the website. The stack is returned as a Python object with the site URL as an output. You can manage the site and view updates using the Pulumi Service.

Because it’s code, you can create a separate module containing all the functions to create your infrastructure, import it into your notebooks, and reuse it. Pulumi lets you perform analysis and build models within a familiar environment while reducing the burden of managing infrastructure. You can find the complete example on GitHub including a Jupyter notebook for database migration.

Be sure to watch our video that explains the code in greater detail.

Create Secure Jupyter Notebooks on Kubernetes using Pulumi

Nishi Davidson — Thu, 30 May 2019 00:00:00 +0000

In this post, we will work through an example that shows how to use Pulumi to create Jupyter Notebooks on Kubernetes. Having worked on Kubernetes since 2015, a couple of critical benefits jump out that may resonate with you as well:

You write everything in code - TypeScript in our example here.
You need not initialize Tiller or Helm to work with existing Helm charts like nginx-ingress-controller that we use here.
The security patterns in Helm and Tiller are no longer concerns, rather you get to focus on the RBAC of the actual service which is Jupyter-notebook in this example.
You accomplish more with less YAML and iteratively work towards your use cases.

Prerequisites

Install Pulumi
Install Node.js version 6 or later
Install a package manager for Node.js, such as npm or Yarn
Install Google Cloud SDK
Install Helm and only initialize with helm init —client-only

We will work this example on a GKE cluster so lets first configure Google Cloud Auth:

$ gcloud auth login
$ gcloud config set project <YOUR_GCP_PROJECT_HERE>
$ gcloud auth application-default login

Step 1: Create a Pulumi Project and Stack with a Pulumi TypeScript Template

$ mkdir gke-jupyter-notebook && cd gke-jupyter-notebook
$ pulumi new typescript
$ npm install --save @pulumi/kubernetes @pulumi/gcp

Step 2: Create a GKE Cluster

To create a GKE cluster, simply update the following code in index.ts file and run pulumi up.

import * as k8s from "@pulumi/kubernetes";
import * as pulumi from "@pulumi/pulumi";
import * as gcp from "@pulumi/gcp";
import { readFileSync, fstat } from "fs";

const name = "jupyter-notebook";

/*
 * STEP 2: Create a GKE Cluster
 */
const engineVersion = gcp.container.getEngineVersions().then(v => v.latestMasterVersion);
const cluster = new gcp.container.Cluster(name, {
 initialNodeCount: 2,
 minMasterVersion: engineVersion,
 nodeVersion: engineVersion,
 nodeConfig: {
 machineType: "n1-standard-1",
 oauthScopes: [
 "https://www.googleapis.com/auth/compute",
 "https://www.googleapis.com/auth/devstorage.read_only",
 "https://www.googleapis.com/auth/logging.write",
 "https://www.googleapis.com/auth/monitoring"
 ],
 },
});

// Export the Cluster name
export const clusterName = cluster.name;

// Manufacture a GKE-style kubeconfig.
export const kubeconfig = pulumi.
 all([ cluster.name, cluster.endpoint, cluster.masterAuth ]).
 apply(([ name, endpoint, masterAuth ]) => {
 const context = `${gcp.config.project}_${gcp.config.zone}_${name}`;
 return `apiVersion: v1
clusters:
- cluster:
 certificate-authority-data: ${masterAuth.clusterCaCertificate}
 server: https://${endpoint}
 name: ${context}
contexts:
- context:
 cluster: ${context}
 user: ${context}
 name: ${context}
current-context: ${context}
kind: Config
preferences: {}
users:
- name: ${context}
 user:
 auth-provider:
 config:
 cmd-args: config config-helper --format=json
 cmd-path: gcloud
 expiry-key: '{.credential.token_expiry}'
 token-key: '{.credential.access_token}'
 name: gcp
`;
 });

// Create a Kubernetes provider instance that uses our cluster from above.
const clusterProvider = new k8s.Provider(name, { kubeconfig: kubeconfig });

Step 3: Create a NGINX-Ingress-Controller to Generate Ingresses

You can use the default L7 load balancer in Google Cloud but here we add an NGINX-Ingress-Controller to the cluster. With Pulumi, you write four lines of typescript to reuse the stable NGINX ingress controller helm chart and have the controller, default backends, RBAC, Service account, Config map all running within seconds.

Add the following lines of code in index.ts file and run pulumi up once again.

/*
 * STEP 3: Create NGINX Ingress Controller in GKE
 */

const nginxingresscntlr = new k8s.helm.v2.Chart("nginxingresscontroller", {
 repo: "stable",
 chart: "nginx-ingress",
 version: "0.24.1",
 values: {},
}, { providers: { kubernetes: clusterProvider } });

Step 4: Create a Jupyter Notebook Deployment and Service with Type NodePort

Bringing up Jupyter notebook deployment and service requires adding the following lines of code in index.ts file and running pulumi up to apply the changes.

/*
 * STEP 4: Create Jupyter notebook deployment and service in the GKE cluster
 */

const appName = "jupyter-notebook";
const appLabels = { app: appName };
const jupyterNotebook = new k8s.apps.v1beta1.Deployment(appName, {
 metadata: { name: appName, labels: appLabels },
 spec: {
 selector: { matchLabels: appLabels },
 replicas: 1,
 template: {
 metadata: { labels: appLabels },
 spec: {
 containers: [
 {
 name: appName,
 image: "jupyter/tensorflow-notebook",
 ports: [{ containerPort: 8888 }],
 command: ["start-notebook.sh"],
 args: ["--NotebookApp.token=''"]
 }]
 }
 }
 }
}, { provider: clusterProvider });

const jupyterService = new k8s.core.v1.Service(appName, {
 metadata: {
 name: appName,
 labels: appLabels
 },
 spec: {
 type: "NodePort",
 selector: appLabels,
 ports: [{ protocol: "TCP", nodePort: 30040, port: 8888, targetPort: 8888 }],
 }
}, { provider: clusterProvider });

Step 5: Create a Secret that is used with your Jupyter Notebook Domain Name

We first create a local auth.txt file with the password using the following command:

htpasswd -c auth.txt jupyter

We then read this file synchronously, convert it to base64 and add it as a secret in the GKE cluster. We use this secret as the TLS password to access the Jupyter notebook ingress endpoint accessible from the domain name defined in the host section of the ingress declaration. The annotations in the ingress declarations are required to enable this behavior on the ingress object.

/*
 * STEP 5: Create a secret to enable "basic-auth" for your Jupyter
 * notebook ingress and add it to the ingress declaration in the
 * GKE cluster
 */

const authContents = (readFileSync("<path-to-auth.txt-file>")).toString()

function toBase64(s: string): string {
 return Buffer.from(s).toString("base64");
}

const authContents_base64 = toBase64(authContents);

const jupyternotebooksecret = new k8s.core.v1.Secret("jupyter-notebook-tls", {
 metadata: { name: "basic-auth", namespace: "default" },
 type: "Opaque",
 data: {
 auth: authContents_base64,
 }
})

export const jupyternotebookingress = new k8s.extensions.v1beta1.Ingress(appName, {
 metadata: {
 name: appName,
 labels: appLabels,
 annotations: {
 "kubernetes.io/tls-acme": "true",
 "kubernetes.io/ingress.class": "nginx",
 "nginx.ingress.kubernetes.io/auth-type": "basic",
 "nginx.ingress.kubernetes.io/auth-secret": "basic-auth",
 "nginx.ingress.kubernetes.io/auth-realm": "Authentication Required - jupyter"
 },
 },
 spec: {
 rules: [{
 host: "nishidavidson.com",
 http: {
 paths: [{
 path: "/",
 backend: { serviceName: "jupyter-notebook", servicePort: 8888 }
 }]
 }
 }],
 tls: [{
 secretName: "jupyter-notebook-tls",
 hosts: ["nishidavidson.com"]
 }],
 }
}, { provider: clusterProvider });

Once complete, you will see the GKE cluster components show up as follows:

Open a browser to access the jupyter-notebook Service using the domain name declared in the host section of your ingress code above. In our example, I used the domain name nishidavidson.com.

When you open your DNS, you will be asked for the “username: jupyter” and “password” for the secret you created in auth.txt file as shown below:

As soon as the password is accepted, You should now be able to go to your website from anywhere in the world and access your password-protected Jupyter notebook running on GKE on secure SSL connection.

Success! We worked through a simple example of creating a GKE cluster, an NGINX ingress controller and stood up our password protected Jupyter notebook Ingress, Service, and Deployment with a simple secret for authentication.

You can get started today using more solutions on Kubernetes from our Pulumi examples repository.