Hosted Scalable Jupyter Notebooks on AWS EC2

Question

Pulumi · Accepted Answer

To host scalable Jupyter Notebooks on AWS EC2, you will need to set up an EC2 instance with the appropriate resources and then configure Jupyter Notebooks to run on that instance. In the context of using Pulumi to automate the creation of the necessary resources on AWS, we'll outline a program that will do the following:

Create an EC2 instance with a suitable instance type.
Provision the instance with an initial setup script via user data to install Jupyter Notebooks.
Set up a security group to allow traffic to the Jupyter server, typically on port 8888.

We will use the following Pulumi resources:

aws.ec2.Instance: This resource will create an EC2 instance where we will host Jupyter Notebook.
aws.ec2.SecurityGroup and aws.ec2.SecurityGroupRule: These resources will set up the security group to control the inbound and outbound traffic for the EC2 instance.
aws.ec2.KeyPair: This optional resource can be used if you need to SSH into the EC2 instance for any manual setup or debugging.

Below is the Pulumi program written in Python that outlines the steps needed to create this setup.

import pulumi
import pulumi_aws as aws

# Create an EC2 key pair for SSH access (optional)
key_pair = aws.ec2.KeyPair("keyPair",
    key_name="jupyter-notebook-key",
    public_key="ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCu... user@host"  # Replace with your actual SSH public key
)

# Create a security group to allow HTTP and SSH traffic
secgroup = aws.ec2.SecurityGroup('jupyter-notebook-sg',
    description='Enable HTTP and SSH access',
    ingress=[
        aws.ec2.SecurityGroupIngressArgs(
            protocol='tcp',
            from_port=8888,
            to_port=8888,
            cidr_blocks=['0.0.0.0/0'],  # Allows access from any IP, consider restricting it
        ),
        aws.ec2.SecurityGroupIngressArgs(
            protocol='tcp',
            from_port=22,
            to_port=22,
            cidr_blocks=['0.0.0.0/0'],  # Allows SSH access from any IP, consider restricting it
        )
    ]
)

# Specify the user_data script to install Jupyter Notebook
user_data = """#!/bin/bash
sudo apt update
sudo apt install -y python3-pip
pip3 install notebook
# Run Jupyter Notebook on startup
echo "c.NotebookApp.ip = '*'" >> /home/ubuntu/.jupyter/jupyter_notebook_config.py
echo "c.NotebookApp.open_browser = False" >> /home/ubuntu/.jupyter/jupyter_notebook_config.py
nohup jupyter notebook &
"""

# Create an EC2 instance to host Jupyter Notebook
instance = aws.ec2.Instance('jupyter-notebook-instance',
    instance_type='t3.medium',  # Choose an instance type that suits your workload
    security_groups=[secgroup.name],  # Reference the security group
    user_data=user_data,  # Pass the user_data script to the instance
    ami=ami_id,  # You need to provide the AMI ID of the instance, e.g. ami-0d5eff06f840b45e9 for Ubuntu Server 18.04 LTS
    key_name=key_pair.key_name if key_pair else None
)

# Export the public IP address of the instance
pulumi.export('instance_ip', instance.public_ip)

Explanation:

We create a key pair so that you can SSH into the EC2 instance if needed. You should replace the public key with your SSH public key.
We initialize a security group that allows us to connect to the Jupyter Notebook server through HTTP on port 8888 and also enable SSH connections.
The user_data script automates the installation of Jupyter Notebook on the EC2 instance when it's being created. This script runs when the EC2 instance first boots up.
aws.ec2.Instance is then used to create the EC2 instance. We specify the instance type, attach the security group, and include the user data script. The AMI ID should be set to an appropriate value for your needs, such as the latest Ubuntu Server AMI.
We export the public IP address of the instance to make it easy to access the Jupyter server from a web browser.

Once the Pulumi program is executed, it will create all the resources mentioned in the code. After the resources are provisioned, you should be able to access your hosted Jupyter Notebook server by navigating to http://<instance_ip>:8888 in your web browser. Replace <instance_ip> with the public IP address outputted by Pulumi after the deployment.