How do I run a jupyter notebook server on discovery?

I would like to launch a jupyter server as an interactive job and ssh tunnel into the server so that I can execute code in the browser of my local computer.

Andy Connolly has written up a nice response on how to do this in the DBIC Handbook.

At some point, we should copy his tutorial here.

Andy Connolly wrote up an example in the DBIC Handbook. However, this is unfortunately now outdated since the migration to the SLURM scheduler. Follow his instructions on how to create a password for your jupyter notebook servers.

Here is an updated job submission script. Note that this script assumes that you have a python distribution available in your shell environment. For example, I maintain my own conda distribution. However, if you would prefer to use the research computing distribution, then load their module by uncommenting the line module load jupyter.

#!/bin/bash -l
#SBATCH --job-name=JupyterNotebook
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --time=10:00:00

# get tunneling info
node=$(hostname -s)

# This next command chooses a random port number between 8000 and 8999
port=`echo $(( 8000 + RANDOM % 1000 ))`

# print tunneling instructions jupyter-log
echo -e "
# Command to create ssh tunnel:
ssh -N -f -L ${port}:${node}:${port} ${user}@${cluster}

# Use a Browser on your local machine to go to:
# Uncomment the following line to load Python Module, Otherwise this assumes you are using your own python distribution 
# module load python
jupyter-notebook --no-browser --port=${port} --ip=${node}
# keep it up and running
sleep 3600

now submit job to the cluster.

(base) [f00275v@discovery7 code]$ sbatch jupyter_notebook.pbs
Submitted batch job 1295256

There will be a new text file generated with the unique job id containing instructions on how to tunnel into your server. Let’s read this file:

(base) [f00275v@discovery7 code]$ cat slurm-1295256.out

# Command to create ssh tunnel:
ssh -N -f -L 8644:j12:8644

# Use a Browser on your local machine to go to:

Now open a terminal on your local machine and paste in the command to create the ssh tunnel:

(base) lukechang@Lukes-MacBook-Air ~ % ssh -N -f -L 8644:j12:8644

Open a web browser on your local machine and type localhost:8254. You will now be prompted for the password you generated using jupyter notebook password

If you would like to view the status of your jobs, use squeue.

(base) [f00275v@discovery7 code]$ squeue
           1295256  standard JupyterN  f00275v  R      18:06      1 j12

if you would like to kill your job running the jupyter server when you are done, use scancel

(base) [f00275v@discovery7 code]$ scancel 1295256