How do I run a jupyter notebook server on discovery?

Discovery Questions
1

I would like to launch a jupyter server as an interactive job and ssh tunnel into the server so that I can execute code in the browser of my local computer.

2

Andy Connolly has written up a nice response on how to do this in the DBIC Handbook.

At some point, we should copy his tutorial here.

3

Andy Connolly wrote up an example in the DBIC Handbook. However, this is unfortunately now outdated since the migration to the SLURM scheduler. Follow his instructions on how to create a password for your jupyter notebook servers.

Here is an updated job submission script. Note that this script assumes that you have a python distribution available in your shell environment. For example, I maintain my own conda distribution. However, if you would prefer to use the research computing distribution, then load their module by uncommenting the line module load jupyter.

#!/bin/bash -l
#SBATCH --job-name=JupyterNotebook
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --time=10:00:00

# get tunneling info
XDG_RUNTIME_DIR=""
node=$(hostname -s)
user=$(whoami)
cluster="discovery7"

# This next command chooses a random port number between 8000 and 8999
port=`echo $(( 8000 + RANDOM % 1000 ))`

# print tunneling instructions jupyter-log
echo -e "
# Command to create ssh tunnel:
ssh -N -f -L ${port}:${node}:${port} ${user}@${cluster}.dartmouth.edu

# Use a Browser on your local machine to go to:
localhost:${port}  
"
# Uncomment the following line to load Python Module, Otherwise this assumes you are using your own python distribution 
# module load python
jupyter-notebook --no-browser --port=${port} --ip=${node}
# keep it up and running
sleep 3600

now submit job to the cluster.

(base) [f00275v@discovery7 code]$ sbatch jupyter_notebook.pbs
Submitted batch job 1295256

There will be a new text file generated with the unique job id containing instructions on how to tunnel into your server. Let’s read this file:

(base) [f00275v@discovery7 code]$ cat slurm-1295256.out

# Command to create ssh tunnel:
ssh -N -f -L 8644:j12:8644 f00275v@discovery7.dartmouth.edu

# Use a Browser on your local machine to go to:
localhost:8644

Now open a terminal on your local machine and paste in the command to create the ssh tunnel:

(base) lukechang@Lukes-MacBook-Air ~ % ssh -N -f -L 8644:j12:8644 f00275v@discovery.dartmouth.edu

Open a web browser on your local machine and type localhost:8254. You will now be prompted for the password you generated using jupyter notebook password

If you would like to view the status of your jobs, use squeue.

(base) [f00275v@discovery7 code]$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
           1295256  standard JupyterN  f00275v  R      18:06      1 j12

if you would like to kill your job running the jupyter server when you are done, use scancel

(base) [f00275v@discovery7 code]$ scancel 1295256