Jupyter on UConn’s HPC cluster

Fundamentals of Data Science

Author

Jeremy Teitelbaum

  1. On your local machine: Login to the login node:
$ ssh netID@hpc2.storrs.hpc.uconn.edu

You need to install anaconda on your account on the cluster. To do this you can use wget to download the installer and then run it and follow the installation prompts as usual. You only need to do this part once!

$ wget https://repo.anaconda.com/archive/Anaconda3-2023.03-1-Linux-x86_64.sh
$ sh Anaconda3-2023.03-1-Linux-x86_64.sh

Now check if python and jupyter lab work.

If jupyter lab won’t run because of a libc++.so.6.o error, you can run

$ conda install -c anaconda libstdcxx-ng
  1. On the cluster: Start an interactive process
$ srun  --partition=class --account=classroom --qos=classroom --pty bash

Make a note of the node where your interactive process is running. This is in the prompt. It will be something like cn560.

  1. On the cluster: Start jupyter on the interactive node
$ jupyter lab --no-browser --ip='*'

Make a note of the token provided by the jupyter lab process and the port where the server is running.

  1. On your local machine: Create an ssh tunnel to the jupyter server.
$ ssh -NL localhost:8888:node:port netID@hpc2.storrs.hpc.uconn.edu
  1. On your local machine: Open a web browser and point it at localhost:8888.

With luck, you’re running jupyter on a node in the cluster!

Moving a file to the cluster

If you can login to the cluster successfully using ssh then you can transfer a file to the cluster from your laptop using rsync. From a shell on your local machine:

rsync filename <netID>@hpc2.storrs.hpc.uconn.edu

Getting a file from the cluster

To transfer a file from the cluster to your local machine, run the following command from a shell on your local machine.

rsync <netID>@hpc2.storrs.hpc.uconn.edu:filename .