Differences

This shows you the differences between two versions of the page.

--- software:juypter:caviness [2026-03-26 16:51] – Progress on section on running with an interactive job mbotto
+++ software:juypter:caviness [2026-03-30 14:16] (current) – Delete obsolete copy (with misspelled path name) of Jupyter Notebook Caviness page mbotto
@@ Line 1: / Line 1: @@
-======Jupyter Notebook Python Virtual Environment on Caviness======
-<note warning>
-This page is under construction.
-</note>
-The following steps will walk you through setting up a Conda virtual environment with Python 3 and Jupyter Notebook. It will also cover the steps of requesting a compute node to run a Jupyter Notebook session on Caviness. Lastly, it will explain how to set up SSH connections to be able to connect to the Jupyter Notebook session running on a compute node.
-Before starting, make sure you set your workgroup. This example will demonstrate how to create a shared directory in your workgroup storage where you can create Conda environments that are accessible to everyone in your workgroup.
-<note warning>
-If you do not have a ''sw'' or ''sw/valet'' directory in your workgroup storage directory ''WORKDIR'', please consult with your PI (stakeholder of the workgroup) on how to setup for your workgroup software installs on Caviness before proceeding with these instructions (see [[abstract:caviness:install_software:workgroup-sw|Workgroup Software Installs on Caviness]] for more details).
-</note>
-Before creating the Jupyter Notebook environment, start by creating a directory in your workgroup storage for Conda environments. The ''%%--mode=2775%%'' option ensures that other people in your workgroup will also be able to create Conda environments in this directory:
-<code bash>
-[user@login00.caviness ~]$ workgroup -g my_workgroup
-[(my_workgroup:user)@login00.caviness ~]$ mkdir --mode=2775 ${WORKDIR_SW}/conda-envs
-</code>
-<note important>
-The environment variable ''WORKDIR_SW'' will only be set if the directory ''${WORKDIR}/sw'' already existed before you set your workgroup with ''workgroup -g''. You can check for existence of the directory ''${WORKDIR}/sw'' before setting your workgroup by running:
-<code bash>
-[user@login00.caviness ~]$ ls -d ${WORKDIR}/sw
-</code>
-</note>
-=====Create Jupyter Notebook Virtual Environment with Conda=====
-This example will use the Miniforge VALET package to enable usage of the Conda package manager. If you are new to Conda, make sure to review the documentation for [[software:miniforge:caviness|Miniforge on Caviness]].
-Start by loading the Miniforge VALET package (this command loads the default version, but you can specify a different version as needed):
-<code bash>
-[(my_workgroup:user)@login00.caviness ~]$ vpkg_require miniforge
-Adding package `miniforge/25.11.0-1` to your environment
-</code>
-After loading Miniforge, we can now use ''conda create'' to create the Jupyter Notebook virtual environment.  In this example we will call it ''jupyter-notebook'', but you are welcome to call it anything you would like. We will specify a version directory by date ''20260326'' for the installation of the March 2026 Jupyter Notebook virtual environment. This is needed for setting up a VALET package definition, and allows for multiple versions to be installed based on need. Note that the list of packages to be installed is omitted from the output below.
-<code bash>
-[(my_workgroup:user)@login00.caviness ~]$ conda create --prefix=${WORKDIR_SW}/conda-envs/jupyter-notebook/20260326 python=3 jupyter
-Retrieving notices: done
-Channels:
- - conda-forge
-Platform: linux-64
-Collecting package metadata (repodata.json): done
-Solving environment: done
-  .
-  .
-  .
-## Package Plan ##
-  environment location: /work/my_workgroup/sw/conda-envs/jupyter-notebook/20260326
-  added / updated specs:
-    - jupyter
-    - python=3
-  .
-  .
-  .
-Proceed ([y]/n)? y
-  .
-  .
-  .
-Downloading and Extracting Packages:
-Preparing transaction: done
-Verifying transaction: done
-Executing transaction: done
-#
-# To activate this environment, use
-#
-#     $ conda activate /work/it_css/sw/miniforge-testing/jupyter-notebook/20260326
-#
-# To deactivate an active environment, use
-#
-#     $ conda deactivate
-</code>
-=====VALET Package Definition for the Virtual Environment=====
-The new virtual environment can easily be added to your login shell and job runtime environments using VALET. Ensure you have a workgroup VALET package definition directory present, and get the path to the parent directory of your Jupyter Notebook virtual environment:
-<code bash>
-[(my_workgroup:user)@login00.caviness ~]$ ls -d ${WORKDIR_SW}/valet
-[(my_workgroup:user)@login00.caviness ~]$ echo ${WORKDIR_SW}/conda-envs/jupyter-notebook
-/work/my_workgroup/sw/conda-envs/jupyter-notebook
-</code>
-Take note of the path echoed, then create a new file in ''${WORKDIR_SW}/valet'' named ''jupyter-notebook.vpkg_yaml'' and add the following text to it, **making sure to replace ''my_workgroup'' with the name of your workgroup**:
-<code yaml jupyter-notebook.vpkg_yaml>
-jupyter-notebook:
-    prefix: /work/my_workgroup/sw/conda-envs/jupyter-notebook
-    description: Jupyter Notebook virtual environments created with Conda
-    flags:
-        - no-standard-paths
-    actions:
-        - action: source
-          script:
-              sh: miniforge-activate.sh
-          order: failure-first
-          success: 0
-    versions:
-        "20260326":
-            description: environment built March 26, 2026
-            dependencies:
-                - miniforge
-</code>
-=====Activating the Jupyter Notebook Virtual Environment with VALET=====
-The versions of the virtual environment declared in the VALET package are listed using the ''vpkg_versions'' command:
-<code bash>
-[(my_workgroup:user)@login00.caviness ~]$ vpkg_versions jupyter-notebook
-Available versions in package (* = default version):
-[/work/my_workgroup/sw/valet/jupyter-notebook.vpkg_yaml]
-jupyter-notebook  Jupyter Notebook virtual environments created with Conda
-* 20260326        environment built March 26, 2026
-</code>
-Activating the virtual environment is accomplished using the ''vpkg_require'' command (in your login shell or inside job scripts). If you load the VALET package without having first loaded the ''miniforge'' package, ''miniforge'' will also be added as a dependency:
-<code bash>
-[(my_workgroup:user)@login00.caviness ~]$ vpkg_require jupyter-notebook/20260326
-Adding dependency `miniforge/25.11.0-1` to your environment
-Adding package `jupyter-notebook/20260326` to your environment
-(/work/my_workgroup/sw/conda-envs/jupyter-notebook/20260326) [(my_workgroup:user)@login00.caviness ~]$
-</code>
-You can deactivate the virtual environment and remove the VALET packages it depends on with:
-<code bash>
-(/work/my_workgroup/sw/conda-envs/jupyter-notebook/20260326) [(my_workgroup:user)@login00.caviness ~]$ vpkg_rollback
-[(my_workgroup:user)@login00.caviness ~]$
-</code>
-===== Running Jupyter Notebook on Caviness =====
-Running Jupyter Notebook on Caviness, or any HPC cluster for that matter, takes some extra steps. You can simply install Jupyter Notebook on your personal laptop and start it up. On Caviness, you need to run the Jupyter Notebook on a compute node. The steps below will show you how to request an interactive compute node and use VALET to load the Jupyter Notebook virtual environment. After starting the virtual environment, we will run Jupyter Notebook with arguments that will allow for the session to be accessed via an SSH tunnel connection on your local system.
-<note warning>
-On Caviness you're **required** to run Jupyter notebook on a **compute node**. If you run it on the login node, you could cause slowness or other issues with the login node, and IT might kill your Jupyter Notebook session without warning.
-</note>
-====Requesting an Interactive Job====
-In this example we will request an interactive job to connect us to a compute node with 2GB of memory for 1 hour on the ''my_workgroup'' workgroup partition (replace ''my_workgroup'' with the name of your workgroup). Also it is important to pass the ''SLURM_EXPORT_ENV=NONE'' when requesting the interactive compute node. It will prevent issues with setting up a clean environment on the compute node.
-<code bash>
-[(my_workgroup:user)@login00.caviness ~]$ SLURM_EXPORT_ENV=NONE salloc --mem=2G --time=1:00:00 --partition=my_workgroup
-</code>
-==== Loading the Jupyter Notebook virtual environment ====
-After the interactive job has been established, it is time to load the Jupyter Notebook virtual environment with ''VALET''. Even though we did this earlier, it needs to be done again since we are now on a compute node.
-<code bash>
-[(my_workgroup:user)@r06n21 ~]$ vpkg_require jupyter-notebook/20260326
-Adding dependency `miniforge/25.11.0-1` to your environment
-Adding package `jupyter-notebook/20260326` to your environment
-(/work/my_workgroup/sw/conda-envs/jupyter-notebook/20260326) [(my_workgroup:user)@r06n21 ~]$
-</code>