********************************************************
 Running CanESM on DRAC Niagara using imsi v0.3b (legacy)
********************************************************

Important Note
###################
These are legacy instructions. Niagara is being decomissioned and is offline as of October 2025. It is replaced by Trillium (University of Toronto) and other new clusters as a part of the 2025 DRAC Infrastructure Renewal. Thse instructions may be useful for those who wish to run CanESM on the new DRAC machines, using the legacy version of imsi. 

Overview and resources
######################

This document provides instructions for running the CanESM5 model on the
Compute Canada machine *Niagara*, using bare metal (no container). This
guide is relying on a model configuration system in development called
*imsi*. 

To get a pre-installed, stable version of imsi, and some required modules:

.. code-block:: bash

    source /project/c/cp4c/cp4c/pyenvs/pyenv-imsi-v0.2-cp4c-0.4/bin/activate


You may also install imsi yourself (see `imsi repo <https://gitlab.com/cccma/imsi>`_).
Read the `imsi documentation <https://imsi.readthedocs.io/en/latest/?version=latest>`_ to get an
overview of how the tools work.

Setup a piControl run
#####################

Navigate to your scratch space and setup a new piControl run as follows [#f1]_:

.. code-block:: bash
    
    cd $SCRATCH
    imsi setup --repo=https://gitlab.com/cccma/canesm.git --ver=v5.1_cp4c --exp=cmip6-piControl --model=canesm51_p1 --runid=<unique-runid>

where you replace `--ver` and `--exp` as needed. Be sure to specify a unique runid, that won't clash with other
users, and keep it below about 15 characters. Using initials is good. e.g. `--runid=ncs-hist-tst-01` (see guidance below)


Next change into the run directory (named for <runid>) and build the executables:


.. code-block:: bash
    
    imsi build

Save the default restart files:

.. code-block:: bash
    
    ./save_restart_files.sh

At present, the default behavior for this run will execute 11 years, one year at a time, resubmitting jobs to the Slurm queue after each year. If you would like to change that or other run settings, see the IMSI documentation `here <https://imsi.readthedocs.io/en/latest/usage.html#modifying-basic-run-parameters>`_. 

If you would like to configure email notifications for your jobs, see :ref:`below <Configuring email notifications` and the `Slurm documentation <https://slurm.schedmd.com/documentation.html>`_.

Submit the run to the batch queue:

.. code-block:: bash
    
    imsi submit

This run will complete a year, and output will appear by year in the `output` directory.

Runid Guidance
^^^^^^^^^^^^^^
 As part creating a run, users must select a "runid", or run identifier, that
 will be used to differeniate their runs from others. In *general* the chosen
 runids are free form, but there are *some* restrictions, specifically, they
 must contain **only lower case alphanumeric characters [a-z] and [0-9], the
 hyphen "-" and the period "."**

    .. warning::

        **Do not** re-use runids of existing runs
        
Configuring email notifications
###############################

Email notifications can be configured by editing the file ``imsi-submission-<runid>.sh``. Add the following to the Slurm preamble at the top:

.. code-block:: bash

    #SBATCH --mail-type=<mail-option>
    #SBATCH --mail-user=<email-address>

Replace <email-address> above with your email. See the `Slurm documentation <https://slurm.schedmd.com/documentation.html>`_ for a list of values for <mail-option>, but some useful options are ``BEGIN`` (notify you when run begins execution), ``END`` (notify when run ends), ``FAIL`` (notify when run fails), and ``ALL`` (notify `almost` any time Slurm does something).

       
.. rubric:: Footnotes

.. [#f1] Note, using the `https` address for gitlab is used here as it has no additional requirements. In general using `--repo=git@gitlab.com:cccma/canesm.git` is superior, but requires `adding your ssh keys to gitlab <https://docs.gitlab.com/ee/user/ssh.html>`_.