********************************************************
 Running CanESM on DRAC Niagara using imsi
********************************************************

Overview and resources
######################

This document provides instructions for running the CanESM model on the
Compute Canada machine *Niagara*, using bare metal (no container). This
guide is relying on a model configuration system in development called
*imsi*. 

To get a pre-installed, stable version of imsi, and some required modules:

.. code-block:: bash

    source /project/c/cp4c/cp4c/pyenvs/pyenv-imsi-cp4c-0.3b/bin/activate


You may also install imsi yourself (see `imsi repo <https://gitlab.com/cccma/imsi>`_).
Read the `imsi documentation <https://imsi.readthedocs.io/en/latest/?version=latest>`_ to get an
overview of how the tools work.

Setup a piControl run
#####################

Navigate to your scratch space and setup a new piControl run as follows [#f1]_:

.. code-block:: bash
    
    cd $SCRATCH
    imsi setup --repo=https://gitlab.com/cccma/canesm.git --ver=v5.1_cp4c --exp=cmip6-piControl --model=canesm51_p1 --runid=<unique-runid>

where you replace `--ver` and `--exp` as needed. Be sure to specify a unique runid, that won't clash with other
users, and keep it below about 15 characters. Using initials is good. e.g. `--runid=ncs-hist-tst-01` (see guidance below)


Next change into the run directory (named for <runid>) and build the executables:


.. code-block:: bash
    
    imsi build

Save the default restart files:

.. code-block:: bash
    
    ./save_restart_files.sh

At present, the default behavior for this run will execute 11 years, one year at a time, resubmitting jobs to the Slurm queue after each year. If you would like to change that or other run settings, see the IMSI documentation `here <https://imsi.readthedocs.io/en/latest/usage.html#modifying-basic-run-parameters>`_. 

If you would like to configure email notifications for your jobs, see :ref:`below <Configuring email notifications` and the `Slurm documentation <https://slurm.schedmd.com/documentation.html>`_.

Submit the run to the batch queue:

.. code-block:: bash
    
    imsi submit

This run will complete a year, and output will appear by year in the `output` directory.

Runid Guidance
^^^^^^^^^^^^^^
 As part creating a run, users must select a "runid", or run identifier, that
 will be used to differeniate their runs from others. In *general* the chosen
 runids are free form, but there are *some* restrictions, specifically, they
 must contain **only lower case alphanumeric characters [a-z] and [0-9], the
 hyphen "-" and the period "."**

    .. warning::

        **Do not** re-use runids of existing runs
        
Configuring email notifications
###############################

Email notifications can be configured by editing the file ``imsi-submission-<runid>.sh``. Add the following to the Slurm preamble at the top:

.. code-block:: bash

    #SBATCH --mail-type=<mail-option>
    #SBATCH --mail-user=<email-address>

Replace <email-address> above with your email. See the `Slurm documentation <https://slurm.schedmd.com/documentation.html>`_ for a list of values for <mail-option>, but some useful options are ``BEGIN`` (notify you when run begins execution), ``END`` (notify when run ends), ``FAIL`` (notify when run fails), and ``ALL`` (notify `almost` any time Slurm does something).

       
.. rubric:: Footnotes

.. [#f1] Note, using the `https` address for gitlab is used here as it has no additional requirements. In general using `--repo=git@gitlab.com:cccma/canesm.git` is superior, but requires `adding your ssh keys to gitlab <https://docs.gitlab.com/ee/user/ssh.html>`_.