Running CanESM on DRAC Niagara using imsi

Overview and resources

This document provides instructions for running the CanESM model on the Compute Canada machine Niagara, using bare metal (no container). This guide is relying on a model configuration system in development called imsi.

To get a pre-installed, stable version of imsi, and some required modules:

source /project/c/cp4c/cp4c/pyenvs/pyenv-imsi-cp4c-0.3b/bin/activate

You may also install imsi yourself (see imsi repo). Read the imsi documentation to get an overview of how the tools work.

Setup a piControl run

Navigate to your scratch space and setup a new piControl run as follows [1]:

cd $SCRATCH
imsi setup --repo=https://gitlab.com/cccma/canesm.git --ver=v5.1_cp4c --exp=cmip6-piControl --model=canesm51_p1 --runid=<unique-runid>

where you replace –ver and –exp as needed. Be sure to specify a unique runid, that won’t clash with other users, and keep it below about 15 characters. Using initials is good. e.g. –runid=ncs-hist-tst-01 (see guidance below)

Next change into the run directory (named for <runid>) and build the executables:

imsi build

Save the default restart files:

./save_restart_files.sh

At present, the default behavior for this run will execute 11 years, one year at a time, resubmitting jobs to the Slurm queue after each year. If you would like to change that or other run settings, see the IMSI documentation here.

If you would like to configure email notifications for your jobs, see below <Configuring email notifications and the Slurm documentation.

Submit the run to the batch queue:

imsi submit

This run will complete a year, and output will appear by year in the output directory.

Runid Guidance

As part creating a run, users must select a “runid”, or run identifier, that will be used to differeniate their runs from others. In general the chosen runids are free form, but there are some restrictions, specifically, they must contain only lower case alphanumeric characters [a-z] and [0-9], the hyphen “-” and the period “.”

Warning

Do not re-use runids of existing runs

Configuring email notifications

Email notifications can be configured by editing the file imsi-submission-<runid>.sh. Add the following to the Slurm preamble at the top:

#SBATCH --mail-type=<mail-option>
#SBATCH --mail-user=<email-address>

Replace <email-address> above with your email. See the Slurm documentation for a list of values for <mail-option>, but some useful options are BEGIN (notify you when run begins execution), END (notify when run ends), FAIL (notify when run fails), and ALL (notify almost any time Slurm does something).

Footnotes

[1]

Note, using the https address for gitlab is used here as it has no additional requirements. In general using –repo=git@gitlab.com:cccma/canesm.git is superior, but requires adding your ssh keys to gitlab.