Running CanESM on ECCC U2 using imsi

Overview and resources

This document provides instructions for running the CanESM model on the ECCC HPC systems using IMSI.

Read the imsi documentation to get an overview of how the tools work.

Accessing IMSI Commands

To activate the IMSI CLI environment:

source /home/scrd102/cccma_libs/imsi/latest/bin/activate

# or if you have the site profile loaded, simply:
load_imsi_latest

Note

Versions will change. The above command gives you access to the latest version of imsi, but this will change over time. Look at environments deployed beside the latest softlink for specific version, or refer to IMSI_VENV from your run’s config/shell_parameters if you want to use the same env originally used on the run.

You may also install imsi yourself (not recommended unless you are an imsi developer, see imsi repo).

Setup a test run

Navigate to your scratch space (ords or sitestore is good since all code is kept in the setup directory) and setup a new piControl run as follows:

imsi setup --repo=git@gitlab.science.gc.ca:CCCma/CanESM.git --ver=<branch-name> --exp=<experiment-name> --model=<model-name> --runid=<unique-runid>

where you should set the desired branch and appropriate model and experiment names. Use imsi list or see here for details on how to query what models/experiments are available for your repo/version. Be sure to specify a unique runid, that won’t clash with other users, and keep it below 20 characters. Using initials is good. e.g. --runid=ncs-hist-tst-01

Note

Use imsi setup-menu instead of setup, to select options from a menu, based on a curated set of on disk repositories. See imsi setup-menu -h for details.

If all goes right, you’ll eventually see:

IMSI setup complete. You can now:

                 cd <runid>
to continue with configuration/compilation/submission, see:
                 imsi -h

So follow the directions and build the executables (or see here to make config modifications prior):

imsi build

Save the configured restart files:

imsi save-restarts

Submit the run to the batch queue:

imsi submit

By default, imsi will work with maestro, so this will launch the imsi maestro suite, which has the same structure as the regular maestro suite. To monitor a run, you can export SEQ_EXP_HOME=$(pwd)/sequencer and use xflow as per normal, or chain together via:

export SEQ_EXP_HOME=$(pwd)/sequencer; xflow

As another option, you can make use of imsi status

imsi status

or, if you’re using the ensemble tool, use imsi ensemble --config-path={ensemble-config.yaml} status to see the status of various jobs in your simulation.

Supported Models/Experiments

In order to see what models/experiments are supported, you can utilize imsi list. You can either call it from within a setup run directory, or you can utilize --repo /path/to/repo to query a specific on-disk repo. See imsi list -h for additional details, but simply put:

imsi list

or

imsi list --repo /path/to/repo

which will give you output about the supported options in the provided repo!

Note

Both models and experiments are known to imsi as a “selection”. If you’d like to change the values used after setting up a run, you can utilize imsi set to reload the configuration for the desired model experiment.

For example - say you wanted to change to a historical experiment, you could:

imsi set -s experiment_name=cmip6-historical

Or if you want to change to an amip model/experiment

imsi set -s experiment_name=cmip6-amip -s model_name=canam51_p1 # note the second -s

Warning

Not every experiment supports all models contained in the imsi configurations. To see what models are supported for a given experiment, the final source of truth is the supported_models in the experiment config files.

Note that if you give an inconsistent set of models and experiments, imsi will tell you the supported options

How to alter/develop configurations

Following the directions above, and using supported experiment and model combinations, developers and users can easily setup and launch “canned configurations”. However, there will be a time when one wishes to modify existing configurations to suit their needs or build entirely new experiments, models, post processing profiles, etc.

For developing new configurations, such as model or experiments, see the imsi documentation for Developing or Modifying configurations,

For simple workflows including one-off changes to “canned configurations” and modifying basic run parameters, see the examples below.

Modifying common run parameters

The following are all simple, standard modifications of single variables, which make them well posed for easy modification.

Changing restarts

  1. one off changes for testing in a setup run:

    You can utilize quick testing by modifying the values in imsi_configuration_<runid>.yaml - so just open the file and modify parent_runid and parent_branch_time, then reconfigure imsi with

    >> imsi config

  2. changing the defaults for a given experiment:

    To change the values in reproducible way, simply

    1. navigate to imsi_config in your source repo

    2. find the desired experiment definition (typically under experiments)

    3. modify parent_runid and parent_branch_time as desired

    Then either setup a new run from the updated repo, or if you’ve already setup a run with the experiment you modified, run

    >> imsi reload

    To have the new settings applied

Changing namelist parameters

  1. one off changes for testing in a setup run:

    For fast tests, open imsi_configuration_<runid>.yaml and modify the desired parameter, then reconfigure with:

    >> imsi config

  2. changing defaults for a given model/experiment combo:

    First it should be noted that, by design, experiment settings override the settings defined in model definitions. As such

    1. navigate to imsi_config in the source repo

    2. find desired model or experiment file

    3. search for the desired setting under components and change the default value

    Then either setup a new run with imsi setup or run imsi reload from a setup run.

Changing input files

See above notes on changing namelist parameters, except note that you’d be altering fields under input_files.

Altering post processing settings

  1. one off changes for testing in a setup run:

    For fast, non shareable tests that won’t have values in the config directory overwritten, open imsi_configuration_<runid>.yaml search for postproc and modify the desired parameter, then reconfigure with:

    >> imsi config

  2. changing defaults for a given “postproc profile”:

    For a given model/experiment combination, the postprocessing settings are defined by the postproc_profile, which contains the downstream settings for any postprocessing operations. So:

    1. navigate to imsi_config in the source repo

    2. identify what postproc_profile your model/experiment is using (experiment definition takes precedence)

    3. find the associated postproc_profile definition

    4. modify the desired settings

    Then either setup a new run with imsi setup or run imsi reload from a setup run.

Altering resource settings

  1. one off changes for testing in a setup run:

    For fast, non shareable tests that won’t have values in the config directory overwritten, open imsi_configuration_<runid>.yaml search for jobs and modify the desired parameter, then reconfigure with:

    >> imsi config

  2. changing defaults for a given sequencing_flow:

    With imsi, the resource specifications are defined under the sequencing_flow section of the configuration, which specific job resources under jobs. There is some nuance in how it is set but you can find which files under the imsi-config directory contain these sections via a simple:

    >> cd src/imsi-config
>> grep -nr "sequencing_flow"

    Which one is selected depends on the sequencer (defaults to maestro for ECCC systems, and iss for Niagara), the value of model_type associated with your model (eg: ESM, AMIP, OMIP), and lastly the machine you’re running on.

    For example, say I have a config repo setup such that I have files like:

    sequencing:
   sequencers:
      maestro:
         ...
         baseflows:
            ESM:
               canesm_split_job_flow:
            AMIP:
               canam_split_job_flow:
            OMIP:
               cannemo_split_job_flow:

    This will mean that for the “baseflow”, ESM simulations will use canesm_split_job_flow as the default for maestro and likewise, AMIP will use canam_split_job_flow, while OMIP will use cannemo_split_job_flow. With this baseflow in hand, imsi uses this combined with default_sequencing_suffix in the machine configuration files to build the machine specific flow name : {baseflow}-{default_sequencing_suffix}.

    And so to alter default resources:

    1. navigate to imsi_config in the source repo

    2. determine the model_type used in your model/experiment

    3. determine the baseflow used your sequencer

    4. to modify resources for all machines that use this flow, change the values for the flows without default_sequencing_suffix

    5. to modify resources for a specific machine that uses this flow, find default_sequencing_suffix and modify the settings under {baseflow}-{default_sequencing_suffix}

    Then either setup a new run with imsi setup or run imsi reload from a setup run.

Note

In some job specification sections you might see directives in addition to more specific resources fields like wallclock, memory, or processors. This is only utilized by specific sequencers - specifically, it is used for iss and ignored by maestro.

Notes on model versions

As of June 2025, imsi supports multiple versions of canesm, see below for notes on these versions and how to access them:

  • v5.1:

    • via the v5.1.8 tag

  • v5.3:

  • v6.0:

    Note

    some experiments may be setup to start from rest by default. If this is the case and you want to instead start from a restart - to achieve this (after setting up):

    1. open imsi_configuration_<runid>.yaml and, depending on if you’re running and AMIP, ESM, or OMIP experiment, search for ocean_from_initialization, coupler_from_initialization, and agcm_from_initialization and set them to off.

    2. search for and change parent_runid and parent_branch_time as you would in the config-canesm system

    3. save the updated imsi_configuragion_<runid>.yaml file

    4. run imsi config to apply the changes

    5. run imsi save-restarts to save the desired restarts

Setting up a test ensemble

To aid in the setup of test/development/production ensembles, imsi has full ensemble support integrated directly into the tool. Documentation on this useful extension can be found here - see there for specifics on the commands used to setup/launch an ensemble. To help get users started in the canesm world, we provide some simple ensemble config files here.

Simple ensemble with varying adjustable params:

Simple ensemble with varying adjustable params

Simple ensemble for testing experiments