******************************************************** Running CanESM on ECCC U2 using imsi ******************************************************** .. TODO: consider splitting this page .. contents:: :local: :depth: 1 Overview and resources ###################### This document provides instructions for running the CanESM model on the ECCC HPC systems using IMSI. Read the `imsi documentation `_ to get an overview of how the tools work. Accessing IMSI Commands ------------------------- To activate the IMSI CLI environment: .. code-block:: bash source /home/scrd102/cccma_libs/imsi/latest/bin/activate # or if you have the site profile loaded, simply: load_imsi_latest .. note:: **Versions will change**. The above command gives you access to the latest version of imsi, but this will change over time. Look at environments deployed beside the ``latest`` softlink for specific version, or refer to ``IMSI_VENV`` from your run's ``config/shell_parameters`` if you want to use the same env originally used on the run. You may also install imsi yourself (not recommended unless you are an imsi developer, see `imsi repo `_). Setup a test run ################## Navigate to your scratch space (ords or sitestore is good since all code is kept in the setup directory) and setup a new piControl run as follows: .. code-block:: bash imsi setup --repo=git@gitlab.science.gc.ca:CCCma/CanESM.git --ver= --exp= --model= --runid= where you should set the desired branch and appropriate model and experiment names. Use `imsi list` or see :ref:`here ` for details on how to query what models/experiments are available for your repo/version. Be sure to specify a unique runid, that won't clash with other users, and keep it below 20 characters. Using initials is good. e.g. ``--runid=ncs-hist-tst-01`` .. note:: Use **imsi setup-menu** instead of `setup`, to select options from a menu, based on a curated set of on disk repositories. See `imsi setup-menu -h` for details. If all goes right, you'll eventually see: .. code-block:: bash IMSI setup complete. You can now: cd to continue with configuration/compilation/submission, see: imsi -h So follow the directions and build the executables (or see `here <#how-to-alter-develop-configurations>`_ to make config modifications prior): .. code-block:: bash imsi build Save the configured restart files: .. code-block:: bash imsi save-restarts Submit the run to the batch queue: .. code-block:: bash imsi submit By default, ``imsi`` will work with ``maestro``, so this will launch the imsi maestro suite, which has the same structure as the regular maestro suite. To monitor a run, you can ``export SEQ_EXP_HOME=$(pwd)/sequencer`` and use ``xflow`` as per normal, or chain together via: .. code-block:: bash export SEQ_EXP_HOME=$(pwd)/sequencer; xflow As `another` option, you can make use of `imsi status `_ .. code-block:: bash imsi status or, if you're using the ensemble tool, use ``imsi ensemble --config-path={ensemble-config.yaml} status`` to see the status of various jobs in your simulation. Supported Models/Experiments ############################ In order to see what models/experiments are supported, you can utilize ``imsi list``. You can either call it from within a setup run directory, or you can utilize ``--repo /path/to/repo`` to query a specific on-disk repo. See ``imsi list -h`` for additional details, but simply put: .. code-block:: bash imsi list or .. code-block:: bash imsi list --repo /path/to/repo which will give you output about the supported options in the provided repo! .. note:: Both models and experiments are known to ``imsi`` as a "selection". If you'd like to change the values used after setting up a run, you can utilize ``imsi set`` to reload the configuration for the desired model experiment. For example - say you wanted to change to a historical experiment, you could: .. code-block:: imsi set -s experiment_name=cmip6-historical Or if you want to change to an amip model/experiment .. code-block:: imsi set -s experiment_name=cmip6-amip -s model_name=canam51_p1 # note the second -s .. warning:: Not every experiment supports all models contained in the ``imsi`` configurations. To see what models are supported for a given experiment, the final source of truth is the ``supported_models`` in the experiment config files. **Note that if you give an inconsistent set of models and experiments,** ``imsi`` **will tell you the supported options** How to alter/develop configurations ################################### Following the directions above, and using supported ``experiment`` and ``model`` combinations, developers and users can easily setup and launch "canned configurations". However, there will be a time when one wishes to modify existing configurations to suit their needs or build entirely new experiments, models, post processing profiles, etc. For developing new configurations, such as model or experiments, see the `imsi documentation for Developing or Modifying configurations `_, For simple workflows including one-off changes to "canned configurations" and modifying basic run parameters, see the examples below. Modifying common run parameters ------------------------------- The following are all simple, standard modifications of single variables, which make them well posed for easy modification. Changing restarts ^^^^^^^^^^^^^^^^^ 1. **one off changes for testing in a setup run**: You can utilize quick testing by modifying the values in ``imsi_configuration_.yaml`` - so just open the file and modify ``parent_runid`` and ``parent_branch_time``, then reconfigure ``imsi`` with .. code-block:: >> imsi config 2. **changing the defaults for a given experiment**: To change the values in reproducible way, simply 1. navigate to ``imsi_config`` in your source repo 2. find the desired experiment definition (typically under ``experiments``) 3. modify ``parent_runid`` and ``parent_branch_time`` as desired Then either setup a new run from the updated repo, or if you've already setup a run with the experiment you modified, run .. code-block:: >> imsi reload To have the new settings applied Changing namelist parameters ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 1. **one off changes for testing in a setup run**: For fast tests, open ``imsi_configuration_.yaml`` and modify the desired parameter, then reconfigure with: .. code-block:: >> imsi config 2. **changing defaults for a given model/experiment combo**: First it should be noted that, by design, experiment settings override the settings defined in model definitions. As such 1. navigate to ``imsi_config`` in the source repo 2. find desired model or experiment file 3. search for the desired setting under ``components`` and change the default value Then either setup a new run with ``imsi setup`` or run ``imsi reload`` from a setup run. Changing input files ^^^^^^^^^^^^^^^^^^^^ See above notes on `changing namelist parameters `_, except note that you'd be altering fields under ``input_files``. Altering post processing settings ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 1. **one off changes for testing in a setup run**: For fast, non shareable tests that won't have values in the ``config`` directory overwritten, open ``imsi_configuration_.yaml`` search for ``postproc`` and modify the desired parameter, then reconfigure with: .. code-block:: >> imsi config 2. **changing defaults for a given "postproc profile"**: For a given model/experiment combination, the postprocessing settings are defined by the ``postproc_profile``, which contains the downstream settings for any postprocessing operations. So: 1. navigate to ``imsi_config`` in the source repo 2. identify what ``postproc_profile`` your model/experiment is using (experiment definition takes precedence) 3. find the associated ``postproc_profile`` definition 4. modify the desired settings Then either setup a new run with ``imsi setup`` or run ``imsi reload`` from a setup run. Altering resource settings ^^^^^^^^^^^^^^^^^^^^^^^^^^ 1. **one off changes for testing in a setup run**: For fast, non shareable tests that won't have values in the ``config`` directory overwritten, open ``imsi_configuration_.yaml`` search for ``jobs`` and modify the desired parameter, then reconfigure with: .. code-block:: >> imsi config 2. **changing defaults for a given** ``sequencing_flow``: With ``imsi``, the resource specifications are defined under the ``sequencing_flow`` :ref:`section of the configuration `, which specific job resources under ``jobs``. There is some nuance in how it is set but you can find which files under the ``imsi-config`` directory contain these sections via a simple: .. code-block:: >> cd src/imsi-config >> grep -nr "sequencing_flow" Which one is selected depends on the ``sequencer`` (defaults to ``maestro`` for ECCC systems, and ``iss`` for Niagara), the value of ``model_type`` associated with your model (eg: ``ESM``, ``AMIP``, ``OMIP``), and lastly the machine you're running on. For example, say I have a config repo setup such that I have files like: .. code-block:: yaml sequencing: sequencers: maestro: ... baseflows: ESM: canesm_split_job_flow: AMIP: canam_split_job_flow: OMIP: cannemo_split_job_flow: This will mean that for the "baseflow", ``ESM`` simulations will use ``canesm_split_job_flow`` as the default for ``maestro`` and likewise, ``AMIP`` will use ``canam_split_job_flow``, while ``OMIP`` will use ``cannemo_split_job_flow``. With this baseflow in hand, ``imsi`` uses this combined with ``default_sequencing_suffix`` in the ``machine`` configuration files to build the machine specific flow name : ``{baseflow}-{default_sequencing_suffix}``. And so to alter default resources: 1. navigate to ``imsi_config`` in the source repo 2. determine the ``model_type`` used in your model/experiment 3. determine the ``baseflow`` used your sequencer 4. to modify resources for all machines that use this flow, change the values for the flows `without` ``default_sequencing_suffix`` 5. to modify resources for a specific machine that uses this flow, find ``default_sequencing_suffix`` and modify the settings under ``{baseflow}-{default_sequencing_suffix}`` Then either setup a new run with ``imsi setup`` or run ``imsi reload`` from a setup run. .. note:: In some job specification sections you might see ``directives`` in addition to more specific ``resources`` fields like ``wallclock``, ``memory``, or ``processors``. This is only utilized by specific sequencers - specifically, it is used for ``iss`` and ignored by ``maestro``. Notes on model versions ####################### As of June 2025, ``imsi`` supports multiple versions of ``canesm``, see below for notes on these versions and how to access them: * ``v5.1``: * via the ``v5.1.8`` `tag `_ * ``v5.3``: * via ``develop_canesm`` on the `central repository `_ * ``v6.0``: * via ``v6.0_release`` on the `central repository `_ .. note:: some experiments may be setup to start from rest by default. If this is the case and you want to instead start from a restart - to achieve this (after setting up): 1. open ``imsi_configuration_.yaml`` and, depending on if you're running and ``AMIP``, ``ESM``, or ``OMIP`` experiment, search for ``ocean_from_initialization``, ``coupler_from_initialization``, and ``agcm_from_initialization`` and set them to ``off``. 2. search for and change ``parent_runid`` and ``parent_branch_time`` as you would in the ``config-canesm`` system 3. save the updated ``imsi_configuragion_.yaml`` file 4. run ``imsi config`` to apply the changes 5. run ``imsi save-restarts`` to save the desired restarts Setting up a test ensemble ########################## To aid in the setup of test/development/production ensembles, ``imsi`` has full ensemble support integrated directly into the tool. Documentation on this useful extension can be found `here `_ - see there for specifics on the commands used to setup/launch an ensemble. To help get users started in the ``canesm`` world, we provide some simple ensemble config files here. Simple ensemble with varying adjustable params: ----------------------------------------------- `Simple ensemble with varying adjustable params `_ ------------------------------------------------------------------------------------------------------------------------------------------------------ `Simple ensemble for testing experiments `_ ------------------------------------------------------------------------------------------------------------------------------------------