CanESM Compilation

Summary

At a high level, CanESM compilation is achieved utilizing the following build tools:

where raw GNU Make is used for the AGCM and Ocean diagnostics, along with CanNEMO utilities (i.e. rebuild_nemo); mkmf is used in conjuction with GNU Make for the AGCM and Coupler executables; and makenemo is used for the Ocean executable.

Compile Time Options/Configuration

While many options for CanESM and its various components are set via Fortran namelists, there are still many cpp definitions and compiler flags that must be/are defined at compile time. These options are contained/defined in:

  • cppdef_config.h :

    A header file containing cpp configuration directives. Used for the compilation of the atmosphere and coupler executables, along with the coupler interface for ocean executable.

  • cppdef_sizes.h :

    Similar to cppdef_config.h except that it contains macros used to set various size parameters. Used for the compilation of the atmosphere and coupler executables, along coupler interface for ocean executable.

  • cpp_${NEMO_CONFIG}.fcm :

    A file that gets picked up by makenemo to define cpp definitions for NEMO. Used in the compilation of the ocean executable.

  • make.template.${COMPILER} :

    (Name can be overwritten) Defines compiler specific compilation flags for when raw GNU Make is used. Used in the compilation of the atmosphere diagnostics, ocean diagnostics/tools.

  • mkmf.template.${COMPILER} :

    (Name can be overwritten) Defines compiler specific compilation flags for when mkmf is used. Used in the compilation of the atmosphere and coupler executables.

  • arch_${COMPILER}.${PLATFORM}.fcm :

    A file that gets picked up by makenemo to define compiler specific compilation flags. Used in the compilation of the ocean executable.

For additional information on these files, see the various sections below.

compile-canesm.sh

While it is possible for users to execute/build each component/diagnostic/utility program individually, this requires knowledge of the individual build tools and how the above mentioned files are chosen/generated, which depends on the PLATFORM, COMPILER, and desired configuration/experiment.

As such, when compiling within a setup run, the compilation tool, compile-canesm.sh tool is provided (which is often linked into run directories as compile_${RUNID}) which:

  1. determines what needs to be compiled according to CONFIG (AMIP | OMIP | ESM), which comes from canesm.cfg.

  2. sources the necessary compilation_environment file, which should be in the local config/ directory before running the compilation. For information on where this file comes from, see here.

  3. compiles the required components using the suitable compiler/platform configuration files, which should also be in the local config/ directory before running the compilation. For information on the platform/compiler configuration settings, or compilation of specific components, see here.

Note

For information on the compile-canesm.sh interface, see the help output from compile-canesm.sh -h

Generating the cppdef files

Prior to running the compilation tool, the necessary cppdef_* files must be generated, which is done via config-canesm (found in CCCma_tools/scripts/comm). Users rarely need to execute this script manually, but it is ran via

config-canesm config_file=/path/to/canesm.cfg

To generate the cppdef_* files, this script:

  • simply uses cat to join the contents of the files defined by cppdef_file and cppdef_diag in canesm.cfg, to create cppdef_config.h

  • executes the program set_sizes, which extracts various size parameters from the Atmosphere namelist file (modl.dat) to generate cppdef_sizes.h

Note

The use of cppdef_sizes.h is slated for removal with the hope to have the model read in the size parameters from suitable namelists. However, as of writing this many of the variables that get placed into cppdef_sizes.h are used to defined array dimensions within legacy common blocks, which need static dimensions which is why they can’t be read at run-time.

Note

set_sizes requires a python environment with python 3+ and the f90nml package

Platform/Compiler Dependent Configuration

For different platforms, compilation may require:

  • different compilers,

  • different compilation flags, and/or

  • different environment consideration

as such, this might require different make/mkmf templates, along with arch_*.fcm for the Ocean. Due to the self contained nature of NEMO, the arch file for the Ocean is selected by makenemo’s -m option from files under CanNEMO/nemo/ARCH. However, for everything else (Atmosphere/Coupler executables, Atmosphere/Ocean diagnostics and utilities), these files/settings exist under CONFIG/PLATFORM/${PLATFORM}. Additionally, a platform specific environment may need to set with a different environment file.

Users are free to pull these files into the various build directories and run the compilations manually (see information below). However for a setup run, after the initial setup and calling of a configuration script, make_job or config-canesm, these files get extracted into a local config directory, according to the ${PLATFORM} configuration variable, to easily be picked up by compile-canesm.sh.

It should be noted that the make/mkmf templates are designed in such a way that they can be used by all components/executables that require them. It is the combination of these templates with each component specific Makefile that are used to build the unique executables - the templates are just meant to define the flags that might be used for the platform/compiler combination.

Component Based Compilation

Note

Depending on your platform, to compile any of these components, a specific environment might be required. If this is the case, unless otherwise stated, you should source the necessary compilation_environment file underneath the provided PLATFORM link.

Atmosphere

The compilation of the Atmosphere component is executed within CanAM/build, where the provided Makefile is used according to the following call:

make MKMF_TEMPLATE=/path/to/mkmf/template CPP_CONFIG_FILE=/path/to/cppdef_config.h \
     CPP_SIZES_FILE=/path/to/cppdef_sizes.h

where:

  • MKMF_TEMPLATE is used to define the compiler flags

  • CPP_CONFIG_FILE defines the location of the desired cppdef_config.h file and

  • CPP_SIZES_FILE defines the location of the desired cppdef_sizes.h file

When called as part of compile-canesm.sh, the above arguments are automatically selected, but if executing outside of a run, the MKMF_TEMPLATE can be retrieved from the desired PLATFORM directory, while the CPP_*_FILEs can be generated according by mimicing the behaviour here, or default files can be pulled from CanAM/build/include/.defaults.

It should be noted that the above call is used to generate additional Makefiles using a provided mkmf.template. The parent Makefile then calls make internally (again) to produce the required libraries/executables. To add/remove source files, to the Atmosphere compilation, developers are directed to the README file within the build directory.

Atmosphere Diagnostics (CanDIAG)

The compilation of Atmosphere diagnostics is achieved within CanDIAG/build, using a single Makefile and a specified make.template, i.e.

make COMPILER_TEMPLATE=/path/to/make/template

where COMPILER_TEMPLATE defines the make template that is used to define the compilation flags. As noted above, when compiled as part of a run, a suitable COMPILER_TEMPLATE is automatically selected by compile-canesm.sh, but if a user wishes to compile the diagnostics outside the context of a run, they can use the appropriate make.template* underneath the provided PLATFORM link.

It should be noted that due to the shear amount of programs under the CanDIAG suite, the provided Makefile generates the vast majority of the compilation commands programatically according to three “phony” targets:

  • default_bins

  • all_diag_bins

  • plot_pgms

and unless one of these targets is specified, the Makefile uses default_bins by default. For additional information on these targets and how to add to the compilation see the README file within the build directory.

Coupler

The compilation of the Coupler component is executed within CanCPL/build, where the provided Makefile is used according to the following call:

make MKMF_TEMPLATE=/path/to/mkmf/template CPP_CONFIG_FILE=/path/to/cppdef_config.h \
     CPP_SIZES_FILE=/path/to/cppdef_sizes.h

where:

  • MKMF_TEMPLATE is used to define the compiler flags

  • CPP_CONFIG_FILE defines the location of the desired cppdef_config.h file and

  • CPP_SIZES_FILE defines the location of the desired cppdef_sizes.h file

When called as part of compile-canesm.sh, the above arguments are automatically selected, but if executing outside of a run, the MKMF_TEMPLATE can be retrieved from the desired PLATFORM directory, while the CPP_*_FILEs can be generated according by mimicing the behaviour here, or default files can be pulled from CanCPL/build/include/.defaults.

It should be noted that the above call is used to generate additional Makefiles using a provided mkmf.template. The parent Makefile then calls make internally (again) to produce the required libraries/executables. To add/remove source files, to the Coupler compilation, developers are directed to the README file within the build directory.

Ocean

As mentioned above, due to the self contained nature of the NEMO ocean model, the building of CanNEMO is achieved via the makenemo tool. However, developers will note the use of another layer of abstraction within compile-canesm.sh - build-nemo. This additional layer is used because CanNEMO can be ran in “ocean-only” mode outside of the main CanESM infrastructure, and thus instead of having two separate build tools, compile-canesm.sh simply calls build-nemo to handle to compiling of the Ocean executable and its diagnostics/utility programs. However, if a user wishes to compile outside the context of a run, they can follow the guidance below.

Using build-nemo

If a user wants to compile the ocean executable and the diagnostics/utilities (see here), they can call build-nemo from anywhere (either by having it on their $PATH or using the path to it), noting that it lives in CanNEMO/bin. Interested readers are directed to the output of build-nemo -h for comprehensive information on it’s interface, but as a simple example, it can be called via:

build-nemo cfg=CCC_CANCPL_ORCA1_LIM_CMOC src_path=/path/to/CanESM/CanNEMO nemo_arch=intel.eccc-xc50 \
             util_arch=/path/to/util/make/tempalte comp_env=/path/to/environment_file

where:

  • cfg defines what source code gets compiled into the Ocean executable, according to nemo/CONFIG/cfg.txt

  • src_path defines the location of the CanNEMO repo

  • nemo_arch is used to supply makenemo’s -m flag and choose the arch-*.fcm file to use from nemo/ARCH/ (when called as part of compile-canesm.sh this is generally set to COMPILER.PLATFORM)

  • util_arch defines the make template to use in the compilation of the diagnostics/utilities, which can be pulled /path/to/CanESM/CONFIG/PLATFORM or /path/to/CanNEMO/nemo/ARCH.

  • comp_env defines the environment file to source before compilation

Using makenemo

If a user just wants the ocean executable, they can navigate to /path/to/CanNEMO/nemo/CONFIG and call makenemo directly i.e.

./makenemo -m $nemo_arch -n $CanNEMO_CONFIG

where:

  • $nemo_arch defines the arch-*.fcm file to use from nemo/ARCH/, and

  • $nemo-config defines what source code gets compiled into the executable, according nemo/CONFIG/cfg.txt

For additional information on this tool, readers are directed to ./makenemo -h.

Warning

Due to technical requirements for the coupling interface for coupled runs, the ocean executable needs to have code from CanCPL compiled in, which requires cppdef_config.h and cppdef_sizes.h to be in a specific location. Additionally, due to makenemo design, some additional consideration is required for the extensions of the header files. Interested readers are directed to the build-nemo code.

Due to these complications, it is recommended that developers limit themselves to compiling non CANCPL configs when doing so outside the context of a CanESM run.

Ocean Diagnostics/Utilities

The compilation of Ocean diagnostics can be lumped into three categories

  • Ocean diagnostics

  • Ocean run time diagnostics (“rtds”), which provide useful reductions of the ocean state to allow for easy inspection of intermediate model results, and

  • the rebuild_nemo tool, which rejoins ocean tiles

where the general ocean diagnostics are built in CanNEMO/diag/build; the rtds built in CanNEMO/rtd/build; and rebuild_nemo built in CanNEMO/nemo/TOOLS/REBUILD_NEMO/src/build.

For each of these groupings, the compilation is handled with a Makefile and a make.template, i.e.

make COMPILER_TEMPLATE=/path/to/make/template

When compiled as part of a CanESM run, each of the above mentioned groupings are handled automatically by the build-nemo tool (see here for additional information), which selects a COMPILER_TEMPLATE according to the util_arch argument. If a user wishes to compile outside the context of a setup run, they can pull the desired template from either CONFIG/PLATFORM or CanNEMO/nemo/ARCH.