==================
CanESM Compilation
==================

#########
 Summary 
#########
At a high level, ``CanESM`` compilation is achieved utilizing the following build tools:

* `GNU Make <https://www.gnu.org/software/make/>`_
* The ``Makefile`` generator, `mkmf (Make Makefile) <https://github.com/NOAA-GFDL/mkmf>`_
* ``makenemo``, which is a build tool shipped with the ``NEMO`` ocean model.

where raw ``GNU Make`` is used for the AGCM and Ocean diagnostics, along with 
``CanNEMO`` utilities (i.e. ``rebuild_nemo``); ``mkmf`` is used in conjuction with ``GNU Make``
for the AGCM and Coupler executables; and ``makenemo`` is used for the Ocean executable.

************************************
 Compile Time Options/Configuration
************************************
While many options for ``CanESM`` and its various components are set via
``Fortran`` ``namelists``, there are still many ``cpp`` definitions and
compiler flags that must be/are defined at compile time. These options are
contained/defined in:

* ``cppdef_config.h`` : 
 
   A header file containing ``cpp`` configuration directives. Used for the
   compilation of the atmosphere and coupler executables, along with the
   coupler interface for ocean executable.
* ``cppdef_sizes.h`` :
 
   Similar to ``cppdef_config.h`` except that it contains macros used to set
   various size parameters. Used for the compilation of the atmosphere and
   coupler executables, along coupler interface for ocean executable.
* ``cpp_${NEMO_CONFIG}.fcm`` :
   
   A file that gets picked up by ``makenemo`` to define ``cpp`` definitions for
   NEMO.  Used in the compilation of the ocean executable.
* ``make.template.${COMPILER}`` : 

   (Name can be overwritten) Defines compiler specific compilation flags for
   when raw ``GNU Make`` is used. Used in the compilation of the atmosphere
   diagnostics, ocean diagnostics/tools.
* ``mkmf.template.${COMPILER}`` : 

   (Name can be overwritten) Defines compiler specific compilation flags for
   when ``mkmf`` is used. Used in the compilation of the atmosphere and coupler
   executables.
* ``arch_${COMPILER}.${PLATFORM}.fcm`` : 

   A file that gets picked up by ``makenemo`` to define compiler specific
   compilation flags. Used in the compilation of the ocean executable.

For additional information on these files, see the various sections below. 

*******************
 compile-canesm.sh
*******************

While it is possible for users to execute/build each component/diagnostic/utility program
individually, this requires knowledge of the individual build tools and how the above
mentioned files are chosen/generated, which depends on the ``PLATFORM``, ``COMPILER``, and
desired configuration/experiment.

As such, when compiling within a setup run, the compilation tool,
``compile-canesm.sh`` tool is provided (which is often linked into run
directories as ``compile_${RUNID}``) which:

1. determines what needs to be compiled according to ``CONFIG`` (``AMIP`` | ``OMIP`` | ``ESM``), which comes
   from ``canesm.cfg``.
2. sources the necessary ``compilation_environment`` file, which should be in
   the local ``config/`` directory before running the compilation. For information on where
   this file comes from, see :ref:`here <Platform/Compiler Dependent Configuration>`.
3. compiles the required components using the suitable compiler/platform
   configuration files, which should also be in the local ``config/`` directory before
   running the compilation. For information on the platform/compiler configuration settings, 
   or compilation of specific components, see :ref:`here <Platform/Compiler Dependent Configuration>`.

.. note::

   For information on the ``compile-canesm.sh`` interface, see the help output from ``compile-canesm.sh -h``

Generating the cppdef files
=============================

Prior to running the compilation tool, the necessary ``cppdef_*`` files must be generated, which is
done via ``config-canesm`` (found in ``CCCma_tools/scripts/comm``). Users rarely need to
execute this script manually, but it is ran via

.. code-block::

    config-canesm config_file=/path/to/canesm.cfg

To generate the ``cppdef_*`` files, this script:

* simply uses ``cat`` to join the contents of the files defined by ``cppdef_file`` and ``cppdef_diag`` in ``canesm.cfg``, to create ``cppdef_config.h``
* executes the program ``set_sizes``, which extracts various size parameters from the Atmosphere namelist file (``modl.dat``) to generate ``cppdef_sizes.h``

.. note::

   The use of ``cppdef_sizes.h`` is slated for removal with the hope to have the model read in the size parameters from
   suitable namelists. However, as of writing this many of the variables that get placed into ``cppdef_sizes.h`` are used
   to defined array dimensions within legacy ``common`` blocks, which need static dimensions which is why they can't
   be read at run-time.

.. note::

   ``set_sizes`` requires a ``python`` environment with ``python 3+`` and the ``f90nml`` 
   `package <https://pypi.org/project/f90nml/>`_

###########################################
 Platform/Compiler Dependent Configuration
###########################################

For different platforms, compilation may require:

* different compilers,
* different compilation flags, and/or
* different environment consideration

as such, this might require different ``make``/``mkmf`` templates, along
with ``arch_*.fcm`` for the Ocean. Due to the self contained nature of NEMO, the 
``arch`` file for the Ocean is selected by ``makenemo``'s ``-m`` option 
from files under ``CanNEMO/nemo/ARCH``. However, for everything else
(Atmosphere/Coupler executables, Atmosphere/Ocean diagnostics and utilities), 
these files/settings exist under ``CONFIG/PLATFORM/${PLATFORM}``. Additionally,
a platform specific environment may need to set with a different environment file.

Users are free to pull these files into the various ``build`` directories and
run the compilations manually (see information below). However for a setup run,
after the initial setup and calling of a configuration script, ``make_job`` or
``config-canesm``, these files get extracted into a local ``config`` directory,
according to the ``${PLATFORM}`` configuration variable, to easily be picked up
by ``compile-canesm.sh``. 

It should be noted that the ``make``/``mkmf`` templates are designed in such a way
that they can be used by all components/executables that require them. It is the combination
of these templates `with` each component specific ``Makefile`` that are used to
build the unique executables - **the templates are just meant to define the flags that
might be used for the platform/compiler combination**.

#############################
 Component Based Compilation
#############################

.. note::

    Depending on your platform, to compile any of these components, a specific environment might be required.
    If this is the case, unless otherwise stated, you should source the necessary ``compilation_environment`` 
    file underneath the provided ``PLATFORM`` link.

**********
Atmosphere
**********
The compilation of the Atmosphere component is executed within ``CanAM/build``, where the provided ``Makefile``
is used according to the following call:

.. code-block:: bash

   make MKMF_TEMPLATE=/path/to/mkmf/template CPP_CONFIG_FILE=/path/to/cppdef_config.h \
        CPP_SIZES_FILE=/path/to/cppdef_sizes.h

where:

   * ``MKMF_TEMPLATE`` is used to define the compiler flags
   * ``CPP_CONFIG_FILE`` defines the location of the desired ``cppdef_config.h`` file and
   * ``CPP_SIZES_FILE`` defines the location of the desired ``cppdef_sizes.h`` file

When called as part of ``compile-canesm.sh``, the above arguments are automatically selected, but if executing outside of a 
run, the ``MKMF_TEMPLATE`` can be retrieved from the desired ``PLATFORM`` directory, while the ``CPP_*_FILE``\s can be
generated according by mimicing the behaviour :ref:`here <Generating the cppdef files>`, or default files can be pulled from
``CanAM/build/include/.defaults``.

It should be noted that the above call is used to generate additional ``Makefile``\s using a provided ``mkmf.template``. The parent
``Makefile`` then calls ``make`` internally (again) to produce the required libraries/executables. To add/remove source files, 
to the Atmosphere compilation, developers are directed to the ``README`` file within the ``build`` directory.

********************************
Atmosphere Diagnostics (CanDIAG)
********************************
The compilation of Atmosphere diagnostics is achieved within ``CanDIAG/build``,
using a single ``Makefile`` and a specified ``make.template``, i.e.

.. code-block:: bash

   make COMPILER_TEMPLATE=/path/to/make/template

where ``COMPILER_TEMPLATE`` defines the ``make`` template that is used to
define the compilation flags. As noted above, when compiled as part of a run, a
suitable ``COMPILER_TEMPLATE`` is automatically selected by
``compile-canesm.sh``, but if a user wishes to compile the diagnostics outside
the context of a run, they can use the appropriate ``make.template*``
underneath the provided ``PLATFORM`` link.

It should be noted that due to the shear amount of programs under the
``CanDIAG`` suite, the provided ``Makefile`` generates the vast majority of the
compilation commands programatically according to three "phony" targets:

   * ``default_bins``
   * ``all_diag_bins``
   * ``plot_pgms``

and unless one of these targets is specified, the ``Makefile`` uses
``default_bins`` by default. For additional information on these targets and
how to add to the compilation see the ``README`` file within the ``build``
directory.

********
Coupler
********
The compilation of the Coupler component is executed within ``CanCPL/build``,
where the provided ``Makefile`` is used according to the following call:

.. code-block:: bash

   make MKMF_TEMPLATE=/path/to/mkmf/template CPP_CONFIG_FILE=/path/to/cppdef_config.h \
        CPP_SIZES_FILE=/path/to/cppdef_sizes.h

where:

   * ``MKMF_TEMPLATE`` is used to define the compiler flags
   * ``CPP_CONFIG_FILE`` defines the location of the desired ``cppdef_config.h`` file and
   * ``CPP_SIZES_FILE`` defines the location of the desired ``cppdef_sizes.h`` file

When called as part of ``compile-canesm.sh``, the above arguments are automatically selected, but if executing outside of a 
run, the ``MKMF_TEMPLATE`` can be retrieved from the desired ``PLATFORM`` directory, while the ``CPP_*_FILE``\s can be
generated according by mimicing the behaviour :ref:`here <Generating the cppdef files>`, or default files can be pulled from
``CanCPL/build/include/.defaults``.

It should be noted that the above call is used to generate additional ``Makefile``\s using a provided ``mkmf.template``. The parent
``Makefile`` then calls ``make`` internally (again) to produce the required libraries/executables. To add/remove source files, 
to the Coupler compilation, developers are directed to the ``README`` file within the ``build`` directory.

******
Ocean
******
As mentioned above, due to the self contained nature of the NEMO ocean model,
the building of ``CanNEMO`` is achieved via the ``makenemo`` tool. However,
developers will note the use of another layer of abstraction within
``compile-canesm.sh`` - ``build-nemo``. This additional layer is used because
``CanNEMO`` can be ran in "ocean-only" mode outside of the main ``CanESM``
infrastructure, and thus instead of having two separate build tools,
``compile-canesm.sh`` simply calls ``build-nemo`` to handle to compiling of the
Ocean executable and its diagnostics/utility programs. However, if a user
wishes to compile outside the context of a run, they can follow the guidance
below.

**Using build-nemo**

If a user wants to compile the ocean executable **and** the
diagnostics/utilities (see :ref:`here <Ocean Diagnostics/Utilities>`), they can
call ``build-nemo`` from anywhere (either by having it on their ``$PATH`` or
using the path to it), noting that it lives in ``CanNEMO/bin``. Interested readers
are directed to the output of ``build-nemo -h`` for comprehensive information
on it's interface, but as a simple example, it can be called via:

.. code-block:: bash

   build-nemo cfg=CCC_CANCPL_ORCA1_LIM_CMOC src_path=/path/to/CanESM/CanNEMO nemo_arch=intel.eccc-xc50 \
                util_arch=/path/to/util/make/tempalte comp_env=/path/to/environment_file

where:

   * ``cfg`` defines what source code gets compiled into the Ocean executable, according to ``nemo/CONFIG/cfg.txt``
   * ``src_path`` defines the location of the ``CanNEMO`` repo
   * ``nemo_arch`` is used to supply ``makenemo``'s ``-m`` flag and choose the ``arch-*.fcm`` file to use from ``nemo/ARCH/`` (when called as part of ``compile-canesm.sh`` this is generally set to ``COMPILER.PLATFORM``)
   * ``util_arch`` defines the ``make`` template to use in the compilation of the diagnostics/utilities, which can be pulled ``/path/to/CanESM/CONFIG/PLATFORM`` or ``/path/to/CanNEMO/nemo/ARCH``.
   * ``comp_env`` defines the environment file to source before compilation

**Using makenemo**

If a user `just` wants the ocean executable, they can navigate to ``/path/to/CanNEMO/nemo/CONFIG`` and call
``makenemo`` directly i.e.

.. code-block:: bash

   ./makenemo -m $nemo_arch -n $CanNEMO_CONFIG

where:

    * ``$nemo_arch`` defines the ``arch-*.fcm`` file to use from ``nemo/ARCH/``, and
    * ``$nemo-config`` defines what source code gets compiled into the executable, according ``nemo/CONFIG/cfg.txt``

For additional information on this tool, readers are directed to ``./makenemo -h``.

.. warning::

    Due to technical requirements for the coupling interface for coupled runs, the ocean executable needs to have
    code from ``CanCPL`` compiled in, which requires ``cppdef_config.h`` and ``cppdef_sizes.h`` to be in a specific
    location. Additionally, due to ``makenemo`` design, some additional consideration is 
    required for the extensions of the header files. Interested readers are directed to the ``build-nemo`` code.
    
    **Due to these complications**, it is recommended that developers limit
    themselves to compiling `non` ``CANCPL`` configs when doing so outside the
    context of a ``CanESM`` run.

***************************
Ocean Diagnostics/Utilities
***************************
The compilation of Ocean diagnostics can be lumped into three categories

   * Ocean diagnostics
   * Ocean run time diagnostics ("rtds"), which provide useful reductions of the ocean state
     to allow for easy inspection of intermediate model results, and
   * the ``rebuild_nemo`` tool, which rejoins ocean tiles

where the general ocean diagnostics are built in ``CanNEMO/diag/build``; the rtds built in ``CanNEMO/rtd/build``; and
``rebuild_nemo`` built in ``CanNEMO/nemo/TOOLS/REBUILD_NEMO/src/build``. 

For each of these groupings, the compilation is handled with a ``Makefile`` and
a ``make.template``, i.e.

.. code-block:: bash

   make COMPILER_TEMPLATE=/path/to/make/template

When compiled as part of a ``CanESM`` run, each of the above mentioned groupings are handled
automatically by the ``build-nemo`` tool (see :ref:`here <Ocean>` for additional information), 
which selects a ``COMPILER_TEMPLATE`` according to the ``util_arch`` argument. If a user wishes
to compile outside the context of a setup run, they can pull the desired template from either
``CONFIG/PLATFORM`` or ``CanNEMO/nemo/ARCH``.