Scripting Environments
======================

If all the dependencies for a model can be satisfied on the new platform, the next stage
is to install the necessary scripting environments used to configure, prep, and post process
the model.

.. contents::
   :local:
   :depth: 2

imsi environment
----------------

For the ``imsi`` based system, it require an environment that users can use to
get access to all the supporting codes/infrastructure. ``imsi`` is version controlled
`here <https://gitlab.com/cccma/imsi>`_, with its environment defined with
`this file <https://gitlab.com/cccma/imsi/-/blob/main/pyproject.toml?ref_type=heads>`_.

This environment can generally be simply installed using something like this:

.. code-block:: bash

   # clone the imsi code and check out desired version
   >> git clone git@gitlab.com:cccma/imsi.git /path/to/cloned/imsi
   >> ( cd /path/to/cloned/imsi ; git checkout <desired_commit|branch|tag> )

   # create a simple venv with the desired python
   #    version
   >> /path/to/desired/version/python -m venv /path/to/desired/env

   # activate environment
   >> source /path/to/desired/env/bin/activate

   # install imsi
   >> pip install /path/to/cloned/imsi

.. note::

   Once imsi is packaged and served via ``PyPI``, you will be able to ignore the clone step, and just install the desired ``imsi`` version
   via:

   .. code-block:: bash

      >> pip install imsi==X.Y.Z

cmorization environment
-----------------------

In order to support "cmorization", a python environment is required to run the
``pycmor`` `codebase <https://gitlab.com/cccma/cccma_tools/-/tree/develop_canesm/data_conv/ncconv/pycmor?ref_type=heads>`_.
Unfortunately, the `easiest` way to install these environments is via ``conda`` - specifically, with
`this file <https://gitlab.com/cccma/cccma_tools/-/blob/develop_canesm/data_conv/ncconv/pycmor/cmor.yml?ref_type=heads>`_.
This is because the `cmor package <https://github.com/PCMDI/cmor>`_, which is the back-bone of ``pycmor`` doesn't
have a ``pip`` installer. 

On systems that don't allow ``conda`` usage, ``mamba`` `can be used <https://cmor.llnl.gov/mydoc_cmor3_conda/>`_, or
the ``cmor`` dependency could `be built from source <https://cmor.llnl.gov/mydoc_cmor3_github/>`_. Outside of the ``cmor`` dependency,
the other packages can likely be installed via ``pip`` directly.

.. note::

   On ``niagara``, with the ``v5.1_cp4c`` branch, we are still utilizing a ``python2`` version of
   the cmorization software. Additionally, the ``niagara`` sys-admins very much don't want users
   to make use of personal ``conda`` installs. As such, building the supporting environment on ``niagara``
   was painful. Nevertheless, Clint built one here:

   .. code-block::

      /project/c/cp4c/cp4c/pyenvs/pyenv-ncconv-0.1

   using this directory as the build location

   .. code-block::

      /project/c/cp4c/cp4c/canesm_deps/pycmor

   Clint attempted to document the process there, but for posterity, includes some pertinent files here:

   .. code-block::

      #README.md

      This directory contains everything necessary to build a pycmor supporting (using python 2.7)
      virtualenv on DRAC machines. However, getting this to work had a few hiccups:

      ### Python Version and the CCCma Format Problem
      Since the atmosphere model _still_ writes out CCCma format, instead of something like netcdf, we require
      a method to read in CCCma format files into python, which is achieved through an old python library
      that (binary format) interacts with old CCCma tools and acts as an interface between fortran and python.
      This module was written in python2.7 and would require entirely new bindings to upgrade to python3 (work is
      scheduled to do this, but it will take some time)

      As such, we need to do some painful gymnastics here to create a python 2.7 environment.

      The use of this "cccma_py.so" module also means that we have a dependence on libgfortran.so.3 (note the 3). This
      isn't available under NiaEnv/2022a, so we need to carry a version of this around here and add its location
      manually to LD_LIBRARY_PATH

      ### Generating a requirements.txt file
      For reference the cmor.yml file has been provided here, but it can't be used to install. We need
      a requirements.txt file that can be used by pip. As such, Clint Seinen generated one by activating
      the environment on the science network, and running

          $ pip freeze > requirements.txt

      However, this ended up including some modules that aren't truly required and couldn't be found by pip.
      As such, Clint removed them. For reference the original requirements file is provided as
      pycmor-requirements-science-work.txt, while the modified version that is used is pycmor-requirements.txt

      ### Installing CMOR and jsonc
      On the ECCC science network, conda is utilized to serve the necessary environment but conda is considered
      problematic on DRAC machines, and virtualenv is the desired environment solution. However, this complicates
      matters because CMOR is only provided via conda, or by building from source - see here:

      https://cmor.llnl.gov/mydoc_cmor3_conda/

      As such, to use CMOR (and its dependency jsonc) we need to build them from source.

      NOTE: json-c needs to have its library added to LD_LIBRARY_PATH

      ### Setting LD_LIBRARY_PATH

      To properly pickup the following libraries:
        - libgfortran.so.3 and
        - json-c's library

      these need to be added to LD_LIBRARY_PATH when the environment is activated, as such, the
      build-env.sh script adds to the environment's 'activate' script to do so.

      NOTE: it also makes it so the 'deactivate' function (within 'activate' script) pulls theses
      out of LD_LIBRARYPATH

      ### PYCMOR_DEPS

      An additional dependency, from the legacy cccma_py module (see above), is on the PYCMOR_DEPS
      variable, which amor.py uses to find 'cccma_py'. On other platforms, for non-imsi systems, we've
      set this variable in a "site-profile", but in this instance we've made it so the activate/deactivate
      script/function handles the setting/clearing of this.

      The 'build-env.sh' script handles this.

   .. code-block:: bash

      # build-env.sh
      #!/bin/bash
      # THIS IS A SCRIPT THAT IS USED TO BUILD A PYCMOR2 SUPPORTING VIRTUAL ENV.
      # NOTE THAT THIS SCRIPT IS MOSTLY TO DOCUMENT THE BUILD PROCESS AND IF NEW VERSIONS
      # OF JSON-C OR CMOR NEED TO BE BUILT, THE SCRIPT WILL LIKELY NEED TO BE ADAPTED
      #
      # AUTHOR: Clint Seinen clint.seinen@ec.gc.ca
      #
      #   When using this script, it builds all the dependencies in _this_ directory, and it will clear
      #   old directories/files associated with the CMOR and JSONC downloads
      set -ex

      #~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      # Set configuration vars!
      #
      #   env_name : the name of the python environment
      #
      #   env_location : where the enviornment directory will be stored
      #
      #   modules_for_build : defines what modules will be loaded when json-c and cmor are built from source.
      #                           Upon guidance from DRAC support, we built CMOR with the netcdf libraries
      #                           that will be loaded when we try to use it.
      #
      #   modules_for_env_init : defines the modules that will be loaded to initate the python environment.
      #                           These need to be different from modules_for_build so we can get a python2.7
      #                           module. Note that if the modules within modules_for_build have a supporting
      #                           python2.7 module, this would NOT be needed.
      #
      #   jsonc_version : the version of jsonc that will support CMOR, that will be built from source
      #
      #   cmor_version : the version of cmor that will be built from source
      #
      #~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      env_name="pyenv-ncconv-0.1"
      env_location="/project/c/cp4c/cp4c/pyenvs"
      modules_for_build="NiaEnv/2022a intel/2022u2 ucx/1.11.2 intelmpi/2022u2+ucx-1.11.2 hdf5/1.10.9 netcdf/4.9.0 cmake cdo udunits/2.2.28"
      modules_for_env_init="NiaEnv/2019b python/2.7.15"
      jsonc_version="0.13.1-20180305"
      cmor_version="3.5.0"

      #~~~~~~~~~~~~~~~~~~~~~~~~
      # Define local locations
      #~~~~~~~~~~~~~~~~~~~~~~~~
      install_dir=$(pwd)
      lib_dir=$(pwd)/lib
      pycmor_deps_dir=$(pwd)/pycmor_deps

      #~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      # Initiate python environment and activate it
      #~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      for mod in $modules_for_env_init; do
          module load $mod
      done
      ( cd $env_location; virtualenv $env_name; )
      source ${env_location}/${env_name}/bin/activate
      pip install numpy six

      #~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      # Build the libraries from source
      #~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      # load libs for building of libs
      for mod in $modules_for_build; do
          module load $mod
      done

      #-- JSON-C
      # get and install json-c and add its locations to required path vars
      rm -rf json-c*
      wget https://github.com/json-c/json-c/archive/refs/tags/json-c-${jsonc_version}.tar.gz
      tar -zxf json-c-${jsonc_version}.tar.gz
      cd json-c-json-c-${jsonc_version}
      rm -rf build install
      mkdir build install
      cd build
      cmake .. -DCMAKE_INSTALL_PREFIX=$(pwd)/../install
      make install
      cd ..
      CPATH=$(pwd):$CPATH
      cd install/include
      CPATH=$(pwd):$CPATH
      cd ../lib
      LIBRARY_PATH=$(pwd):$LIBRARY_PATH
      LD_LIBRARY_PATH=$(pwd):$LD_LIBRARY_PATH
      JSONC_LIB_PATH=$(pwd)

      #-- CMOR
      cd ${install_dir}
      rm -rf cmor-${cmor_version}
      rm -rf ${cmor_version}.tar.gz
      wget https://github.com/PCMDI/cmor/archive/refs/tags/${cmor_version}.tar.gz
      tar -zxf ${cmor_version}.tar.gz
      cd cmor-${cmor_version}
      mkdir install
      ./configure --prefix=$(pwd)/install --with-python
      make install || : # apparently we accept a failure here...

      # install into this environment
      pip install .

      #~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      # Install the remaining python modules
      #~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      cd ${install_dir}
      pip install -r pycmor-requirements.txt

      #~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      # Add extra env considerations to activate script
      #~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      echo "export LD_LIBRARY_PATH=\${LD_LIBRARY_PATH}:${JSONC_LIB_PATH}" >> ${env_location}/${env_name}/bin/activate
      echo "export LD_LIBRARY_PATH=\${LD_LIBRARY_PATH}:${lib_dir}" >> ${env_location}/${env_name}/bin/activate
      echo "export PYCMOR_DEPS=${pycmor_deps_dir}" >> ${env_location}/${env_name}/bin/activate

      #~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      # Add extra env considerations to deactivate function
      #   - remove LD_LIBRARY_PATH entries
      #   - unset PYCMOR_DEPS
      #~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      # find the starting line of the function
      deactivate_function_start_line=$(grep -n 'deactivate () {' ${env_location}/${env_name}/bin/activate | awk -F ':' '{print $1}')
      inject_line=$(( deactivate_function_start_line + 1 ))
      sed -i "${inject_line}i"'export LD_LIBRARY_PATH=$( tr : $\x27\\n\x27 <<< $LD_LIBRARY_PATH | grep -v'" $JSONC_LIB_PATH "'| paste -s -d : )' ${env_location}/${env_name}/bin/activate
      sed -i "${inject_line}i"'export LD_LIBRARY_PATH=$( tr : $\x27\\n\x27 <<< $LD_LIBRARY_PATH | grep -v'" $lib_dir "'| paste -s -d : )' ${env_location}/${env_name}/bin/activate
      sed -i "${inject_line}i"'unset PYCMOR_DEPS' ${env_location}/${env_name}/bin/activate

      #~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      # Make the environment sharable by making it readable
      #   and executable to all
      #~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      deactivate
      chmod -R a+rx ${env_location}/${env_name}

   .. code-block::

      # pycmor-requirements.txt
      asn1crypto==0.24.0
      backports-abc==0.5
      backports.shutil-get-terminal-size==1.0.0
      bokeh==1.0.4
      certifi==2018.11.29
      cffi==1.11.5
      cftime==1.0.3.4
      chardet==3.0.4
      Click==7.0
      cloudpickle==0.8.0
      CMOR==3.5.0
      cryptography==2.4.2
      cytoolz==0.9.0.1
      dask==1.1.2
      decorator==4.3.0
      distributed==1.26.0
      enum34==1.1.6
      et-xmlfile==1.0.1
      future==0.17.1
      futures==3.2.0
      heapdict==1.0.0
      idna==2.8
      ipaddress==1.0.22
      ipython==5.8.0
      ipython-genutils==0.2.0
      jdcal==1.4
      Jinja2==2.10
      locket==0.2.0
      MarkupSafe==1.1.1
      msgpack==0.6.1
      netCDF4==1.5.1.2
      numpy==1.16.0
      olefile==0.46
      openpyxl==2.5.12
      packaging==19.0
      pandas==0.23.4
      partd==0.3.9
      pathlib2==2.3.3
      pexpect==4.6.0
      pickleshare==0.7.5
      Pillow==5.4.1
      prompt-toolkit==1.0.15
      psutil==5.5.0
      ptyprocess==0.6.0
      pycparser==2.19
      Pygments==2.3.1
      pyOpenSSL==18.0.0
      pyparsing==2.3.1
      PySocks==1.6.8
      python-dateutil==2.7.5
      pytz==2018.9
      PyYAML==3.13
      requests==2.21.0
      scandir==1.9.0
      simplegeneric==0.8.1
      singledispatch==3.4.0.3
      six==1.12.0
      sortedcontainers==2.1.0
      tblib==1.3.2
      toolz==0.9.0
      tornado==5.1.1
      traitlets==4.3.2
      urllib3==1.24.1
      wcwidth==0.1.7
      xarray==0.11.0
      zict==0.1.3

classic rtd environment
-----------------------

Work was recently done to add a ``python`` based rtd program for ``CLASSIC`` variables specifically. Nic Annau worked with Vivek and Luke Grant
in order to integrate this into internal ``CanESM5.3`` and ``CanESM6.0`` versions - as these versions require a classic rtd environment.

On ``eccc-u2`` this enviornment is currently provided as part of our centralized ``python`` `environments <https://gitlab.science.gc.ca/CanESM/Python_Envs>`_.

.. note::

   Due to the higher memory load of ``CanESM6.0``, extra work was done to reduce the memory footprint of the ``classic-rtd``
   job. This is documented `here <https://gitlab.science.gc.ca/CanESM/CanESM5/-/issues/549>`_. As a result of this, there
   are differences between the ``develop_canesm`` (``CanESM5.3``) and ``v6.0_release`` (``CanESM6.0``) versions.

.. note::

   The classic rtds currently rely on a compiled version of ``nccrip`` in order to convert ``ccc`` files into
   ``netcdf`` files before loading and running the analysis. Once work is done to have ``netcdf`` files available
   by default, work should be done to adjust those tools.
.. note::


   **This is only required for CanESM versions >= 5.3.**

agcm diagnostic environment
---------------------------

For ``CanESM6.0``, as of July '25, work is being done to integrate a new ``AGCM`` diagnostic pipeline, which involves
the use of the new ``canesm-processor`` package. As such, we have installed a centralized environment for it - see
`here <https://gitlab.science.gc.ca/CanESM/Python_Envs>`_ for details.

.. note::

   **This is only required for CanESM versions >= 5.3.**

cdo
---

With the use of netcdf files within the ``CanESM`` ecosystem, users/devs often write scripting to do simple file manipulations
using ``cdo`` (see `here <https://code.mpimet.mpg.de/projects/cdo>`_). Notably, to run some of the ocean diagnostics,
``cdo`` must be available - see below for version details and how this has been loaded on different machines.

eccc-u2
~~~~~~~

.. code-block::

   >> . ssmuse-sh -x eccc/cmd/cmds/ext/20220331
   >> cdo --version
   Climate Data Operators version 2.0.3 (https://mpimet.mpg.de/cdo)
   System: x86_64-pc-linux-gnu
   CXX Compiler: c++ -std=gnu++14 -march=x86-64 -mtune=generic -O2 -pipe -fopenmp -pthread
   CXX version : unknown
   C Compiler: gcc -march=x86-64 -mtune=generic -O2 -pipe -fopenmp -pthread -pthread
   C version : gcc (GCC) 8.3.1 20191121 (Red Hat 8.3.1-5)
   F77 Compiler: gfortran -g -O2
   F77 version : GNU Fortran (GCC) 8.3.1 20191121 (Red Hat 8.3.1-5)
   Features: 992GB 80threads C++14 OpenMP45 Fortran PTHREADS HDF5 NC4/HDF5 OPeNDAP SZ PROJ XML2 CURL FFTW3 SSE2
   Libraries: HDF5/1.10.5 proj/6.3.2 xml2/2.9.7 curl/7.61.1
   CDI data types: SizeType=size_t  DateType=int64_t
   CDI file types: srv ext ieg grb1 grb2 nc1 nc2 nc4 nc4c nc5
        CDI library version : 2.0.3
    cgribex library version : 2.0.0
    ecCodes library version : 2.24.2
     NetCDF library version : 4.7.0 of Aug 15 2019 22:03:10 $
       hdf5 library version : 1.10.5
       exse library version : 1.4.2
       FILE library version : 1.9.1

niagara
~~~~~~~

.. code-block::

   >> module load CCEnv arch/avx512 StdEnv/2020 cdo/2.0.5
   >> cdo --version
   Climate Data Operators version 2.0.5 (https://mpimet.mpg.de/cdo)
   System: x86_64-pc-linux-gnu
   CXX Compiler: mpicxx -std=gnu++14 -O2 -xCore-AVX512 -ftz -fp-speculation=safe -fp-model source -fPIC -fopenmp -pthread
   CXX version : Intel(R) C++ Intel(R) 64 Compiler for applications running on Intel(R) 64, Version 19.1.1.217 Build 20200306
   C Compiler: mpicc -O2 -xCore-AVX512 -ftz -fp-speculation=safe -fp-model source -fPIC -fopenmp -pthread -pthread
   C version : icc (ICC) 19.1.1.217 20200306
   F77 Compiler: mpifort -O2 -xCore-AVX512 -ftz -fp-speculation=safe -fp-model source -fPIC
   F77 version : ifort (IFORT) 19.1.1.217 20200306
   Features: 93GB 40threads C++14 OpenMP45 Fortran PTHREADS HDF5 NC4/HDF5 OPeNDAP PROJ AVX2
   Libraries: HDF5/1.10.6 proj/9.0.0
   CDI data types: SizeType=size_t  DateType=int64_t
   CDI file types: srv ext ieg grb1 grb2 nc1 nc2 nc4 nc4c nc5
        CDI library version : 2.0.5
    cgribex library version : 2.0.1
    ecCodes library version : 2.25.0
     NetCDF library version : 4.7.4 of May 14 2020 00:07:21 $
       hdf5 library version : 1.10.6
       exse library version : 1.4.2
       FILE library version : 1.9.1