Update the chapter on Adding a new scheme to CCPP (#68)

This chapter more thoroughly describes the process of adding a new scheme to CCPP. Includes instructions on logistics (adapting scheme to CCPP-compliance vs using a driver), adding new variables, and debugging a scheme once added. Also included a new glossary definition for "entry point".

This PR also includes a new .readthedocs.yaml file, which is now required for auto-builds on ReadTheDocs

.readthedocs.yaml

@@ -0,0 +1,22 @@
+# .readthedocs.yaml
+# Read the Docs configuration file
+# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
+
+# Required
+version: 2
+
+# Set the version of Python and other tools you might need
+build:
+  os: ubuntu-22.04
+  tools:
+    python: "3.11"
+
+# Build documentation in the docs/ directory with Sphinx
+sphinx:
+  configuration: CCPPtechnical/source/conf.py
+
+# We recommend specifying your dependencies to enable reproducible builds:
+# https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html
+python:
+  install:
+  - requirements: CCPPtechnical/requirements.txt

CCPPtechnical/source/AddingNewSchemes.rst

@@ -1,65 +1,110 @@
 .. _AddNewSchemes:
 
 ****************************************
-Tips for Adding a New Scheme
+Connecting a scheme to CCPP
 ****************************************
 
-This chapter contains a brief description on how to add a new :term:`scheme` to the :term:`CCPP Physics` pool.
+This chapter contains a brief description on how to add a :term:`scheme` to the :term:`CCPP Physics` pool. Aside from the basic design elements for interoperability (:term:`entry points <entry point>`, metadata files), most CCPP requirements simply follow from coding best practices.
 
-* Identify the variables required for the new scheme and check if they are already available for use in the CCPP by checking the metadata information in ``GFS_typedefs.meta`` or by perusing file ``ccpp-framework/doc/DevelopersGuide/CCPP_VARIABLES_{FV3,SCM}.pdf`` generated by ``ccpp_prebuild.py``.
+     .. note:: The instructions in this chapter assume the user is implementing this scheme for use with the CCPP Single-Column model (:term:`SCM`); not only is the SCM more lightweight than a full 3D NWP model for development purposes, but using the SCM as a :term:`host model` is a requirement for all new CCPP schemes for testing purposes. For implementation in another host model, especially for adding new variables, some modifications to that host model's metadata may be required; see :numref:`Chapter %s <Host-side Coding>` for details
 
-    * If the variables are already available, they can be invoked in the scheme’s metadata file and one can skip the rest of this subsection. If the variable required is not available, consider if it can be calculated from the existing variables in the CCPP. If so, an :term:`interstitial scheme` (such as ``scheme_pre``; see more in :numref:`Chapter %s <CompliantPhysParams>`) can be created to calculate the variable. However, the variable must be defined but not initialized in the :term:`host model` as the memory for this variable must be allocated on the host model side.  Instructions for how to add variables to the host model side is described in :numref:`Chapter %s <Host-side Coding>`.
+==============================
+Preparing a scheme for CCPP
+==============================
+There are a few steps that can be taken to prepare a scheme for addition to CCPP prior to starting the process of implementing it in the CCPP Framework:
 
-     .. note:: The :term:`CCPP framework` is capable of performing automatic unit conversions between variables provided by the host model and variables required by the new scheme. See :numref:`Section %s <AutomaticUnitConversions>` for details.
+1. Remove/refactor any incompatible features described in :numref:`Section %s <CodingRules>`. This includes updating Fortran code to at least Fortran 90 standards, removing STOP and GOTO statements, removing common blocks, and refactoring any other disallowed features.
+2. Make an inventory of all variables that are inputs and/or outputs to the scheme. Check the file ``ccpp-framework/doc/DevelopersGuide/CCPP_VARIABLES_SCM.pdf`` to see if each variable has already been implemented in the single column model. If there are variables that are not available, see :numref:`Section %s <Adding new variables to CCPP>`.
 
-    * If new namelist variables need to be added, the ``GFS_control_type`` DDT should be used. In this case, it is also important to modify the namelist file ``input.nml`` to include the new variable.
+=============================
+Implementing a scheme in CCPP
+=============================
 
-    * It is important to note that not all data types are persistent in memory. Most variables in the interstitial data type are reset (to zero or other initial values) at the beginning of a physics :term:`group` and do not persist from one :term:`set` to another or from one group to another. The diagnostic data type is periodically reset because it is used to accumulate variables for given time intervals.  However, there is a small subset of interstitial variables that are set at creation time and are not reset; these are typically dimensions used in other interstitial variables. 
+There are, broadly speaking, two approaches for connecting an existing physics scheme to the CCPP Framework: 
 
-     .. note:: If the value of a variable must be remembered from one call to the next, it should not be in the interstitial or diagnostic data types.
+1. Refactor the existing scheme to CCPP format standards, using ``pre_`` and ``post_`` :term:`interstitial schemes <interstitial scheme>` to interface to and from the existing scheme if necessary.
+2. Create a driver scheme as an interface from the existing scheme's Fortran module to the CCPP Framework.
 
-    * If information from the previous timestep is needed, it is important to identify if the host model readily provides this information. For example, in the Model for Prediction Across Scales (MPAS), variables containing the values of several quantities in the preceding timesteps are available. When that is not the case, as in the :term:`UFS Atmosphere`, interstitial schemes are needed to compute these variables. As an example, the reader is referred to the `GF convective scheme <https://dtcenter.ucar.edu/GMTB/v6.0.0/sci_doc/_c_u__g_f.html>`_, which makes use of interstitials to obtain the previous timestep information.
+.. figure:: _static/ccpp_scheme_diagram.png
 
-    * Consider allocating the new variable only when needed (i.e. when the new scheme is used and/or when a certain control flag is set). If this is a viable option, following the existing examples in ``GFS_typedefs.F90`` and ``GFS_typedefs.meta`` for allocating the variable and setting the ``active`` attribute in the metadata correctly.
+   *Diagram of the methods described in this section.*
 
-* If an entirely new variable needs to be added, consult the CCPP :term:`standard names<standard name>` dictionary and the rules for creating new standard names at https://github.com/escomp/CCPPStandardNames. If in doubt, use the GitHub discussions page in the CCPP Framework repository (https://github.com/ncar/ccpp-framework) to discuss the suggested new standard name(s) with the CCPP developers.
+Method 1 is the preferred method of adapting a scheme to CCPP. This involves making modifications to the original scheme so that it is CCPP-compliant (see :numref:`Chapter %s <CompliantPhysParams>`), containing subroutines that correspond to CCPP entry points (i.e. ``schemename_init``, ``schemename_run``, etc.) as necessary. It should be accompanied by appropriate metadata files (see :numref:`Section %s <MetadataRules>`), and it must be updated to remove any disallowed features as listed in :numref:`Section %s <CodingRules>`.
 
-* Examine scheme-specific and :term:`suite` interstitials to see what needs to be replaced/changed; then check existing scheme interstitial and determine what needs to replicated. Identify if your new scheme requires additional interstitial code that must be run before or after the scheme and that cannot be part of the scheme itself, for example because of dependencies on other schemes and/or the order the scheme is run in the :term:`SDF`.
+While method 1 is preferred, there are cases where method 1 may not be possible: for example, in schemes that are shared with other, non-CCPP hosts, and so require specialized, model-specific drivers, and might be beholden to different coding standards required by another model. In cases such as this, method 2 may be employed.
 
-* Follow the guidelines outlined in :numref:`Chapter %s <CompliantPhysParams>` to make your scheme CCPP-compliant. Make sure to use an uppercase suffix ``.F90`` to enable C preprocessing.
+Method 2 involves fewer changes to the original scheme's Fortran module: A CCPP-compliant driver module (see :numref:`Chapter %s <CompliantPhysParams>`) handles defining the inputs to and outputs from the scheme module in terms of state variables, constants, and tendencies provided by the model as defined in the scheme's .meta file. The calculation of variables that are not available directly from the model, and conversion of scheme output back into the variables expected by CCPP, should be handled by interstitial schemes (``schemename_pre`` and ``schemename_post``). While this method puts most CCPP-required features in the driver and interstitial subroutines, the original scheme must still be updated to remove STOP statements, common blocks, or any other disallowed features as listed in :numref:`Section %s <CodingRules>`. 
 
-* Locate the CCPP *prebuild* configuration files for the target host model, for example:
+For both methods, optional interstitial schemes can be used for code that can not be handled within the scheme itself. For example, if different code needs to be run for coupling with other schemes or in different orders (e.g. because of dependencies on other schemes and/or the order the scheme is run in the :term:`SDF`), or if variables needed by the scheme must be derived from variables provided by the host. See  :numref:`Chapter %s <CompliantPhysParams>` for more details on primary and interstitial schemes.
 
-    * ``ufs-weather-model/FV3/ccpp/config/ccpp_prebuild_config.py`` for the :term:`UFS Atmosphere`
-    * ``ccpp-scm/ccpp/config/ccpp_prebuild_config.py`` for the :term:`SCM`
+     .. note:: Depending on the complexity of the scheme and how it works together with other schemes, multiple interstitial schemes may be necessary. 
 
-* Add the new scheme to the Python dictionary in ``ccpp_prebuild_config.py`` using the same path
-  as the existing schemes:
+------------------------------
+Adding new variables to CCPP
+------------------------------
 
-  .. code-block:: console
+This section gives guidance on adding new variables to the CCPP, which is often necessary when adding a new scheme or adding capabilities to an existing one.
 
-    SCHEME_FILES = [ ...
-    ’../some_relative_path/existing_scheme.F90’,
-    ’../some_relative_path/new_scheme.F90’,
-    ...]
+     .. note:: The instructions in this chapter assume the user is implementing this scheme for use in the CCPP Single-Column model (SCM). Other host model variables can be found in different files; see :numref:`Chapter %s <Host-side Coding>` for details
 
-* Place new scheme in the same location as existing schemes in the CCPP directory structure, e.g., ``../some_relative_path/new_scheme.F90``.
+The first step is to be absolutely sure that a new variable is required: the desired variable may already be included in the CCPP for use by other schemes. Check the metadata information in ``CCPP_typedefs.meta`` or the file ``ccpp-framework/doc/DevelopersGuide/CCPP_VARIABLES_SCM.pdf`` generated by ``ccpp_prebuild.py``. If all quantities needed by the scheme are already available as variables in CCPP, they can be invoked in the scheme's metadata file without any further work necessary.
 
-* Edit the SDF and add the new scheme at the place it should be run. SDFs are located in
+If an input variable needed by the scheme is not available, first consider if it can be calculated from the existing CCPP variables. If so, an :term:`interstitial scheme` (such as ``schemename_pre``; see  :numref:`Chapter %s <CompliantPhysParams>` for more details) can be created to calculate the variable(s). If this path is taken, the variable must be defined (but not initialized) in the :term:`host model`, as the memory for this variable must be allocated by the host. Instructions for how to add variables on the host model side can be found in :numref:`Chapter %s <Host-side Coding>`.
 
-    * ``ufs-weather-model/FV3/ccpp/suites`` for the UFS Atmosphere
-    * ``ccpp-scm/ccpp/suites`` for the SCM
+     .. note:: The CCPP Framework is capable of performing automatic unit conversions between variables provided by the host model and variables required by the new scheme. See :numref:`Section %s <AutomaticUnitConversions>` for details.
 
-* Before running, check for consistency between the namelist and the SDF. There is no default consistency check between the SDF and the namelist unless the developer adds one. Errors may result in segmentation faults in running something you did not intend to run if the arrays are not allocated.
+If an entirely new variable needs to be added, consult the CCPP standard names dictionary and the rules for creating new :term:`standard names <standard name>` at https://github.com/escomp/CCPPStandardNames. If in doubt, use the GitHub discussions page in the CCPP Framework repository (https://github.com/ncar/ccpp-framework) to discuss the suggested new standard name(s) with the CCPP developers.
 
-* Test and debug the new scheme:
+     .. note:: It is important to keep in mind that not all data types are persistent in memory. If the value of a variable must be remembered from one call to the next, it should not be in the interstitial or diagnostic data types. Most variables in the interstitial data type are reset (to zero or other initial values) at the beginning of a physics :term:`group` and do not persist from one :term:`set` to another or from one group to another. The diagnostic data type is periodically reset because it is used to accumulate variables for given time intervals. However, there is a small subset of interstitial variables that are set at creation time and are not reset; these are typically dimensions used in other interstitial variables.
 
-    * Typical problems include segmentation faults related to variables and array allocations.
-    * Make sure the SDF and namelist are compatible. Inconsistencies may result in segmentation faults because arrays are not allocated or in unintended scheme(s) being executed.
-    * A scheme called GFS_debug (``GFS_debug.F90``) may be added to the SDF where needed to print state variables and interstitial variables. If needed, edit the scheme beforehand to add new variables that need to be printed.
-    * Check the *prebuild* script for success/failure and associated messages; run the *prebuild* script with the `--debug` and `--verbose` flags.
-    * Compile code in DEBUG mode, run through debugger if necessary (gdb, Allinea DDT, totalview, ...).
-    * Use memory check utilities such as valgrind.
-    * Double-check the metadata file associated with your scheme to make sure that all information, including standard names and units, correspond to the correct local variables.
+For variables that can be set via namelist, the ``GFS_control_type`` Derived Data Type (DDT) should be used. In this case, it is also important to modify the namelist file to include the new variable.
+
+If information from the previous timestep is needed, it is important to identify if the host model provides this information, or if it needs to be stored as a special variable. For example, in the Model for Prediction Across Scales (MPAS), variables containing the values of several quantities in the preceding timesteps are available. When that is not the case, as in the :term:`UFS Atmosphere`, interstitial schemes are needed to access these quantities.
+
+     .. note:: As an example, the reader is referred to the `GF convective scheme <https://dtcenter.ucar.edu/GMTB/v6.0.0/sci_doc/_c_u__g_f.html>`_, which makes use of interstitials to obtain the previous timestep information. 
+
+Consider allocating the new variable only when needed (i.e. when the new scheme is used and/or when a certain control flag is set). If this is a viable option, following the existing examples in ``CCPP_typedefs.F90`` and ``GFS_typedefs.meta`` for allocating the variable and setting the ``active`` attribute in the metadata correctly.
+
+------------------------------
+Adding a new scheme to CCPP
+------------------------------
+The new scheme and any interstitials will need to be added to the CCPP prebuild configuration file. Add the new scheme to the Python dictionary in ``ccpp-scm/ccpp/config/ccpp_prebuild_config.py`` using the same path as the existing schemes:
+
+.. code-block:: 
+
+   SCHEME_FILES = [ ...
+   '../some_relative_path/existing_scheme.F90',
+   '../some_relative_path/new_scheme.F90',
+   ...]
+
+     .. note:: Different host models will have different prebuild config files. For example, the :term:`UFS Atmosphere's <UFS Atmosphere>` config file is located at ``ufs-weather-model/FV3/ccpp/config/ccpp_prebuild_config.py`` 
+
+The source code and ``.meta`` files for the new scheme should be placed in the same location as existing schemes in the CCPP: in the ccpp-physics repository under the ``physics/`` directory.
+
+To add this new scheme to a suite definition file (:term:`SDF`) for running within a :term:`host model`, follow the examples found in ``ccpp-scm/ccpp/suites``. For more information about suites and SDFs, see :numref:`Chapter %s <ConstructingSuite>`.
+
+     .. note:: For the :term:`UFS Atmosphere`, suites can be found in the ``ufs-weather-model/FV3/ccpp/suites`` directory
+
+No further modifications of the build system are required, since the :term:`CCPP Framework` will auto-generate the necessary makefiles that allow the host model to compile the scheme.
+
+
+
+==================================
+Testing and debugging a new scheme
+==================================
+
+Before running this new scheme, check for consistency between the namelist and the :term:`SDF`. There is no default consistency check between the SDF and the namelist unless the developer adds one. Errors such as segmentation faults may occur if this consistency is not upheld, due to appropriate arrays not being allocated.
+
+To test a new scheme that has been added to the :term:`SCM`, compile the SCM with a suite definition file that contains the newly added scheme.
+
+Some tips for debugging problems:
+
+* Segmentation faults are often related to variables and array allocations.
+* As mentioned above, make sure the SDF and namelist are compatible. Inconsistencies may result in segmentation faults because arrays are not allocated or in unintended scheme(s) being executed.
+* Make sure to use an uppercase suffix ``.F90`` to enable C preprocessing.
+* A scheme called GFS_debug (GFS_debug.F90) may be added to the SDF where needed to print state variables and interstitial variables. If needed, edit the scheme beforehand to add new variables that need to be printed.
+* Check the ``ccpp_prebuild.py`` script for success/failure and associated messages; run the prebuild script with the --debug and --verbose flags. See :numref:`Chapter %s <ConstructingSuite>` for more details
+* Compile code in DEBUG mode (see section 2.3 of the `SCM User's Guide <https://github.com/NCAR/ccpp-scm/blob/main/scm/doc/TechGuide/main.pdf>`_, run through debugger if necessary (gdb, Allinea DDT, totalview, …).
+* Use memory check utilities such as ``valgrind``.
+* Double-check the metadata file associated with your scheme to make sure that all information, including standard names and units, correspond to the correct local variables.
 
-* Done. Note that no further modifications of the build system are required, since the *CCPP Framework* will autogenerate the necessary makefiles that allow the host model to compile the scheme.

CCPPtechnical/source/Glossary.rst

@@ -16,6 +16,9 @@ Glossary
       The pool of CCPP-compliant physics schemes; also refers to a `software repository of the same name
       <https://github.com/NCAR/ccpp-physics>`_
 
+   Entry point
+      An entry point is a subroutine for one :term:`phase` of CCPP physics that is called by the :term:`host model`. Entry points are described in more detail in :numref:`Section %s <GeneralRules>`
+
    Fast physics
       Physical parameterizations that require tighter coupling with the dynamical core than “slow”
       physics (due to the approximated processes within the parameterization acting on a shorter