Add New Coupling Option of LND (Noah-MP) and ATM (FV3)

.github/pull_request_template.md

@@ -9,6 +9,7 @@ Provide a detailed description of what this PR does. What bug does it fix, or wh
 ## Subcomponents involved:
 - [ ] apply_incr.fd (NOAA-PSL/land-apply_jedi_incr)
 - [ ] ufs_model.fd (ufs-community/ufs-weather-model)
+- [ ] UFS_UTILS.fd (ufs-community/UFS_UTILS)
 - [ ] none
 
 ## Linked PR's and Issues:

.gitignore

@@ -6,14 +6,8 @@ sorc/conda/
 sorc/lib/
 sorc/lib64
 parm/conda_loc
-parm/*.xml
-parm/*.db
-parm/*.out
-parm/*.err
-parm/*.log
-parm/parm_xml.yaml
-parm/land_analysis.yaml
-parm/log.*
+parm/config.yaml
+ush/__pycache__/
 
 __pycache__
 *.swp

.gitmodules

@@ -2,8 +2,13 @@
 	path = sorc/ufs_model.fd
 	url = https://github.com/ufs-community/ufs-weather-model.git
 	ignore = dirty
+
 [submodule "sorc/apply_incr.fd"]
 	path = sorc/apply_incr.fd
 	url = https://github.com/NOAA-PSL/land-apply_jedi_incr.git
         ignore = dirty
 
+[submodule "sorc/UFS_UTILS.fd"]
+	path = sorc/UFS_UTILS.fd
+	url = https://github.com/ufs-community/UFS_UTILS.git
+	ignore = dirty

README.md

@@ -1,12 +1,12 @@
-# UFS Offline Land Data Assimilation System
+# UFS Land Data Assimilation System
 
 The Unified Forecast System (UFS) is a community-based, coupled, comprehensive Earth modeling system. It is designed to be the source system for NOAA's operational numerical weather prediction applications while enabling research, development, and contribution opportunities for the broader Weather Enterprise. For more information about the UFS, visit the UFS Portal at https://ufs.epic.noaa.gov/.
 
-The UFS includes [multiple applications](https://ufs.epic.noaa.gov/applications/) that support different forecast durations and spatial domains. This repository hosts the source code for the UFS Land Data Assimilation (DA) System. Land DA is an offline version of the Noah Multi-Physics (Noah-MP) land surface model (LSM) used in the UFS Weather Model (WM). Its data assimilation framework uses the Joint Effort for Data assimilation Integration (JEDI) software stack, which includes the Object-Oriented Prediction System (OOPS) for the data assimilation algorithm, the Interface for Observation Data Access (IODA) for observation formatting and processing, and the Unified Forward Operator (UFO) for comparing model forecasts and observations. 
+The UFS includes [multiple applications](https://ufs.epic.noaa.gov/applications/) that support different forecast durations and spatial domains. This repository hosts the source code for the UFS Land Data Assimilation (DA) System. Land DA applies the Noah Multi-Physics (Noah-MP) land surface model (LSM) of the UFS Weather Model (WM) as the key component. Its data assimilation framework uses the Joint Effort for Data assimilation Integration (JEDI) software stack, which includes the Object-Oriented Prediction System (OOPS) for the data assimilation algorithm, the Interface for Observation Data Access (IODA) for observation formatting and processing, and the Unified Forward Operator (UFO) for comparing model forecasts and observations. 
 
-The offline Noah-MP LSM is a standalone, uncoupled model used to execute land surface simulations. In this traditional uncoupled mode, near-surface atmospheric forcing data is required as input forcing. This LSM simulates soil moisture (both liquid and frozen), soil temperature, skin temperature, snow depth, snow water equivalent (SWE), snow density, canopy water content, and the energy flux and water flux terms of the surface energy balance and surface water balance. Its data assimilation framework applies the Local Ensemble Transform Kalman Filter-Optimal Interpolation (LETKF-OI) algorithm to combine the state-dependent background error derived from an ensemble forecast with the observations and their corresponding uncertainties to produce an analysis ensemble (Hunt et al., 2007).
+The Noah-MP LSM is used to execute land surface simulations. A near-surface atmospheric forcing data is required as input forcing. This LSM simulates soil moisture (both liquid and frozen), soil temperature, skin temperature, snow depth, snow water equivalent (SWE), snow density, canopy water content, and the energy flux and water flux terms of the surface energy balance and surface water balance. Its data assimilation framework applies the Local Ensemble Transform Kalman Filter-Optimal Interpolation (LETKF-OI) algorithm to combine the state-dependent background error derived from an ensemble forecast with the observations and their corresponding uncertainties to produce an analysis ensemble (Hunt et al., 2007).
 
-The Noah-MP LSM has evolved through community efforts to pursue and refine a modern-era LSM suitable for use in the National Centers for Environmental Prediction (NCEP) operational weather and climate prediction models. This collaborative effort continues with participation from entities such as NCAR, NCEP, NASA, and university groups. The development branch of the Land DA System is continually evolving as the system undergoes open development. The latest Land DA release (v1.2.0) represents a snapshot of this continuously evolving system. 
+The Noah-MP LSM has evolved through community efforts to pursue and refine a modern-era LSM suitable for use in the National Centers for Environmental Prediction (NCEP) operational weather and climate prediction models. This collaborative effort continues with participation from entities such as NCAR, NCEP, NASA, and university groups. The development branch of the Land DA System is continually evolving as the system undergoes open development. The latest Land DA release (v2.0.0) represents a snapshot of this continuously evolving system. 
 
 The Land DA System User's Guide associated with the development branch is at: https://land-da-workflow.readthedocs.io/en/develop/, while the guide specific to the Land DA v2.0.0 release can be found at: https://land-da-workflow.readthedocs.io/en/release-public-v2.0.0/. Users may download data for use with the most recent release from the [Land DA data bucket](https://registry.opendata.aws/noaa-ufs-land-da/). The [Land DA Docker Hub](https://hub.docker.com/r/noaaepic/ubuntu22.04-intel21.10-landda) hosts Land DA containers. These containers package the Land DA System together with all its software dependencies for an easier experience building and running Land DA.
 

jobs/JLANDDA_ANALYSIS

@@ -73,15 +73,14 @@ fi
 mkdir -p ${COMOUT}
 export COMOUThofx="${COMOUThofx:-${COMOUT}/hofx}"
 mkdir -p ${COMOUThofx}
+export COMOUTplot="${COMOUTplot:-${COMOUT}/plot}"
+mkdir -p ${COMOUTplot}
 
 # Create a teomporary share directory
-export DATA_SHARE="${DATA_SHARE:-${DATAROOT}/DATA_SHARE/${PDY}}"
-mkdir -p ${DATA_SHARE}
+export DATA_RESTART="${DATA_RESTART:-${DATAROOT}/DATA_SHARE/RESTART}"
+mkdir -p ${DATA_RESTART}
 export DATA_HOFX="${DATA_HOFX:-${DATAROOT}/DATA_SHARE/hofx}"
 mkdir -p ${DATA_HOFX}
-
-# Set other dates
-export PTIME=$($NDATE -24 $PDY$cyc)
 #
 #-----------------------------------------------------------------------
 #

jobs/JLANDDA_FCST_IC

@@ -0,0 +1,105 @@
+#!/bin/bash
+
+date
+export PS4='+ $SECONDS + '
+set -xue
+#
+#-----------------------------------------------------------------------
+#
+# Set the NCO standard environment variables (Table 1, pp.4)
+#
+#-----------------------------------------------------------------------
+#
+export USHlandda="${HOMElandda}/ush"
+export EXEClandda="${HOMElandda}/exec"
+export PARMlandda="${HOMElandda}/parm"
+export FIXlandda="${HOMElandda}/fix"
+export SCRIPTSlandda="${HOMElandda}/scripts"
+#
+#-----------------------------------------------------------------------
+#
+# Define job and jobid by default for rocoto
+#
+#-----------------------------------------------------------------------
+#
+WORKFLOW_MANAGER="${WORKFLOW_MANAGER:-rocoto}"
+if [ "${WORKFLOW_MANAGER}" = "rocoto" ]; then
+  if [ "${SCHED}" = "slurm" ]; then
+    job=${SLURM_JOB_NAME}
+    pid=${SLURM_JOB_ID}
+  elif [ "${SCHED}" = "pbspro" ]; then
+    job=${PBS_JOBNAME}
+    pid=${PBS_JOBID}
+  else
+    job="task"
+    pid=$$
+  fi
+  jobid="${job}.${PDY}${cyc}.${pid}"
+fi
+#
+#-----------------------------------------------------------------------
+#
+# Create a temp working directory (DATA) and cd into it.
+#
+#-----------------------------------------------------------------------
+#
+export DATA="${DATA:-${DATAROOT}/${jobid}}"
+mkdir -p $DATA
+cd $DATA
+#
+#-----------------------------------------------------------------------
+#
+# Define NCO environment variables and set COM type definitions.
+#
+#-----------------------------------------------------------------------
+#
+export NET="${NET:-landda}"
+export RUN="${RUN:-landda}"
+
+# Run setpdy to initialize PDYm and PDYp variables
+export cycle="${cycle:-t${cyc}z}"
+setpdy.sh
+. ./PDY
+
+[[ "$WORKFLOW_MANAGER" = "rocoto" ]] && export COMROOT=$COMROOT
+if [ "${MACHINE}" = "WCOSS2" ]; then
+  export COMIN="${COMIN:-$(compath.py -o ${NET}/${model_ver}/${RUN}.${PDY})}"
+  export COMOUT="${COMOUT:-$(compath.py -o ${NET}/${model_ver}/${RUN}.${PDY})}"
+  export COMINgdas="${COMINgdas:-$(compath.py ${envir}/gdas/${gdas_ver})}"
+  export COMINgfs="${COMINgfs:-$(compath.py ${envir}/gfs/${gfs_ver})}"
+else
+  export COMIN="${COMIN:-${COMROOT}/${NET}/${model_ver}/${RUN}.${PDY}}"
+  export COMOUT="${COMOUT:-${COMROOT}/${NET}/${model_ver}/${RUN}.${PDY}}"
+  export COMINgdas="${COMINgdas:-${FIXlandda}/DATA_gdas}"
+  export COMINgfs="${COMINgfs:-${FIXlandda}/DATA_gfs}"
+fi
+
+mkdir -p ${COMOUT}
+
+#
+#-----------------------------------------------------------------------
+#
+# Call the ex-script for this J-job.
+#
+#-----------------------------------------------------------------------
+#
+export pgmout="${DATA}/OUTPUT.$$"
+env
+
+${SCRIPTSlandda}/exlandda_fcst_ic.sh
+export err=$?; err_chk
+
+if [ -e "$pgmout" ]; then
+  cat $pgmout
+fi
+#
+#-----------------------------------------------------------------------
+#
+# Whether or not working directory DATA should be kept.
+#
+#-----------------------------------------------------------------------
+#
+if [ "${KEEPDATA}" = "NO" ]; then
+  rm -rf ${DATA}
+fi
+date

jobs/JLANDDA_FORECAST

@@ -74,14 +74,16 @@ fi
 
 mkdir -p ${COMOUT}
 
+# RESTART directory in COMOUT
+mkdir -p ${COMOUT}/RESTART
+# plot directory in COMOUT
+export COMOUTplot="${COMOUTplot:-${COMOUT}/plot}"
+mkdir -p ${COMOUTplot}
+
 # Create a teomporary share directory
-export DATA_SHARE="${DATA_SHARE:-${DATAROOT}/DATA_SHARE/${PDY}}"
-mkdir -p ${DATA_SHARE}
 export DATA_RESTART="${DATA_RESTART:-${DATAROOT}/DATA_SHARE/RESTART}"
 mkdir -p ${DATA_RESTART}
 
-# Set other dates
-export NTIME=$($NDATE 24 $PDY$cyc)
 #
 #-----------------------------------------------------------------------
 #

jobs/JLANDDA_PLOT_STATS

@@ -75,13 +75,8 @@ export COMOUTplot="${COMOUTplot:-${COMOUT}/plot}"
 mkdir -p ${COMOUTplot}
 
 # Create a teomporary share directory
-export DATA_SHARE="${DATA_SHARE:-${DATAROOT}/DATA_SHARE/${PDY}}"
-mkdir -p ${DATA_SHARE}
 export DATA_HOFX="${DATA_HOFX:-${DATAROOT}/DATA_SHARE/hofx}"
 mkdir -p ${DATA_HOFX}
-
-# Set other dates
-export NTIME=$($NDATE 24 $PDY$cyc)
 #
 #-----------------------------------------------------------------------
 #

jobs/JLANDDA_POST_ANAL

@@ -73,13 +73,9 @@ fi
 mkdir -p ${COMOUT}
 
 # Create a teomporary share directory
-export DATA_SHARE="${DATA_SHARE:-${DATAROOT}/DATA_SHARE/${PDY}}"
-mkdir -p ${DATA_SHARE}
 export DATA_RESTART="${DATA_RESTART:-${DATAROOT}/DATA_SHARE/RESTART}"
 mkdir -p ${DATA_RESTART}
 
-# Set other dates
-export NTIME=$($NDATE 24 $PDY$cyc)
 #
 #-----------------------------------------------------------------------
 #

jobs/JLANDDA_PREP_OBS

@@ -73,13 +73,9 @@ fi
 mkdir -p ${COMOUT}
 export COMOUTobs=${COMOUTobs:-${COMOUT}/obs}
 mkdir -p ${COMOUTobs}
+export COMOUTplot="${COMOUTplot:-${COMOUT}/plot}"
+mkdir -p ${COMOUTplot}
 
-# Create a teomporary share directory
-export DATA_SHARE="${DATA_SHARE:-${DATAROOT}/DATA_SHARE/${PDY}}"
-mkdir -p ${DATA_SHARE}
-
-# Set other dates
-export PTIME=$($NDATE -24 $PDY$cyc)
 #
 #-----------------------------------------------------------------------
 #

jobs/JLANDDA_PRE_ANAL

@@ -73,13 +73,9 @@ fi
 mkdir -p ${COMOUT}
 
 # Create a teomporary share directory
-export DATA_SHARE="${DATA_SHARE:-${DATAROOT}/DATA_SHARE/${PDY}}"
-mkdir -p ${DATA_SHARE}
 export DATA_RESTART="${DATA_RESTART:-${DATAROOT}/DATA_SHARE/RESTART}"
 mkdir -p ${DATA_RESTART}
 
-# Set other dates
-export PTIME=$($NDATE -24 $PDY$cyc)
 #
 #-----------------------------------------------------------------------
 #

modulefiles/build_container_intel.lua

@@ -71,5 +71,5 @@ setenv("CC", "mpiicc")
 setenv("CXX", "mpiicpc")
 setenv("FC", "mpiifort")
 
-setenv("JEDI_INSTALL", pathJoin(os.getenv("EPICHOME"),""))
+setenv("JEDI_PATH", pathJoin(os.getenv("EPICHOME"),""))
 

modulefiles/build_hera_intel.lua

@@ -32,6 +32,12 @@ load(pathJoin("w3emc", w3emc_ver))
 load(pathJoin("gftl-shared", gftl_shared_ver))
 load(pathJoin("mapl", mapl_ver))
 load(pathJoin("scotch", scotch_ver))
+
+load(pathJoin("nemsio", nemsio_ver))
+load(pathJoin("sfcio", sfcio_ver))
+load(pathJoin("sigio", sigio_ver))
+load(pathJoin("nccmp", nccmp_ver))
+
 load(pathJoin("prod_util", prod_util_ver))
 load(pathJoin("ufs-pyenv", ufs_pyenv_ver))
 
@@ -40,5 +46,5 @@ setenv("CXX", "mpiicpc")
 setenv("FC", "mpiifort")
 setenv("CMAKE_Platform", "hera.intel")
 
-setenv("EPICHOME", "/scratch2/NAGAPE/epic/UFS_Land-DA_Dev")
-setenv("JEDI_INSTALL", "/scratch2/NAGAPE/epic/UFS_Land-DA_Dev/jedi_v7")
+setenv("EPICHOME", "/scratch2/NAGAPE/epic/UFS_Land-DA_v2.1")
+setenv("JEDI_PATH", "/scratch2/NAGAPE/epic/UFS_Land-DA_v2.1/jedi_v7_ic")

modulefiles/build_hercules_intel.lua

@@ -32,6 +32,12 @@ load(pathJoin("w3emc", w3emc_ver))
 load(pathJoin("gftl-shared", gftl_shared_ver))
 load(pathJoin("mapl", mapl_ver))
 load(pathJoin("scotch", scotch_ver))
+
+load(pathJoin("nemsio", nemsio_ver))
+load(pathJoin("sfcio", sfcio_ver))
+load(pathJoin("sigio", sigio_ver))
+load(pathJoin("nccmp", nccmp_ver))
+
 load(pathJoin("prod_util", prod_util_ver))
 load(pathJoin("ufs-pyenv", ufs_pyenv_ver))
 
@@ -43,5 +49,5 @@ setenv("CXX", "mpiicpc")
 setenv("FC", "mpiifort")
 setenv("CMAKE_Platform", "hercules.intel")
 
-setenv("EPICHOME", "/work/noaa/epic/UFS_Land-DA_Dev")
-setenv("JEDI_INSTALL", "/work/noaa/epic/UFS_Land-DA_Dev/jedi_v7_hercules")
+setenv("EPICHOME", "/work/noaa/epic/UFS_Land-DA_v2.1")
+setenv("JEDI_PATH", "/work/noaa/epic/UFS_Land-DA_v2.1/jedi_v7_ic_hercules")

modulefiles/build_orion_intel.lua

@@ -32,6 +32,12 @@ load(pathJoin("w3emc", w3emc_ver))
 load(pathJoin("gftl-shared", gftl_shared_ver))
 load(pathJoin("mapl", mapl_ver))
 load(pathJoin("scotch", scotch_ver))
+
+load(pathJoin("nemsio", nemsio_ver))
+load(pathJoin("sfcio", sfcio_ver))
+load(pathJoin("sigio", sigio_ver))
+load(pathJoin("nccmp", nccmp_ver))
+
 load(pathJoin("prod_util", prod_util_ver))
 load(pathJoin("ufs-pyenv", ufs_pyenv_ver))
 
@@ -43,5 +49,5 @@ setenv("CXX", "mpiicpc")
 setenv("FC", "mpiifort")
 setenv("CMAKE_Platform", "orion.intel")
 
-setenv("EPICHOME", "/work/noaa/epic/UFS_Land-DA_Dev")
-setenv("JEDI_INSTALL", "/work/noaa/epic/UFS_Land-DA_Dev/jedi_v7_stack1.6")
+setenv("EPICHOME", "/work/noaa/epic/UFS_Land-DA_v2.1")
+setenv("JEDI_PATH", "/work/noaa/epic/UFS_Land-DA_v2.1/jedi_v7_ic_orion")

modulefiles/build_singularity_intel.lua

@@ -100,6 +100,6 @@ setenv("CC", "mpiicc")
 setenv("CXX", "mpiicpc")
 setenv("FC", "mpiifort")
 
-setenv("JEDI_INSTALL", pathJoin(os.getenv("EPICHOME"),""))
+setenv("JEDI_PATH", pathJoin(os.getenv("EPICHOME"),""))
 
 whatis("Description: UFS build environment")

modulefiles/tasks/hera/task.analysis.lua

@@ -1,13 +1,16 @@
-prepend_path("MODULEPATH", os.getenv("modulepath_spack_stack"))
+prepend_path("MODULEPATH", os.getenv("modulepath_spack_stack_unienv"))
 
 load(pathJoin("stack-intel", stack_intel_ver))
 load(pathJoin("stack-intel-oneapi-mpi", stack_intel_oneapi_mpi_ver))
 load(pathJoin("stack-python", stack_python_ver))
 
 load(pathJoin("prod_util", prod_util_ver))
-
+load(pathJoin("py-cartopy", py_cartopy_ver))
 load(pathJoin("py-jinja2", py_jinja2_ver))
+load(pathJoin("py-matplotlib", py_matplotlib_ver))
 load(pathJoin("py-netcdf4", py_netcdf4_ver))
 load(pathJoin("py-numpy", py_numpy_ver))
 load(pathJoin("py-pyyaml", py_pyyaml_ver))
+load(pathJoin("py-scipy", py_scipy_ver))
+load(pathJoin("py-xarray", py_xarray_ver))
 

modulefiles/tasks/hera/task.fcst_ic.lua

@@ -0,0 +1,20 @@
+prepend_path("MODULEPATH", os.getenv("modulepath_spack_stack"))
+
+load(pathJoin("stack-intel", stack_intel_ver))
+load(pathJoin("stack-intel-oneapi-mpi", stack_intel_oneapi_mpi_ver))
+load(pathJoin("stack-python", stack_python_ver))
+
+load(pathJoin("jasper", jasper_ver))
+load(pathJoin("hdf5", hdf5_ver))
+load(pathJoin("netcdf-c", netcdf_c_ver))
+load(pathJoin("netcdf-fortran", netcdf_fortran_ver))
+load(pathJoin("parallelio", parallelio_ver))
+load(pathJoin("esmf", esmf_ver))
+load(pathJoin("g2", g2_ver))
+load(pathJoin("prod_util", prod_util_ver))
+
+load(pathJoin("py-jinja2", py_jinja2_ver))
+load(pathJoin("py-netcdf4", py_netcdf4_ver))
+load(pathJoin("py-numpy", py_numpy_ver))
+load(pathJoin("py-pyyaml", py_pyyaml_ver))
+

modulefiles/tasks/hera/task.prep_obs.lua

@@ -1,7 +1,15 @@
-prepend_path("MODULEPATH", os.getenv("modulepath_spack_stack"))
+prepend_path("MODULEPATH", os.getenv("modulepath_spack_stack_unienv"))
 
 load(pathJoin("stack-intel", stack_intel_ver))
 load(pathJoin("stack-intel-oneapi-mpi", stack_intel_oneapi_mpi_ver))
 load(pathJoin("stack-python", stack_python_ver))
 
 load(pathJoin("prod_util", prod_util_ver))
+
+load(pathJoin("py-cartopy", py_cartopy_ver))
+load(pathJoin("py-matplotlib", py_matplotlib_ver))
+load(pathJoin("py-netcdf4", py_netcdf4_ver))
+load(pathJoin("py-numpy", py_numpy_ver))
+load(pathJoin("py-pandas", py_pandas_ver))
+load(pathJoin("py-pyyaml", py_pyyaml_ver))
+