ESMF Profile and Trace Issue

[Daisy0419]

I am trying to profile the GEOS-Chem GCHP simulation using the built-in ESMF profiling and tracing functionality and encountered a few problems:

1. Empty Trace Data
   I have enabled profiling and tracing with the following environment variables:
   export ESMF_RUNTIME_TRACE=ON
   export ESMF_RUNTIME_TRACE_CLOCK=REALTIME
   export ESMF_RUNTIME_TRACE_FLUSH=EAGER
   export ESMF_RUNTIME_TRACE_PETLIST="0-5"
   export ESMF_RUNTIME_PROFILE=ON

When inspecting the contents of the generated trace files in the traceout directory, I see only minimal data (~641 KB in total), and the esmf-profiler tool does not load any useful output. The trace output directory structure is as follows:

[traceout]$ du -la .
16 ./metadata
104 ./esmf_stream_0001
104 ./esmf_stream_0003
104 ./esmf_stream_0000
104 ./esmf_stream_0002
104 ./esmf_stream_0004
104 ./esmf_stream_0005
641 .

The trace files do not seem to capture meaningful data, and the profiler output is empty.

2. Symbol Lookup Errors with libesmftrace_preload.so
   When profiling mpi performance following esmf fortran user reference (mpirun -np ${nCores} ${ESMF_INSTALL_LIBDIR}/preload.sh ./gchp 2>&1 | tee -a ${log}), I repeatedly encounter the following symbol lookup errors:

./gchp: symbol lookup error: /usr/local/esmf/lib/libO/Linux.gfortran.64.openmpi.default/libesmftrace_preload.so: undefined symbol: _ZN5ESMCI21TraceEventRegionEnterENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPi

The error is repeated for all processes.

System Details:
OS: Ubuntu 22.04
ESMF Version: v8.4.2
OpenMPI Version: 4.1.7
Compiler: GCC 11.4.0 (gfortran)

Could you please indicate:
What could be causing the insufficient trace data?
Are there any recommended steps to debug or rebuild ESMF to resolve the symbol lookup issue for mpi performance profile?

Thank you!

[anntsay]

@Daisy0419 have you called ESMF_Finalize to close out the trace file? thanks!