Second rules update in v1 branch, update names, update `write_standard_name_table.py` to allow for subsections #87

mkavulich · 2024-12-13T03:41:07Z

This is the second round of proposed rules changes based on our ongoing discussion to make both rules and names as consistent as possible. Major updates include:

Standard Names

New "base_names" section, with subsections
Convert instances of "surface_X" to "X_at_surface"
Convert some instances of "air_X" or "X_of_air"
Explicitly state what mass mixing ratios are with respect to in long_name
Convert "mixing_ratio" to "water_vapor_mixing_ratio_wrt_moist_air"
Convert "surface_albedo" to simply "albedo", "surface_roughness_length" to simply "roughness_length"

Rules

Update construction template to include [non-instant time] and [non-current time], swap [at level] and [in medium]
More detailed description of transformations and how they can work on multiple base names
Add a few more transformations

`write_standard_name_table.py`

Updated to allow sub-sections in the standard names list

You can see these changes in the form of a Google doc for better visualization here: https://docs.google.com/document/d/19ysUCWDhv53W8fQbElW7opr_1Pm7ck95QRUyKM_qy4E/edit?tab=t.0

Note: with this update, we have 347 standard names that are fully compliant with the rules we have set out. A big portion of the remainder are "flag"-type variables, indices, etc, which are not fully accounted for in the rules yet.

…xner_function --> exner_function, add long names where appropriate

…rd names (keep in long names for reference), slightly revise rules on the proper term to align with other uses of "mixing ratio"

… subsections

StandardNamesRules.rst

standard_names.xml

climbfuji · 2024-12-13T23:22:39Z

standard_names.xml

+                     long_name="Molecular oxygen, O₂">
+      </standard_name>
+      <standard_name name="ozone"
+                     long_name="Ozone, O₃">


Will these non-ASCII characters work? (also line 246)

long_name should be able to handle any unicode characters since it is not used in any FORTRAN code context. Even if long names were eventually used in comments or even character variables, that would be fine according to the FORTRAN 90 standard*, so long as it's a character set supported by the OS. I think anything in UTF-32 should be allowed for long names, since Python can handle those characters and it's essentially universally supported among OS's.

This does bring up the fact that we haven't defined a character set for the standard names; I'll add that to the to-do list. So long as we make anything that may possibly be used as a fortran variable/subroutine/other code strictly within the allowed Fortran character set, we should be good.

The relevant section on page 18 is here:

Additional characters may be representable in the processor, but may appear only in comments (3.3.1.1,
15 3.3.2.1), character constants (4.4.4), input/output records (9.1.1), and character string edit descriptors
16 (10.2.1).

Can we talk about this at the next standard names meeting, please? My immediate reaction is to stick with the basic ASCII characters and nothing else, but maybe I can be convinced otherwise ;-) At the very least, we need to run a test with CCPP (SCM, UFS, ...) if the parsers (prebuild and capgen) can handle those characters.

From today's discussion, we will restrict standard names (including long names) to ASCII character set.

climbfuji · 2024-12-13T23:24:34Z

standard_names.xml

@@ -2311,7 +2831,8 @@
      <standard_name name="upward_virtual_potential_temperature_flux">
         <type kind="kind_phys" units="K m s-1">real</type>
      </standard_name>
-      <standard_name name="surface_upward_specific_humidity_flux_for_mellor_yamada_janjic_surface_layer_scheme">
+      <standard_name name="upward_flux_of_water_vapor_mixing_ratio_wrt_moist_air_at_surface_for_mellor_yamada_janjic_surface_layer_scheme"


We should probably define MYJ as the abbreviation to use in the standard name, and the full name in the long name? Like PBL and GWD in my open PR?

I think this is a good idea, but I will add it to my next round of changes to avoid scope creep. There are probably more abbreviations to introduce beyond this, like MYNN.

standard_names.xml

climbfuji · 2024-12-13T23:26:45Z

standard_names.xml

         <type kind="kind_phys" units="fraction">real</type>
      </standard_name>
      <standard_name name="lwe_surface_snow_from_coupled_process">
         <type kind="kind_phys" units="m">real</type>
      </standard_name>
-      <standard_name name="surface_upward_latent_heat_flux_from_coupled_process">
+      <standard_name name="upward_latent_heat_flux_at_surface_from_coupled_process">


What is the difference between for_coupling and from_coupled_process (not for vs from, but coupling vs coupled process)?

This is a good point: In general I think the use of "coupling" and "coupled" in variable names is ambiguous and problematic. Probably needs a deeper dive after this PR.

We will use "for coupling" suffix, and include a detailed description of what that means (pending further potential discussion).

Also potentially need new suffixes depending on CESM use case, dependent on group names? Maybe drop "coupling" all together in favor of "timing"/"order" suffixes

gold2718

Great (and great big) piece of work!
A few requested changes and a lot of questions along with a few suggestions.
Also, should all long names begin with an capitalized word? Right now, there is a mix.

tools/write_standard_name_table.py

standard_names.xml

mkavulich

@gold2718 @climbfuji Thanks for your thorough reviews. I've addressed most of your comments/suggestions, and those that I haven't I replied with a follow-up comment or question. Let me know if anything else needs resolving.

StandardNamesRules.rst

standard_names.xml

nusbaume

Great work @mkavulich! I had a couple of change requests, but nothing that requires a full re-review.

standard_names.xml

nusbaume · 2025-02-09T20:37:29Z

standard_names.xml

+             comment="These names are used as bases for other names, but may\n
+                      also be considered standard names on their own. See the\n
+                      full list of standard names for further details.\n">
+      <standard_name name="absolute_vorticity"


Since these can be considered formal standard names should we include the units here as well? (I'm happy to hold this off for a future PR if that would be easier).

standard_names.xml

gold2718

This looks better but not all the issues are resolved (e.g., vis to visible), at least as of cd5ba37.

standard_names.xml

…heck_name_rules.py

gold2718 · 2025-02-27T22:23:14Z

Will #86 be merged in before final review?

mkavulich · 2025-03-03T22:46:07Z

@gold2718 I did not plan to resolve conflicts between the main branch and release branch at this time, since the changes are already so divergent. But this is a good idea; I will make a follow-up PR right now to bring in the relevant changes since this branch was cut.

I thought I had responded to your latest comments but I guess I forgot; I made the changes you requested re: vis to visible, was there anything else that needed attention or are you okay with this being merged?

svahl991 · 2025-03-03T22:53:43Z

This PR changes many of the names that we just adapted in JEDI last fall during a code sprint that took considerable effort. Before merging this, can we create a tag or version of this repository that JEDI can reference?

gold2718 · 2025-03-04T12:09:36Z

@gold2718 @climbfuji Thanks for your thorough reviews. I've addressed most of your comments/suggestions, and those that I haven't I replied with a follow-up comment or question. Let me know if anything else needs resolving.

Please just open issues for any changes that need changing (based on comments on this PR) so that they are not lost. I do not see one for the weight to scaling_factor change.

gold2718 · 2025-03-04T12:35:28Z

This PR changes many of the names that we just adapted in JEDI last fall during a code sprint that took considerable effort. Before merging this, can we create a tag or version of this repository that JEDI can reference?

No objections here. v0.1.2?

mkavulich · 2025-03-04T15:49:20Z

This PR changes many of the names that we just adapted in JEDI last fall during a code sprint that took considerable effort. Before merging this, can we create a tag or version of this repository that JEDI can reference?

@svahl991 Just a note, this PR is not being merged to main, but to a branch release/v1 as part of a multi-stage effort to make the names and rules consistent. We will not be applying these changes to the main branch until there has been substantial input from all interested parties.

We had discussed creating a tag back in the fall for JEDI's "stable" version but last time we had that discussion there were still some changes being debated. Is the current state of main the preferred version to make this tag? Or would it be a previous hash?

mkavulich · 2025-03-04T16:08:33Z

@gold2718 I have opened some issues that I think covers all of your concerns; I have been tracking them in a google doc but making sure they are tracked here as well is a good idea.

svahl991 · 2025-03-04T16:56:34Z

Just a note, this PR is not being merged to main, but to a branch release/v1 as part of a multi-stage effort to make the names and rules consistent.

Thanks @mkavulich . That's an important detail I missed. So, to make sure I understand:

The main branch is still the "official" ESM (formerly CCPP) name for the time being.
The release/v1 branch is building up a major re-work of the rules and names, but is not yet "official".
The plan is to build this re-work up in the release branch, get consensus, and merge it into main at some point in the future.

I apologize if I missed and/or forgot past communications about this.

gold2718

All looks good now, thanks!
(and thanks to @mkavulich for getting all those issues open)

svahl991 · 2025-03-04T18:36:37Z

Metadata-standard-names.md

@@ -1727,13 +1926,13 @@ Variables related to the compute environment
    * `real(kind=kind_phys)`: units = K
 * `volumetric_soil_moisture_between_soil_bottom_and_water_table`: Volumetric soil moisture between soil bottom and water table
    * `real(kind=kind_phys)`: units = m3 m-3
-* `water_vapor_mixing_ratio_wrt_moist_air_at_2m`: mixing ratio of the mass of water vapor to the mass of moist air, at two meters above surface
+* `water_vapor_mixing_ratio_wrt_moist_air_at_2m`: Specific humidity (water vapor mass mixing ratio with respect to moist air) at two meters above surface


We have found in our work on JEDI that the term "specific humidity" does not have an internationally agreed-upon meaning. I suggest not using it at all to avoid confusion. Or, if it must be used, to qualify it. (When the UKMO uses the term "specific humidity" they mean water_vapor_mixing_ratio_wrt_moist_air_and_condensed_water, which is two lines below this one.)

mkavulich · 2025-03-04T19:07:14Z

@svahl991 Your understanding is correct. This re-factor is part of a lead-up to standing up a more formal governance outside of the CCPP framework's purview; we don't yet have anything official but I have been putting out feelers to ensure that all interested groups are represented. As an active contributor from JCSDA, I'll certainly be sure you're in the loop when we start scheduling meetings.

With regards to specific humidity, you'll note that this PR removes specific_humidity from all standard names, exactly because of the problems you mentioned (conflicting/non-standardized definitions), see here for example. But because it is a common term that people are likely to search for, we decided it should stay in long_name entries where relevant, with a parenthetical qualifier of the exact definition as you see here. I could see where it might still be an issue though, so I will open an issue where we can discuss this in more detail.

mkavulich added 16 commits December 12, 2024 17:03

First round of rules updates, mostly discussed at the last meeting

8298ff0

Fix image embedding

6746dba

Fix some links

999c887

minor updates

34f9b03

Fix spelling in standard names

c8f4202

Rename surface_roughness_length --> roughness_length, dimensionless_e…

0b2c439

…xner_function --> exner_function, add long names where appropriate

Update albedo standard names

d66db49

Some more renames: surface_* to *_at_surface, air_* to *_of_air

b47546c

Updates involving specific humidity: remove use of the term in standa…

c4c2600

…rd names (keep in long names for reference), slightly revise rules on the proper term to align with other uses of "mixing ratio"

Add long name descriptions to mass mixing ratios as specified in rules

2d7cfbf

A bunch more surface_x --> x_at_surface

66f61aa

re-factor logic in write_standard_name_table.py so that it can handle…

c3c321e

… subsections

Add first round of base names

9ab244b

Remaining Rules updates from Google Doc

1181782

Add rest of "base names" section

8760c33

Fix format

c1a4dd5

mkavulich requested review from cacraigucar, climbfuji, dustinswales, gold2718, grantfirl, mattldawson, mwaxmonsky, nusbaume, peverwhee, svahl991, MayeulDestouches and ss421 as code owners December 13, 2024 03:41

climbfuji reviewed Dec 13, 2024

View reviewed changes

gold2718 requested changes Jan 1, 2025

View reviewed changes

mkavulich added 2 commits January 29, 2025 13:59

Address reviewer comments

5ad616b

Missed one

f0c6da5

mkavulich commented Jan 29, 2025

View reviewed changes

nusbaume approved these changes Feb 10, 2025

View reviewed changes

mkavulich commented Feb 13, 2025

View reviewed changes

standard_names.xml Outdated Show resolved Hide resolved

mkavulich commented Feb 13, 2025

View reviewed changes

standard_names.xml Outdated Show resolved Hide resolved

Clear up long name confusion about Exner function

cd5ba37

mkavulich requested review from gold2718 and climbfuji February 20, 2025 16:35

gold2718 requested changes Feb 27, 2025

View reviewed changes

standard_names.xml Outdated Show resolved Hide resolved

standard_names.xml Outdated Show resolved Hide resolved

standard_names.xml Show resolved Hide resolved

mkavulich added 2 commits February 27, 2025 08:57

Remove non-ASCII characters, add check for non-ASCII chars in tools/c…

76231fa

…heck_name_rules.py

visible--> vis in standard names

aee20f3

This was referenced Mar 4, 2025

[release/v1] Convert instances of weight to scaling_factor for clarity #92

Open

[release/v1] Expand on long_name descriptions for base names #93

Open

[release/v1] Include more information on the role of "units" #94

Open

mkavulich mentioned this pull request Mar 4, 2025

Create v0.2.0 (v1.0.0?) tag for JEDI stable version. #96

Open

gold2718 approved these changes Mar 4, 2025

View reviewed changes

svahl991 reviewed Mar 4, 2025

View reviewed changes

mkavulich mentioned this pull request Mar 4, 2025

Inclusion of "specific humidity" in long_name entries, yea or nay? #97

Open

mkavulich merged commit 5a7b80f into ESCOMP:release/v1 Mar 4, 2025
3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Second rules update in v1 branch, update names, update `write_standard_name_table.py` to allow for subsections #87

Second rules update in v1 branch, update names, update `write_standard_name_table.py` to allow for subsections #87

mkavulich commented Dec 13, 2024

climbfuji Dec 13, 2024

mkavulich Jan 29, 2025

climbfuji Feb 10, 2025

mkavulich Feb 20, 2025

climbfuji Dec 13, 2024

mkavulich Jan 29, 2025

climbfuji Dec 13, 2024

mkavulich Jan 29, 2025

climbfuji Feb 10, 2025

mkavulich Feb 20, 2025 •

edited

Loading

gold2718 left a comment

mkavulich left a comment

nusbaume left a comment

nusbaume Feb 9, 2025

gold2718 left a comment

gold2718 commented Feb 27, 2025

mkavulich commented Mar 3, 2025

svahl991 commented Mar 3, 2025

gold2718 commented Mar 4, 2025

gold2718 commented Mar 4, 2025

mkavulich commented Mar 4, 2025

mkavulich commented Mar 4, 2025

svahl991 commented Mar 4, 2025

gold2718 left a comment

svahl991 Mar 4, 2025

mkavulich commented Mar 4, 2025 •

edited

Loading

Second rules update in v1 branch, update names, update write_standard_name_table.py to allow for subsections #87

Second rules update in v1 branch, update names, update write_standard_name_table.py to allow for subsections #87

Conversation

mkavulich commented Dec 13, 2024

Standard Names

Rules

write_standard_name_table.py

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mkavulich Feb 20, 2025 • edited Loading

Choose a reason for hiding this comment

gold2718 left a comment

Choose a reason for hiding this comment

mkavulich left a comment

Choose a reason for hiding this comment

nusbaume left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gold2718 left a comment

Choose a reason for hiding this comment

gold2718 commented Feb 27, 2025

mkavulich commented Mar 3, 2025

svahl991 commented Mar 3, 2025

gold2718 commented Mar 4, 2025

gold2718 commented Mar 4, 2025

mkavulich commented Mar 4, 2025

mkavulich commented Mar 4, 2025

svahl991 commented Mar 4, 2025

gold2718 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mkavulich commented Mar 4, 2025 • edited Loading

Second rules update in v1 branch, update names, update `write_standard_name_table.py` to allow for subsections #87

Second rules update in v1 branch, update names, update `write_standard_name_table.py` to allow for subsections #87

`write_standard_name_table.py`

mkavulich Feb 20, 2025 •

edited

Loading

mkavulich commented Mar 4, 2025 •

edited

Loading