CDAT Migration: Refactor `meridional_mean_2d` set #795

tomvothecoder · 2024-02-28T22:17:41Z

Close CDAT Migration Phase 2: Refactor meridional_mean_2d set #657

Checklist

My code follows the style guidelines of this project
I have performed a self-review of my own code
My changes generate no new warnings
Any dependent changes have been merged and published in downstream modules

If applicable:

New and existing unit tests pass with my changes (locally and CI/CD build)
I have added tests that prove my fix is effective or that my feature works
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have noted that this is a breaking change for a major release (fix or feature that would cause existing functionality to not work as expected)

- Reorder these calls for consistency across set drivers

- Fix incorrect left text position float adjustment

tomvothecoder

Hi @chengzhuzhang, this PR is ready for review. I added comments to areas of interest in the file diffs. All of the .nc files are within relative tolerance.

Here are the links to the viewers:

I did an eye-check and all the plots look the same. The only difference I saw was that some of the "Max" metrics value for the second sub-plot differ, which might be due to using xESMF instead of Regrid2.

tomvothecoder · 2024-03-19T19:35:11Z

e3sm_diags/driver/meridional_mean_2d_driver.py

+    ds_t_plevs_rg_avg, ds_r_plevs_rg_avg = align_grids_to_lower_res(
+        ds_t_plevs_avg,
+        ds_r_plevs_avg,
+        var_key,
+        tool="xesmf",
+        method="conservative_normed",
+        axis_to_compare="X",
+    )


I ran into this issue when I tried using xCDAT's regrid2 here. I decided to use xESMF instead, and the results are still within max relative tolerance (only a few mismatching elements).

Results from regression testing notebook:

Comparing: * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-OMEGA-ANN-global_ref.nc * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-OMEGA-ANN-global_ref.nc * var_key: OMEGA Not equal to tolerance rtol=1e-05, atol=0 Mismatched elements: 6 / 24480 (0.0245%) Max absolute difference: 4.56288177e-07 Max relative difference: 5.34042702e-05 x: array([[ 0.784679, 0.796702, 0.792179, ..., 0.804814, 0.773629, 0.774743], [ 1.379984, 1.417249, 1.4173 , ..., 1.432473, 1.391159,... y: array([[ 0.784679, 0.796702, 0.792179, ..., 0.804814, 0.773629, 0.774743], [ 1.379984, 1.417249, 1.4173 , ..., 1.432473, 1.391159,... Comparing: * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-OMEGA-ANN-global_test.nc * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-OMEGA-ANN-global_test.nc * var_key: OMEGA * All close and within relative tolerance (1e-05) Comparing: * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-OMEGA-JJA-global_ref.nc * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-OMEGA-JJA-global_ref.nc * var_key: OMEGA Not equal to tolerance rtol=1e-05, atol=0 Mismatched elements: 3 / 24480 (0.0123%) Max absolute difference: 5.97006945e-07 Max relative difference: 1.49039703e-05 x: array([[1.08258 , 1.069207, 1.07484 , ..., 1.090696, 1.087342, 1.09467 ], [1.563664, 1.593989, 1.628474, ..., 1.553196, 1.556833, 1.570584], [2.220292, 2.307285, 2.364905, ..., 2.215879, 2.220069, 2.228939],... y: array([[1.08258 , 1.069207, 1.07484 , ..., 1.090696, 1.087342, 1.09467 ], [1.563664, 1.593989, 1.628474, ..., 1.553196, 1.556833, 1.570584], [2.220292, 2.307285, 2.364905, ..., 2.215879, 2.220069, 2.228939],... Comparing: * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-OMEGA-JJA-global_test.nc * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-OMEGA-JJA-global_test.nc * var_key: OMEGA Not equal to tolerance rtol=1e-05, atol=0 Mismatched elements: 2 / 6120 (0.0327%) Max absolute difference: 3.39774408e-07 Max relative difference: 0.00015039 x: array([[ 1.150119, 1.168407, 1.16202 , ..., 1.153911, 1.138019, 1.152851], [ 1.567948, 1.599667, 1.601973, ..., 1.42647 , 1.435826,... y: array([[ 1.150119, 1.168407, 1.16202 , ..., 1.153911, 1.138019, 1.152851], [ 1.567948, 1.599667, 1.601973, ..., 1.42647 , 1.435826,... Comparing: * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-RELHUM-ANN-global_ref.nc * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-RELHUM-ANN-global_ref.nc * var_key: RELHUM * All close and within relative tolerance (1e-05) Comparing: * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-RELHUM-ANN-global_test.nc * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-RELHUM-ANN-global_test.nc * var_key: RELHUM * All close and within relative tolerance (1e-05) Comparing: * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-RELHUM-JJA-global_ref.nc * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-RELHUM-JJA-global_ref.nc * var_key: RELHUM * All close and within relative tolerance (1e-05) Comparing: * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-RELHUM-JJA-global_test.nc * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-RELHUM-JJA-global_test.nc * var_key: RELHUM * All close and within relative tolerance (1e-05) Comparing: * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-T-ANN-global_ref.nc * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-T-ANN-global_ref.nc * var_key: T * All close and within relative tolerance (1e-05) Comparing: * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-T-ANN-global_test.nc * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-T-ANN-global_test.nc * var_key: T * All close and within relative tolerance (1e-05) Comparing: * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-T-JJA-global_ref.nc * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-T-JJA-global_ref.nc * var_key: T * All close and within relative tolerance (1e-05) Comparing: * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-T-JJA-global_test.nc * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-T-JJA-global_test.nc * var_key: T * All close and within relative tolerance (1e-05) Comparing: * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-U-ANN-global_ref.nc * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-U-ANN-global_ref.nc * var_key: U Not equal to tolerance rtol=1e-05, atol=0 Mismatched elements: 1 / 24480 (0.00408%) Max absolute difference: 8.52794937e-08 Max relative difference: 1.73621834e-05 x: array([[10.269112, 10.260021, 10.25072 , ..., 10.296745, 10.28736 , 10.278145], [12.365973, 12.353776, 12.341225, ..., 12.402582, 12.390261,... y: array([[10.269112, 10.260021, 10.25072 , ..., 10.296745, 10.28736 , 10.278145], [12.365973, 12.353776, 12.341225, ..., 12.402582, 12.390261,... Comparing: * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-U-ANN-global_test.nc * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-U-ANN-global_test.nc * var_key: U Not equal to tolerance rtol=1e-05, atol=0 Mismatched elements: 1 / 6120 (0.0163%) Max absolute difference: 1.45889873e-07 Max relative difference: 1.08735566e-05 x: array([[ 9.712424, 9.692257, 9.675111, ..., 9.79681 , 9.779955, 9.765068], [12.482069, 12.457917, 12.434379, ..., 12.581674, 12.560391,... y: array([[ 9.712424, 9.692257, 9.675112, ..., 9.79681 , 9.779955, 9.765068], [12.482069, 12.457917, 12.434379, ..., 12.581674, 12.560391,... Comparing: * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-U-JJA-global_ref.nc * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-U-JJA-global_ref.nc * var_key: U Not equal to tolerance rtol=1e-05, atol=0 Mismatched elements: 1 / 24480 (0.00408%) Max absolute difference: 8.38617442e-08 Max relative difference: 1.96239722e-05 x: array([[ 8.311289, 8.302058, 8.292533, ..., 8.338692, 8.329687, 8.320503], [10.096993, 10.085039, 10.072245, ..., 10.131809, 10.120227,... y: array([[ 8.311289, 8.302058, 8.292533, ..., 8.338692, 8.329687, 8.320503], [10.096994, 10.085039, 10.072245, ..., 10.131809, 10.120227,... Comparing: * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/ERA5/ERA5-U-JJA-global_test.nc * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/ERA5/ERA5-U-JJA-global_test.nc * var_key: U * All close and within relative tolerance (1e-05) Comparing: * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-OMEGA-ANN-global_ref.nc * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-OMEGA-ANN-global_ref.nc * var_key: OMEGA Not equal to tolerance rtol=1e-05, atol=0 Mismatched elements: 2 / 9792 (0.0204%) Max absolute difference: 3.15701861e-07 Max relative difference: 2.42041494e-05 x: array([[0.914233, 0.906507, 0.900358, ..., 0.954717, 0.945675, 0.928962], [1.617818, 1.601587, 1.580476, ..., 1.631384, 1.64521 , 1.638424], [2.321404, 2.296666, 2.260593, ..., 2.308052, 2.344744, 2.347885],... y: array([[0.914233, 0.906507, 0.900358, ..., 0.954717, 0.945675, 0.928962], [1.617818, 1.601587, 1.580476, ..., 1.631384, 1.64521 , 1.638424], [2.321404, 2.296666, 2.260593, ..., 2.308052, 2.344744, 2.347885],... Comparing: * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-OMEGA-ANN-global_test.nc * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-OMEGA-ANN-global_test.nc * var_key: OMEGA * All close and within relative tolerance (1e-05) Comparing: * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-OMEGA-JJA-global_ref.nc * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-OMEGA-JJA-global_ref.nc * var_key: OMEGA Not equal to tolerance rtol=1e-05, atol=0 Mismatched elements: 2 / 9792 (0.0204%) Max absolute difference: 6.93198768e-07 Max relative difference: 5.40329309e-05 x: array([[ 1.283442, 1.27651 , 1.25776 , ..., 1.292833, 1.29537 , 1.287421], [ 1.93963 , 1.950937, 1.957762, ..., 1.901146, 1.91872 ,... y: array([[ 1.283442, 1.27651 , 1.25776 , ..., 1.292833, 1.29537 , 1.287421], [ 1.93963 , 1.950937, 1.957762, ..., 1.901146, 1.91872 ,... Comparing: * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-OMEGA-JJA-global_test.nc * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-OMEGA-JJA-global_test.nc * var_key: OMEGA Not equal to tolerance rtol=1e-05, atol=0 Mismatched elements: 2 / 6120 (0.0327%) Max absolute difference: 3.39774408e-07 Max relative difference: 0.00015039 x: array([[ 1.150119, 1.168407, 1.16202 , ..., 1.153911, 1.138019, 1.152851], [ 1.567948, 1.599667, 1.601973, ..., 1.42647 , 1.435826,... y: array([[ 1.150119, 1.168407, 1.16202 , ..., 1.153911, 1.138019, 1.152851], [ 1.567948, 1.599667, 1.601973, ..., 1.42647 , 1.435826,... Comparing: * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-RELHUM-ANN-global_ref.nc * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-RELHUM-ANN-global_ref.nc * var_key: RELHUM * All close and within relative tolerance (1e-05) Comparing: * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-RELHUM-ANN-global_test.nc * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-RELHUM-ANN-global_test.nc * var_key: RELHUM * All close and within relative tolerance (1e-05) Comparing: * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-RELHUM-JJA-global_ref.nc * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-RELHUM-JJA-global_ref.nc * var_key: RELHUM * All close and within relative tolerance (1e-05) Comparing: * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-RELHUM-JJA-global_test.nc * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-RELHUM-JJA-global_test.nc * var_key: RELHUM * All close and within relative tolerance (1e-05) Comparing: * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-T-ANN-global_ref.nc * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-T-ANN-global_ref.nc * var_key: T * All close and within relative tolerance (1e-05) Comparing: * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-T-ANN-global_test.nc * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-T-ANN-global_test.nc * var_key: T * All close and within relative tolerance (1e-05) Comparing: * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-T-JJA-global_ref.nc * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-T-JJA-global_ref.nc * var_key: T * All close and within relative tolerance (1e-05) Comparing: * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-T-JJA-global_test.nc * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-T-JJA-global_test.nc * var_key: T * All close and within relative tolerance (1e-05) Comparing: * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-U-ANN-global_ref.nc * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-U-ANN-global_ref.nc * var_key: U Not equal to tolerance rtol=1e-05, atol=0 Mismatched elements: 1 / 9792 (0.0102%) Max absolute difference: 1.06174015e-07 Max relative difference: 1.43005542e-05 x: array([[10.623722, 10.599673, 10.576724, ..., 10.698371, 10.673217, 10.64816 ], [12.345496, 12.317517, 12.289864, ..., 12.432171, 12.403032,... y: array([[10.623722, 10.599673, 10.576724, ..., 10.698371, 10.673217, 10.64816 ], [12.345496, 12.317517, 12.289864, ..., 12.432171, 12.403032,... Comparing: * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-U-ANN-global_test.nc * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-U-ANN-global_test.nc * var_key: U Not equal to tolerance rtol=1e-05, atol=0 Mismatched elements: 1 / 6120 (0.0163%) Max absolute difference: 1.45889873e-07 Max relative difference: 1.08735566e-05 x: array([[ 9.712424, 9.692257, 9.675111, ..., 9.79681 , 9.779955, 9.765068], [12.482069, 12.457917, 12.434379, ..., 12.581674, 12.560391,... y: array([[ 9.712424, 9.692257, 9.675112, ..., 9.79681 , 9.779955, 9.765068], [12.482069, 12.457917, 12.434379, ..., 12.581674, 12.560391,... Comparing: * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-U-JJA-global_ref.nc * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-U-JJA-global_ref.nc * var_key: U * All close and within relative tolerance (1e-05) Comparing: * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/657-meridional-mean-2d/meridional_mean_2d/MERRA2/MERRA2-U-JJA-global_test.nc * /global/cfs/cdirs/e3sm/www/cdat-migration-fy24/main/meridional_mean_2d/MERRA2/MERRA2-U-JJA-global_test.nc * var_key: U * All close and within relative tolerance (1e-05)

I recall that the default regridding back end for the main is esmf. And the results look good!

"regrid2" is hard-coded as the regrid method in meridional_mean_2d_driver.py. It was probably supposed to be replaced with esmf at some point.

e3sm_diags/e3sm_diags/driver/meridional_mean_2d_driver.py

Line 150 in 43f5f74

mv2_reg = mv2_reg.regrid(grid, regridTool="regrid2")[..., 0]

e3sm_diags/e3sm_diags/driver/meridional_mean_2d_driver.py

Line 173 in 43f5f74

mv1_reg = mv1_reg.regrid(grid, regridTool="regrid2")[..., 0]

I verified that this was the only set driver that had "regrid2" hard-coded

Thank you, @tomvothecoder. I can't recall why we made the choice for using regrid2 when first implemented it. It is good to replace with esmf at this point.

tomvothecoder · 2024-03-19T22:14:22Z

e3sm_diags/plot/utils.py

@@ -580,7 +580,7 @@ def _add_min_mean_max_text(
    fontdict = {"fontsize": fontsize}

    if left_text_pos is None:
-        left_text_pos = (0.6335, 0.2107)
+        left_text_pos = (0.6635, 0.2107)


Fix incorrect float for left text position adjustment

chengzhuzhang

The PR looks good! Thank you.

Refer to the PR for more information because the changelog is massive. Update build workflow to run on `cdat-migration-fy24` branch CDAT Migration Phase 2: Add CDAT regression test notebook template and fix GH Actions build (#743) - Add Makefile for quick access to multiple Python-based commands such as linting, testing, cleaning up cache and build files - Fix some lingering unit tests failure - Update `xcdat=0.6.0rc1` to `xcdat >=0.6.0` in `ci.yml`, `dev.yml` and `dev-nompi.yml` - Add `xskillscore` to `ci.yml` - Fix `pre-commit` issues CDAT Migration Phase 2: Regression testing for `lat_lon`, `lat_lon_land`, and `lat_lon_river` (#744) - Add Makefile that simplifies common development commands (building and installing, testing, etc.) - Write unit tests to cover all new code for utility functions - `dataset_xr.py`, `metrics.py`, `climo_xr.py`, `io.py`, `regrid.py` - Metrics comparison for `cdat-migration-fy24` `lat_lon` and `main` branch of `lat_lon` -- `NET_FLUX_SRF` and `RESTOM` have the highest spatial average diffs - Test run with 3D variables (`_run_3d_diags()`) - Fix Python 3.9 bug with using pipe command to represent Union -- doesn't work with `from __future__ import annotations` still - Fix subsetting syntax bug using ilev - Fix regridding bug where a single plev is passed and xCDAT does not allow generating bounds for coordinates of len <= 1 -- add conditional that just ignores adding new bounds for regridded output datasets, fix related tests - Fix accidentally calling save plots and metrics twice in `_get_metrics_by_region()` - Fix failing integration tests pass in CI/CD - Refactor `test_diags.py` -- replace unittest with pytest - Refactor `test_all_sets.py` -- replace unittest with pytest - Test climatology datasets -- tested with 3d variables using `test_all_sets.py` CDAT Migration Phase 2: Refactor utilities and CoreParameter methods for reusability across diagnostic sets (#746) - Move driver type annotations to `type_annotations.py` - Move `lat_lon_driver._save_data_metrics_and_plots()` to `io.py` - Update `_save_data_metrics_and_plots` args to accept `plot_func` callable - Update `metrics.spatial_avg` to return an optionally `xr.DataArray` with `as_list=False` - Move `parameter` arg to the top in `lat_lon_plot.plot` - Move `_set_param_output_attrs` and `_set_name_yr_attrs` from `lat_lon_driver` to `CoreParameter` class Regression testing for lat_lon variables `NET_FLUX_SRF` and `RESTOM` (#754) Update regression test notebook to show validation of all vars Add `subset_and_align_datasets()` to regrid.py (#776) Add template run scripts CDAT Migration Phase: Refactor `cosp_histogram` set (#748) - Refactor `cosp_histogram_driver.py` and `cosp_histogram_plot.py` - `formulas_cosp.py` (new file) - Includes refactored, Xarray-based `cosp_histogram_standard()` and `cosp_bin_sum()` functions - I wrote a lot of new code in `formulas_cosp.py` to clean up `derivations.py` and the old equivalent functions in `utils.py` - `derivations.py` - Cleaned up portions of `DERIVED_VARIABLES` dictionary - Removed unnecessary `OrderedDict` usage for `cosp_histogram` related variables (we should do this for the rest of the variables in in #716) - Remove unnecessary `convert_units()` function calls - Move cloud levels passed to derived variable formulas to `formulas_cosp.CLOUD_BIN_SUM_MAP` - `utils.py` - Delete deprecated, CDAT-based `cosp_histogram` functions - `dataset_xr.py` - Add `dataset_xr.Dataset._open_climo_dataset()` method with a catch for dataset quality issues where "time" is a scalar variable that does not match the "time" dimension array length, drops this variable and replaces it with the correct coordinate - Update `_get_dataset_with_derivation_func()` to handle derivation functions that require the `xr.Dataset` and `target_var_key` args (e.g., `cosp_histogram_standardize()` and `cosp_bin_sum()`) - `io.py` - Update `_write_vars_to_netcdf()` to write test, ref, and diff variables to individual netCDF (required for easy comparison to CDAT-based code that does the same thing) - Add `cdat_migration_regression_test_netcdf.ipynb` validation notebook template for comparing `.nc` files CDAT Migration Phase 2: Refactor `zonal_mean_2d()` and `zonal_mean_2d_stratosphere()` sets (#774) Refactor 654 zonal mean xy (#752) Co-authored-by: Tom Vo <tomvothecoder@gmail.com> CDAT Migration - Update run script output directory to NERSC public webserver (#793) [PR]: CDAT Migration: Refactor `aerosol_aeronet` set (#788) CDAT Migration: Test `lat_lon` set with run script and debug any issues (#794) CDAT Migration: Refactor `polar` set (#749) Co-authored-by: Tom Vo <tomvothecoder@gmail.com> Align order of calls to `_set_param_output_attrs` CDAT Migration: Refactor `meridional_mean_2d` set (#795) CDAT Migration: Refactor `aerosol_budget` (#800) Add `acme.py` changes from PR #712 (#814) * Add `acme.py` changes from PR #712 * Replace unnecessary lambda call Refactor area_mean_time_series and add ccb slice flag feature (#750) Co-authored-by: Tom Vo <tomvothecoder@gmail.com> [Refactor]: Validate fix in PR #750 for #759 (#815) CDAT Migration Phase 2: Refactor `diurnal_cycle` set (#819) CDAT Migration: Refactor annual_cycle_zonal_mean set (#798) * Refactor `annual_cycle_zonal_mean` set * Address PR review comments * Add lat lon regression testing * Add debugging scripts * Update `_open_climo_dataset()` to decode times as workaround to misaligned time coords - Update `annual_cycle_zonal_mean_plot.py` to convert time coordinates to month integers * Fix unit tests * Remove old plotter * Add script to debug decode_times=True and ncclimo file * Update plotter time values to month integers * Fix slow `.load()` and multiprocessing issue - Due to incorrectly updating `keep_bnds` logic - Add `_encode_time_coords()` to workaround cftime issue `ValueError: "months since" units only allowed for "360_day" calendar` * Update `_encode_time_coords()` docstring * Add AODVIS debug script * update AODVIS obs datasets; regression test results --------- Co-authored-by: Tom Vo <tomvothecoder@gmail.com> CDAT Migration Phase 2: Refactor `qbo` set (#826) CDAT Migration Phase 2: Refactor tc_analysis set (#829) * start tc_analysis_refactor * update driver * update plotting * Clean up plotter - Remove unused variables - Make `plot_info` a constant called `PLOT_INFO`, which is now a dict of dicts - Reorder functions for top-down readability * Remove unused notebook --------- Co-authored-by: tomvothecoder <tomvothecoder@gmail.com> CDAT Migration Phase 2: Refactor `enso_diags` set (#832) CDAT Migration Phase 2: Refactor `streamflow` set (#837) [Bug]: CDAT Migration Phase 2: enso_diags plot fixes (#841) [Refactor]: CDAT Migration Phase 3: testing and documentation update (#846) CDAT Migration Phase 3 - Port QBO Wavelet feature to Xarray/xCDAT codebase (#860) CDAT Migration Phase 2: Refactor arm_diags set (#842) Add performance benchmark material (#864) Add function to add CF axis attr to Z axis if missing for downstream xCDAT operations (#865) CDAT Migration Phase 3: Add Convective Precipitation Fraction in lat-lon (#875) CDAT Migration Phase 3: Fix LHFLX name and add catch for non-existent or empty TE stitch file (#876) Add support for time series datasets via glob and fix `enso_diags` set (#866) Add fix for checking `is_time_series()` property based on `data_type` attr (#881) CDAT migration: Fix African easterly wave density plots in TC analysis and convert H20LNZ units to ppm/volume (#882) CDAT Migration: Update `mp_partition_driver.py` to use Dataset from `dataset_xr.py` (#883) CDAT Migration - Port JJB tropical subseasonal diags to Xarray/xCDAT (#887) CDAT Migration: Prepare branch for merge to `main` (#885) [Refactor]: CDAT Migration - Update dependencies and remove Dataset._add_cf_attrs_to_z_axes() (#891) CDAT Migration Phase 2: Refactor core utilities and `lat_lon` set (#677) Refer to the PR for more information because the changelog is massive. Update build workflow to run on `cdat-migration-fy24` branch CDAT Migration Phase 2: Add CDAT regression test notebook template and fix GH Actions build (#743) - Add Makefile for quick access to multiple Python-based commands such as linting, testing, cleaning up cache and build files - Fix some lingering unit tests failure - Update `xcdat=0.6.0rc1` to `xcdat >=0.6.0` in `ci.yml`, `dev.yml` and `dev-nompi.yml` - Add `xskillscore` to `ci.yml` - Fix `pre-commit` issues CDAT Migration Phase 2: Regression testing for `lat_lon`, `lat_lon_land`, and `lat_lon_river` (#744) - Add Makefile that simplifies common development commands (building and installing, testing, etc.) - Write unit tests to cover all new code for utility functions - `dataset_xr.py`, `metrics.py`, `climo_xr.py`, `io.py`, `regrid.py` - Metrics comparison for `cdat-migration-fy24` `lat_lon` and `main` branch of `lat_lon` -- `NET_FLUX_SRF` and `RESTOM` have the highest spatial average diffs - Test run with 3D variables (`_run_3d_diags()`) - Fix Python 3.9 bug with using pipe command to represent Union -- doesn't work with `from __future__ import annotations` still - Fix subsetting syntax bug using ilev - Fix regridding bug where a single plev is passed and xCDAT does not allow generating bounds for coordinates of len <= 1 -- add conditional that just ignores adding new bounds for regridded output datasets, fix related tests - Fix accidentally calling save plots and metrics twice in `_get_metrics_by_region()` - Fix failing integration tests pass in CI/CD - Refactor `test_diags.py` -- replace unittest with pytest - Refactor `test_all_sets.py` -- replace unittest with pytest - Test climatology datasets -- tested with 3d variables using `test_all_sets.py` CDAT Migration Phase 2: Refactor utilities and CoreParameter methods for reusability across diagnostic sets (#746) - Move driver type annotations to `type_annotations.py` - Move `lat_lon_driver._save_data_metrics_and_plots()` to `io.py` - Update `_save_data_metrics_and_plots` args to accept `plot_func` callable - Update `metrics.spatial_avg` to return an optionally `xr.DataArray` with `as_list=False` - Move `parameter` arg to the top in `lat_lon_plot.plot` - Move `_set_param_output_attrs` and `_set_name_yr_attrs` from `lat_lon_driver` to `CoreParameter` class CDAT Migration Phase 2: Refactor `zonal_mean_2d()` and `zonal_mean_2d_stratosphere()` sets (#774) CDAT Migration Phase 2: Refactor `qbo` set (#826)

tomvothecoder linked an issue Mar 11, 2024 that may be closed by this pull request

CDAT Migration Phase 2: Refactor meridional_mean_2d set #657

Closed

tomvothecoder force-pushed the refactor/657-meridional-mean-2d branch from 5c6eb72 to c907ca0 Compare March 14, 2024 19:41

tomvothecoder marked this pull request as ready for review March 19, 2024 19:26

tomvothecoder added 7 commits March 19, 2024 12:32

Add initial refactored driver

91e331f

Add initial testing scripts

3fa9c84

Update calls to `parameter._set_param_output_attrs

e0bd668

- Reorder these calls for consistency across set drivers

Update calls to `parameter._set_param_output_attrs

87734ca

- Reorder these calls for consistency across set drivers

Refactor plotter and use xesmf for regridding

fc368af

Add regression testing notebook

fcc5782

Delete deprecated plotter and move new plotter to root

c380e77

tomvothecoder force-pushed the refactor/657-meridional-mean-2d branch from d34fea4 to c380e77 Compare March 19, 2024 19:33

tomvothecoder added 2 commits March 19, 2024 12:33

Delete deprecated driver

23c93ac

Fix incorrect plot order due to arg order

18632e3

- Fix incorrect left text position float adjustment

tomvothecoder mentioned this pull request Mar 19, 2024

[Bug]: Regrid2 breaks in _preserve_bounds because _regrid incorrectly transposes output_data shape, which doesn't align with original dim order used by bounds xCDAT/xcdat#625

Closed

tomvothecoder added 2 commits March 19, 2024 15:18

Remove unused code

7d7f1a1

Remove unused add_cyclic function

dc170a4

tomvothecoder commented Mar 19, 2024

View reviewed changes

tomvothecoder requested a review from chengzhuzhang March 19, 2024 22:27

tomvothecoder self-assigned this Mar 19, 2024

tomvothecoder added this to the FY24 Q2 (01/01/24 - 03/31/24) milestone Mar 19, 2024

tomvothecoder added the cdat-migration-fy24 CDAT Migration FY24 Task label Mar 19, 2024

chengzhuzhang approved these changes Mar 22, 2024

View reviewed changes

tomvothecoder merged commit f6c4fdf into cdat-migration-fy24 Mar 25, 2024
4 checks passed

tomvothecoder deleted the refactor/657-meridional-mean-2d branch March 25, 2024 15:59

tomvothecoder added a commit that referenced this pull request May 1, 2024

CDAT Migration: Refactor meridional_mean_2d set (#795)

1e1ab90

tomvothecoder added a commit that referenced this pull request Jul 15, 2024

CDAT Migration: Refactor meridional_mean_2d set (#795)

0170a09

tomvothecoder added a commit that referenced this pull request Jul 15, 2024

CDAT Migration: Refactor meridional_mean_2d set (#795)

710eabf

tomvothecoder mentioned this pull request Jul 31, 2024

CDAT Migration Phase 2: Refactor meridional_mean_2d set #657

Closed

tomvothecoder added a commit that referenced this pull request Aug 21, 2024

CDAT Migration: Refactor meridional_mean_2d set (#795)

5151f61

tomvothecoder added a commit that referenced this pull request Oct 1, 2024

CDAT Migration: Refactor meridional_mean_2d set (#795)

2a95845

tomvothecoder added a commit that referenced this pull request Oct 25, 2024

CDAT Migration: Refactor meridional_mean_2d set (#795)

56b8e3d

tomvothecoder added a commit that referenced this pull request Oct 29, 2024

CDAT Migration: Refactor meridional_mean_2d set (#795)

8014e95

tomvothecoder added a commit that referenced this pull request Oct 29, 2024

CDAT Migration: Refactor meridional_mean_2d set (#795)

908089e

tomvothecoder added a commit that referenced this pull request Dec 4, 2024

CDAT Migration: Refactor meridional_mean_2d set (#795)

bd4fe1b

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CDAT Migration: Refactor `meridional_mean_2d` set #795

CDAT Migration: Refactor `meridional_mean_2d` set #795

tomvothecoder commented Feb 28, 2024 •

edited

Loading

tomvothecoder left a comment

tomvothecoder Mar 19, 2024

chengzhuzhang Mar 22, 2024

tomvothecoder Mar 25, 2024 •

edited

Loading

tomvothecoder Mar 25, 2024

chengzhuzhang Mar 25, 2024

tomvothecoder Mar 19, 2024

chengzhuzhang left a comment

CDAT Migration: Refactor meridional_mean_2d set #795

CDAT Migration: Refactor meridional_mean_2d set #795

Conversation

tomvothecoder commented Feb 28, 2024 • edited Loading

Checklist

tomvothecoder left a comment

Choose a reason for hiding this comment

tomvothecoder Mar 19, 2024

Choose a reason for hiding this comment

chengzhuzhang Mar 22, 2024

Choose a reason for hiding this comment

tomvothecoder Mar 25, 2024 • edited Loading

Choose a reason for hiding this comment

tomvothecoder Mar 25, 2024

Choose a reason for hiding this comment

chengzhuzhang Mar 25, 2024

Choose a reason for hiding this comment

tomvothecoder Mar 19, 2024

Choose a reason for hiding this comment

chengzhuzhang left a comment

Choose a reason for hiding this comment

CDAT Migration: Refactor `meridional_mean_2d` set #795

CDAT Migration: Refactor `meridional_mean_2d` set #795

tomvothecoder commented Feb 28, 2024 •

edited

Loading

tomvothecoder Mar 25, 2024 •

edited

Loading