JJB tropical subseasonal diags - Wheeler and Kiladis #732
Conversation
…ethod (#742) - Update VSCode workspace settings with extension and debugger configs
- Update conda env dependencies - Add `dev-nompi.yml` which includes the nompi version of esmf to avoid the yaksa leak error that breaks VS Code testing API - Update pre-commit dependency versions and fix new issues for `black`, `flake8`, and `mypy` - Move most of `setup.cfg` to `pyproject.toml` - Add `Makefile` with convenient developer commands - Update `test_diags.py` and `test_allsets.py` to use `pytest`
* Add aburdens to lat-lon * Add some ERF calculations * Add mostly auto-generated docs to erf calc * Remove tag entry in lat-lon set * Fix contour levels in lat-lon for burdens * Add more aerosol variables and specs * Add specs for cdnc/lwp vars Co-authored-by: Johannes Mülmenstädt <johannes.muelmenstaedt@pnnl.gov> Co-authored-by: Susannah Burrows <Susannah.Burrows@pnnl.gov> Co-authored-by: Tom Vo <tomvothecoder@gmail.com>
…_diags()` (#747) * Refactor integration tests (_tests/integration_) * Delete `test_dataset.py` because it is a really old and incomplete test file for the legacy `Dataset` class based on CDAT * Delete `test_all_sets.py` and `all_sets_modified.cfg` because it tests for expected image counts which is redundant (`test_all_sets_image_diffs.py` does this already) * Replace `subprocess` calls with direct call to Python API for real-time test results and easier debugging * Move `complete_run.py` to `/test/integration` * Move `test_run.py` to `tests/e3sm_diags` since it is more of a unit test file * Refactor `Run` class (_run.py_) * Closes #735 * Add `use_cfg` boolean argument to `run_diags()` and `get_run_parameters()` * Add `self.cfg_path` attribute, which is set if `.cfg` file(s) are used for diagnostic runs * Add `is_cfg_file_arg_set()` property to check parser for `-d/--diags` * Rename `get_final_parameters()` to `get_run_parameters()` and refactored to smaller methods * Update CI/CD build workflow (_build_workflow.yml_) * Split up testing step into: 1) run unit tests 2) download integration test data 3) run integration tests * Easier to keep track of runtime and results
* support monthly average in viewer * add 12 monthly mean to core sets as options * Initial modifications for EAMxx variable names * add monthly mean option for aerosol budget tables * update SCREAM variable convention * more update for supporting monthly data --------- Co-authored-by: Chris Golaz <golaz1@llnl.gov>
|
@jjbenedict We are on good track to complete the wavenumber frequency diagram. Here is the testing results compare a 5 year model run with the IMERG output you processed: https://portal.nersc.gov/cfs/e3sm/chengzhu/tests/tropical_diags/tropical_variability_model_obs/viewer/tropical_subseasonal/wavenumber-frequency/prect-norm_sym-imerg_daily/plot.html Though I encountered errors when testing ORL/FLUT variable. The error is from the zwf_function script and reads as follows: I'm quite puzzled why the same codes works on |
|
@chengzhuzhang I'm trying to run zwf_plot.py locally, and I'm also getting errors with the OLR file. There may be something wrong with either the file itself or how python reads in the file. I'm testing a few things now... |
|
Hi @jjbenedict I have updated the READ.me for including the wavenumber frequency plot. a8bcf9e Could you let me know if other contributors I should add? I have fixed the dispersion curves for the asymmetric plot, I can help to update the plotting for the other issue if you haven't started. After that, I think this branch is ready to merge. (I also fixed most review comments @tomvothecoder provided. ) |
|
@jjbenedict I'm implementing this new set with zppy, and ran some tests with the v3 picontrol data. And here is the result I have: https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.zhang40/E3SMv3_trop/v3.LR.piControl/e3sm_diags/atm_monthly_180x360_aave/model_vs_obs_0001-0050/viewer/tropical_subseasonal/index.html I think we are very close to merge this feature. Let me know if anything I can help to refine the plots.. |
|
@chengzhuzhang Apologies for the delay, I'm working on the dispersion curve fix now (having dispersion curve lines cross the zero zonal wavenumber... |
…o and add missing curves to most plots
|
@chengzhuzhang I cloned the repo, made edits to zwf_plot.py, and committed the changes (locally). I tried to push back to origin but got an authentication error... I think I need to set up the authentication in github, and I know I've done this before and can probably figure it out. I'll try to do this over the weekend, otherwise early Monday. I had one or two other questions (I'll save those for Monday because I have to leave the office now), but one other minor note: For the asymmetric plots you shared with me, none show the "MRG" wave label. I can see the line in zwf_plot.py that adds the "MRG" label to the plot, but for some reason it didn't show up. The "MRG" label -does- show up in plots that I make using my local version of zwf_plot.py... and the code appears to match what I see from the E3SM Diags repo. We can revisit this Monday. |
|
@jjbenedict thanks for the updates. I will be out of office next week, it looks like we are near completion of this feature. I will get back and we can work towards releasing the week of April 8th. I re-organized the plotting into e3sm_diags/plot/cartopy/tropical_subseasonal_plot.py it could be something I messed up in displaying the labels... |
|
@chengzhuzhang @tomvothecoder I noticed that my Github personal access token had expired again, so I renewed it. I also noticed that I am no longer in the e3sm_diags UNIX group, but I was still able to push my commits to the remote repo. Where we're at now: The commits I just made fix minor bugs related to the shallow water dispersion curves on the plots (some "jumped" the zero zonal wavenumber value). These edits also plot the dispersion lines that were missing on some plots, and label the lines. I've tested these code mods on my local version of the script and they work. @chengzhuzhang could you test these in your workflow? Note that I decided to NOT plot dispersion curves for the "background" spectral plots, following the WK99 paper... but we could decide to change this down the road. Next:
The suggested contour changes should reduce "off-scaling" of large plotted values. And, re: (4) and (5), I hadn't fully tested the WK plotting routines with U850... hopefully these contour changes improve the plots. I think we're extremely close to a final product!! |
|
Hey @jjbenedict I believe all the issues and suggestions from your comment (#732 (comment)) are addressed in my recent commits. Please have a review of the new plots. Please review the results here: https://portal.nersc.gov/cfs/e3sm/chengzhu/tests/tropical_diags_subsetting/tropical_variability_model_obs_refine/viewer/tropical_subseasonal/index.html |
|
@tomvothecoder I'm puzzled about he CI/CD failure, it seems to be downloading data for integration testing was problematic. There is a |
I experimented different combinations of permissions (including 777) for the https://web.lcrc.anl.gov/public/e3sm/e3sm_diags_test_data/integration/expected/prov_cp/ |
|
@jjbenedict I'm preparing a release candidate to test with e3sm unified new release. I will merge this PR for now, but let me know if you'd like any change, we can update in the final release. Thank you! |
|
Awesome work @chengzhuzhang! |
|
@chengzhuzhang and @tomvothecoder -- Super exciting and thanks so much for your help with making this a reality!! The plots look awesome, Jill. And thanks for being patient with my pickiness and novice github knowledge. I expect implementation of the next round of tropical subseasonal variability plots (lag correlation, possibly others) to go more smoothly and quickly. Let me know if anything else comes up regarding the WK plots. |
|
@jjbenedict Thank you so much for your contribution. And I think this set is in great shape with you careful review and thoughtful requests! We are looking forward to working with you again for new enhancements for the tropical variability set. |
This PR is to create the first set of diagnostics for tropical sub-seasonal variability, wave number and frequency analysis from (wheeler Kiladis 1999).
Original authors of the scripts are Jim Benedict and Brian Medeiros.