[Refactor]: Refactor integration tests and add use_cfg flag to run_diags()

[tomvothecoder]

Description

Overview

This PR refactors the integration tests to improve maintainability by eliminating redundant code and making tests more readable. It also adds the use_cfg flag to run_diags(). This flag determines whether to use .cfg file(s) specified either by (1) -d/--diags (2) default diags. The goal is to make it easier to debug only a few diagnostic sets by setting use_cfg=False.

Changes

• Refactor integration tests (tests/integration)
  • Delete test_dataset.py because it is a really old and incomplete test file for the legacy Dataset class based on CDAT
  • Delete test_all_sets.py and all_sets_modified.cfg because it tests for expected image counts which is redundant (test_all_sets_image_diffs.py does this already)
  • Replace subprocess calls with direct call to Python API for real-time test results and easier debugging
  • Move complete_run.py to /test/integration
  • Move test_run.py to tests/e3sm_diags since it is more of a unit test file
• Refactor Run class (run.py)
  • Closes [Refactor]: Stop default diagnostics always running if e3sm_diags is executed via Python script #735
  • Add use_cfg boolean argument to run_diags() and get_run_parameters()
  • Add self.cfg_path attribute, which is set if .cfg file(s) are used for diagnostic runs
  • Add is_cfg_file_arg_set() property to check parser for -d/--diags
  • Rename get_final_parameters() to get_run_parameters() and refactored to smaller methods
• Update CI/CD build workflow (build_workflow.yml)
  • Split up testing step into: 1) run unit tests 2) download integration test data 3) run integration tests
  • Easier to keep track of runtime and results

Example script

import os

from e3sm_diags.parameter.core_parameter import CoreParameter
from e3sm_diags.run import runner

param = CoreParameter()
# Location of the data.
param.test_data_path = "/global/cfs/cdirs/e3sm/e3sm_diags/test_model_data_for_acme_diags/time-series/E3SM_v1"
param.reference_data_path = "/global/cfs/cdirs/e3sm/e3sm_diags/test_model_data_for_acme_diags/time-series/E3SM_v1"

# Set this parameter to True.
# By default, e3sm_diags expects the test data to be climo data.
param.test_timeseries_input = True
# Years to slice the test data, base this off the years in the filenames.
param.test_start_yr = "2011"
param.test_end_yr = "2013"

# Set this parameter to True.
# By default, e3sm_diags expects the ref data to be climo data.
param.ref_timeseries_input = True
# Years to slice the ref data, base this off the years in the filenames.
param.ref_start_yr = "1850"
param.ref_end_yr = "1852"

# When running with time-series data, you don't need to specify the name of the data.
# But you should, otherwise nothing is displayed when the test/ref name is needed.
param.short_test_name = "historical_H1"
param.short_ref_name = "historical_H1"

# This parameter modifies the software to accommodate model vs model runs.
# The default setting for run_type is 'model_vs_obs'.
param.run_type = "model_vs_model"
# Name of the folder where the results are stored.
# Change `prefix` to use your directory.
prefix = "/global/cfs/cdirs/e3sm/www/<your directory>/examples"
param.results_dir = os.path.join(prefix, "ex1_modTS_vs_modTS_3years")

# Below are more optional arguments.

# What plotsets to run the diags on.
# If not defined, then all available sets are used.
param.sets = ["lat_lon"]
# What seasons to run the diags on.
# If not defined, diags are run on ['ANN', 'DJF', 'MAM', 'JJA', 'SON'].
param.seasons = ["ANN"]
# Title of the difference plots.
param.diff_title = "Model (2011-2013) - Model (1850-1852)"
# For running with multiprocessing.
# param.multiprocessing = True
# param.num_workers = 32

runner.sets_to_run = ["lat_lon"]
runner.run_diags([param], use_cfg=False)

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• My changes generate no new warnings
• Any dependent changes have been merged and published in downstream modules

If applicable:

• New and existing unit tests pass with my changes (locally and CI/CD build)
• I have added tests that prove my fix is effective or that my feature works
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have noted that this is a breaking change for a major release (fix or feature that would cause existing functionality to not work as expected)

[tomvothecoder]

Some review comments for areas of interest.

[tomvothecoder]

Currently blocked by #758

EDIT 12/13/23 -- no longer blocked by #758

[tomvothecoder]

Will do a final review and merge this after #766

[tomvothecoder]

parameter_results are now returned so that we can parse them for the results_dir in unit/integration tests.

It is also generally useful to see the results after each diagnostic run.