From 0342316987268201ff0a1e3bed401e3f52f366e5 Mon Sep 17 00:00:00 2001 From: Andreas Pauling Date: Thu, 30 May 2024 09:22:33 +0200 Subject: [PATCH] integraton of reviewer comments --- README.md | 38 +++++++------- config.yaml | 12 ++--- .../mutable_number.py | 51 ------------------- .../update_phenology.py | 11 ++++ .../update_strength.py | 11 ++++ src/realtime_pollen_calibration/utils.py | 11 ++++ tools/setup_env.sh | 8 ++- 7 files changed, 60 insertions(+), 82 deletions(-) delete mode 100644 src/realtime_pollen_calibration/mutable_number.py diff --git a/README.md b/README.md index 409dbea..8485a94 100644 --- a/README.md +++ b/README.md @@ -57,9 +57,6 @@ tools/setup_env.sh -u -e -n *Hint*: If you are the package administrator, it is a good idea to understand what this script does, you can do everything manually with `conda` instructions. -*Hint*: Use the flag `-m` to speed up the installation using mamba. Of course you will have to install mamba first (we recommend to install mamba into your base environment `conda install -c conda-forge mamba`). If you install mamba in another (maybe dedicated) environment, environments installed with mamba will be located -in `/envs/mamba/envs`, which is not very practical. - The package itself is installed with `pip`. For development, install in editable mode: ```bash @@ -94,6 +91,8 @@ The module `update_phenology` uses observed pollen concentrations to check wheth The module `update_strength` uses both observed and modelled pollen concentrations to check whether the current ICON concentrations match the real world. In case of a mismatch the tuning field is adapted accordingly. Technically speaking, the field `tune` is adapted by using the GRIB2 field `saisn` and the observed and modelled pollen concentrations of the last 120 hours at hourly resolution in ATAB format (missing data supported). +Missing fields result in an error and thus no update of the fields. The input fields remain unchanged. If not all station data (observed or modelled) is available, the extraction will still take place and the update of the fields will be tried. However, input fields will remain unchanged if more than 10% of the data is missing. + For further details of the realtime pollen calibration concept one may refer to the paper above. @@ -104,41 +103,40 @@ For further details of the realtime pollen calibration concept one may refer to The implementation includes a command line interface based on the click package. The configuration is done by editing the config.yaml file where the input/output is specified. There is the option to configure the increment of the timestamp of the outfile relative to the infile in hours. The config.yaml should include the following entries (see also the config.yaml provided): ```bash -POV_infile : /store_new/mch/msopr/paa/RTcal_testdata/ART_POV_iconR19B08-grid_0001.gb2 -POV_outfile : ART_POV_iconR19B08-grid_0001_tune -T2M_file : /store_new/mch/msopr/paa/RTcal_testdata/T_2M_KENDA-CH1_2024020118.gb2 -const_file : /store_new/mch/msopr/paa/RTcal_testdata/CLON_CLAT_ICON-CH1.gb2 -station_obs_file : /store_new/mch/msopr/paa/RTcal_testdata/pollen_measured_2024020118.atab -station_mod_file : /store_new/mch/msopr/paa/RTcal_testdata/pollen_modelled_2024020118.atab +POV_infile : /ART_POV_iconR19B08-grid_0001.gb2 +POV_outfile : /ART_POV_iconR19B08-grid_0001_tune +T2M_file : /T_2M_KENDA-CH1_2024020118.gb2 +const_file : /CLON_CLAT_ICON-CH1.gb2 +station_obs_file : /pollen_measured_2024020118.atab +station_mod_file : /pollen_modelled_2024020118.atab hour_incr : 1 ``` -`POV_infile`: This GRIB2 file must include the fields specified above. It is used as template for `POV_outfile` +`POV_infile`: This GRIB2 file must include the fields `tthrs`, `tthre` (for POAC, `saisl` instead), `saisn` and `ctsum` if the module `update_phenology` is called. If the module `update_strength` is called `POV_infile` must include the fields `saisn` and `tune`. `POV_infile` is used as template for `POV_outfile`, i.e. the whole file is copied to `POV_outfile` with adapted values. Date and time information of `POV_infile` does not have to be correct, ICON just throws warnings. `POV_outfile`: Same as `POV_infile` but with adapted values. `T2M_file`: This GRIB2 file must include T_2M (only used if the module `update_phenology` is called). `const_file`: This GRIB2 file must contain CLON and CLAT of the unstructured grid used in `POV_infile` and `T_2M`. -`station_obs_file`: Observed pollen concentrations of the last 120 hours (missing values allowed) in ATAB format. -`station_mod_file`: Modelled pollen concentrations of the last 120 hours (missing values allowed) in ATAB format. Same stations as in `station_obs_file` (only used if the module `update_strength` is called). -`hour_incr`: Increment of the timestamp of the outfile relative to the infile in hours (defaults to 1; negative values also supported). +`station_obs_file`: Observed hourly pollen concentrations (ATAB format) of the latest 120 hours relative to the target date of `POV_outfile`. The timestamps of the data in this file may vary depending on data availability, time of extraction etc. Missing values are allowed. +`station_mod_file`: Modelled hourly pollen concentrations (ATAB format) of the latest 120 hours relative to the target date of `POV_outfile`. The timestamps of the data in this file may vary depending on data availability, time of extraction etc. Missing values are allowed. Same stations as in `station_obs_file` (only used if the module `update_strength` is called). +`hour_incr`: Increment of the timestamp of the outfile relative to the infile in hours (defaults to 1; negative values also supported). This parameter should be adapted if the calibration is done for a subsequent run more than one hour ahead. ### How to run the package The two modules are called this way: ```bash -cd conda activate -python src/realtime_pollen_calibration/run.py update_phenology config.yaml -python src/realtime_pollen_calibration/run.py update_strength config.yaml +realtime-pollen-calibration update_phenology /config.yaml +realtime-pollen-calibration update_strength /config.yaml ``` Help functionalities are also available: ```bash -python src/realtime_pollen_calibration/run.py --help -python src/realtime_pollen_calibration/run.py update_phenology --help -python src/realtime_pollen_calibration/run.py update_strength --help +realtime_pollen_calibration --help +realtime_pollen_calibration update_phenology --help +realtime_pollen_calibration update_strength --help ``` -The implementation assumes hourly resolution of the modelled and observed pollen concentrations (ATAB files). Hence, updating the tuning field once per hour is recommended. The phenology model of ICON makes one step per day at 12 UTC. Hence, we recommend to update the phenological fields also once per day some time before 12 UTC so that the updated fields can be used by the phenology model of ICON at 12 UTC. +The implementation assumes hourly resolution of the modelled and observed pollen concentrations (ATAB files). Hence, updating the tuning field once per hour is recommended. The phenology model of ICON is called once per day at 12 UTC model time. Hence, we recommend to update the phenological fields (i.e. `tthrs` and `tthre` (for POAC, `saisl` instead of `tthre`)) also once per day some time before 12 UTC (model time) so that the updated fields can be used by ICON. ### Unit test diff --git a/config.yaml b/config.yaml index 5a76a54..75b4815 100644 --- a/config.yaml +++ b/config.yaml @@ -1,8 +1,8 @@ -pov_infile : /store_new/mch/msopr/paa/RTcal_testdata/ART_POV_iconR19B08-grid_0001_BETU_POAC_2024042910 -# pov_infile : /store_new/mch/msopr/paa/RTcal_testdata/ART_POV_iconR19B08-grid_0001.gb2 +pov_infile : /users/paa/09_RTcalib/RTcal_testdata/ART_POV_iconR19B08-grid_0001_BETU_POAC_2024042910 +# pov_infile : /users/paa/09_RTcalib/RTcal_testdata/ART_POV_iconR19B08-grid_0001.gb2 pov_outfile : ART_POV_iconR19B08-grid_0001_tune -t2m_file : /store_new/mch/msopr/paa/RTcal_testdata/T_2M_KENDA-CH1_2024020118.gb2 -const_file : /store_new/mch/msopr/paa/RTcal_testdata/CLON_CLAT_ICON-CH1.gb2 -station_obs_file : /store_new/mch/msopr/paa/RTcal_testdata/pollen_measured_2024020118.atab -station_mod_file : /store_new/mch/msopr/paa/RTcal_testdata/pollen_modelled_2024020118.atab +t2m_file : /users/paa/09_RTcalib/RTcal_testdata/T_2M_KENDA-CH1_2024020118.gb2 +const_file : /users/paa/09_RTcalib/RTcal_testdata/CLON_CLAT_ICON-CH1.gb2 +station_obs_file : /users/paa/09_RTcalib/RTcal_testdata/pollen_measured_2024020118.atab +station_mod_file : /users/paa/09_RTcalib/RTcal_testdata/pollen_modelled_2024020118.atab hour_incr : 1 diff --git a/src/realtime_pollen_calibration/mutable_number.py b/src/realtime_pollen_calibration/mutable_number.py deleted file mode 100644 index a5ad63c..0000000 --- a/src/realtime_pollen_calibration/mutable_number.py +++ /dev/null @@ -1,51 +0,0 @@ -"""Mutable number.""" - -# Standard library -from typing import List - - -class MutableNumber: - """A mutable number.""" - - def __init__(self, number: float) -> None: - """Create an instance of ``MutableNumber``. - - Args: - number: Initial number. - - """ - self.history: List[float] = [float(number)] - - def get(self, idx: int = -1) -> float: - """Get the current or a past value of the number. - - Args: - idx (optional): Index since the initial value. Defaults to the most - recent (i.e., current) value. - - """ - return self.history[idx] - - def add(self, addend: float) -> "MutableNumber": - """Add ``addend`` to the current number.""" - number = self.get() + float(addend) - self.history.append(number) - return self - - def subtract(self, subtrahend: float) -> "MutableNumber": - """Subtract ``subtrahend`` from the current number.""" - number = self.get() - float(subtrahend) - self.history.append(number) - return self - - def multiply(self, factor: float) -> "MutableNumber": - """Multiply current number by ``factor``.""" - number = self.get() * float(factor) - self.history.append(number) - return self - - def divide(self, divisor: float) -> "MutableNumber": - """Divide current number by ``divisor``.""" - number = self.get() / float(divisor) - self.history.append(number) - return self diff --git a/src/realtime_pollen_calibration/update_phenology.py b/src/realtime_pollen_calibration/update_phenology.py index 8560c9d..1d4d886 100644 --- a/src/realtime_pollen_calibration/update_phenology.py +++ b/src/realtime_pollen_calibration/update_phenology.py @@ -24,6 +24,17 @@ def read_pov_file(pov_infile, pol_fields): + """Read fields from pov_infile as defined in config.yaml. + + Args: + pov_infile: GRIB2 file containing pollen fields. + pol_fields: Names of the pollen fields. + config_obj: Object containing the configuration + + Returns: + Fields for the pollen calibration. + + """ cal_fields = {} with open(pov_infile, "rb") as fh: while True: diff --git a/src/realtime_pollen_calibration/update_strength.py b/src/realtime_pollen_calibration/update_strength.py index 25c27ed..5136234 100644 --- a/src/realtime_pollen_calibration/update_strength.py +++ b/src/realtime_pollen_calibration/update_strength.py @@ -24,6 +24,17 @@ def read_pov_file(pov_infile, pol_fields, config_obj): + """Read fields from pov_infile as defined in config.yaml. + + Args: + pov_infile: GRIB2 file containing pollen fields. + pol_fields: Names of the pollen fields. + config_obj: Object containing the configuration set in config.yaml + + Returns: + Fields needed for the pollen calibration and the new timestamp. + + """ time_values = None cal_fields = {} with open(pov_infile, "rb") as fh: diff --git a/src/realtime_pollen_calibration/utils.py b/src/realtime_pollen_calibration/utils.py index aeb5922..3d7abee 100644 --- a/src/realtime_pollen_calibration/utils.py +++ b/src/realtime_pollen_calibration/utils.py @@ -64,9 +64,20 @@ class Config: ) pollen_types = ["ALNU", "BETU", "POAC", "CORY"] + +# thr_con_24 and thr_con_120 are thresholds for sums of hourly observed +# pollen observations used to make sure that pollen calibration is only +# performed if pollen concentrations were high enough to ensure robust +# results of the pollen calibration. thr_con_24 = {"ALNU": 240, "BETU": 240, "POAC": 72, "CORY": 240} thr_con_120 = {"ALNU": 720, "BETU": 720, "POAC": 216, "CORY": 720} + +# failsafe is a limiter for the change applied to the phenological fields +# tthrs and tthre (and saisl for POAC instead of tthre). failsafe = {"ALNU": 1000, "BETU": 2500, "POAC": 6000, "CORY": 2500} + +# jul_days_excl is the number of days since Dec. 1 to be excluded +# in the calculation of the temperature sum jul_days_excl = {"ALNU": 14, "BETU": 40, "POAC": 46, "CORY": 3} diff --git a/tools/setup_env.sh b/tools/setup_env.sh index f7e6c51..d8dc52e 100755 --- a/tools/setup_env.sh +++ b/tools/setup_env.sh @@ -19,14 +19,13 @@ EXPORT=false CONDA=conda HELP=false -help_msg="Usage: $(basename "${0}") [-n NAME] [-p VER] [-u] [-e] [-m] [-h] +help_msg="Usage: $(basename "${0}") [-n NAME] [-p VER] [-u] [-e] [-h] Options: -n NAME Env name [default: ${DEFAULT_ENV_NAME} -p VER Python version [default: ${PYVERSION}] -u Use unpinned requirements (minimal version restrictions) -e Export environment files (requires -u) - -m Use mamba instead of conda -h Print this help message and exit " @@ -37,7 +36,6 @@ while getopts n:p:defhimu flag; do p) PYVERSION=${OPTARG};; e) EXPORT=true;; h) HELP=true;; - m) CONDA=mamba;; u) PINNED=false;; ?) echo -e "\n${help_msg}" >&2; exit 1;; esac @@ -49,8 +47,8 @@ if ${HELP}; then fi echo "Setting up environment for installation" -eval "$(conda shell.bash hook)" || exit # NOT ${CONDA} (doesn't work with mamba) -conda activate || exit # NOT ${CONDA} (doesn't work with mamba) +eval "$(conda shell.bash hook)" || exit +conda activate || exit # Create new env; pass -f to overwriting any existing one echo "Creating ${CONDA} environment"