-
Notifications
You must be signed in to change notification settings - Fork 52
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Replace constants.py with data_config.yaml (#31)
**Summary** This PR replaces the `constants.py` file with a `data_config.yaml` file. Dataset related settings can be defined by the user in the new yaml file. Training specific settings were added as additional flags to the `train_model.py` routine. All respective calls to the old files were replaced. **Rationale** - Using a Yaml file for data config gives much more flexibility for various datasets used in the community. It also facilitates the future use of forcing and boundary datasets. In a follow-up PR the dataset paths will be defined in the yaml file, removing the dependency on a pre-structured `/data` folder. - It is best practice to define user input in a yaml file, the usage of python scripts for that purpose is not common. - The old `constants.py` actually combined both constants and variables, many "constants" should rather be flags to `train_models.py` - The introduction of a new ConfigClass in `utils.py` allows for very specific queries of the yaml and calculations based thereon. This branch shows future possibilities of such a class https://github.com/joeloskarsson/neural-lam/tree/feature_dataset_yaml **Testing** Both training and evaluation of the model were succesfully tested with the `meps_example` dataset. **Note** @leifdenby Could you invite Thomas R. to this repo, in case he wanted to give his input on the yaml file? This PR should mostly serve as a basis for discussion. Maybe we should add more information to the yaml file as you outline in https://github.com/mllam/mllam-data-prep. I think we should always keep in mind how the repository will look like with realistic boundary conditions and zarr-archives as data-input. This PR solves parts of #23 --------- Co-authored-by: Simon Adamov <simon.adamov@mailbox.org>
- Loading branch information
Showing
15 changed files
with
274 additions
and
212 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -7,6 +7,7 @@ graphs | |
| *.sif | ||
| sweeps | ||
| test_*.sh | ||
| .vscode | ||
|
|
||
| ### Python ### | ||
| # Byte-compiled / optimized / DLL files | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,62 @@ | ||
| # Standard library | ||
| import functools | ||
| from pathlib import Path | ||
|
|
||
| # Third-party | ||
| import cartopy.crs as ccrs | ||
| import yaml | ||
|
|
||
|
|
||
| class Config: | ||
| """ | ||
| Class for loading configuration files. | ||
| This class loads a configuration file and provides a way to access its | ||
| values as attributes. | ||
| """ | ||
|
|
||
| def __init__(self, values): | ||
| self.values = values | ||
|
|
||
| @classmethod | ||
| def from_file(cls, filepath): | ||
| """Load a configuration file.""" | ||
| if filepath.endswith(".yaml"): | ||
| with open(filepath, encoding="utf-8", mode="r") as file: | ||
| return cls(values=yaml.safe_load(file)) | ||
| else: | ||
| raise NotImplementedError(Path(filepath).suffix) | ||
|
|
||
| def __getattr__(self, name): | ||
| keys = name.split(".") | ||
| value = self.values | ||
| for key in keys: | ||
| if key in value: | ||
| value = value[key] | ||
| else: | ||
| return None | ||
| if isinstance(value, dict): | ||
| return Config(values=value) | ||
| return value | ||
|
|
||
| def __getitem__(self, key): | ||
| value = self.values[key] | ||
| if isinstance(value, dict): | ||
| return Config(values=value) | ||
| return value | ||
|
|
||
| def __contains__(self, key): | ||
| return key in self.values | ||
|
|
||
| def num_data_vars(self): | ||
| """Return the number of data variables for a given key.""" | ||
| return len(self.dataset.var_names) | ||
|
|
||
| @functools.cached_property | ||
| def coords_projection(self): | ||
| """Return the projection.""" | ||
| proj_config = self.values["projection"] | ||
| proj_class_name = proj_config["class"] | ||
| proj_class = getattr(ccrs, proj_class_name) | ||
| proj_params = proj_config.get("kwargs", {}) | ||
| return proj_class(**proj_params) |
This file was deleted.
Oops, something went wrong.
Oops, something went wrong.