Merge pull request #82 from ecmwf/develop

Release `0.4.1`
ecmwf · Nov 26, 2024 · 7f0b63b · 7f0b63b
2 parents 7d09108 + 809efc5
commit 7f0b63b
Show file tree

Hide file tree

Showing 68 changed files with 1,927 additions and 229 deletions.
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -27,7 +27,7 @@ repos:
     -   id: python-check-blanket-noqa # Check for # noqa: all
     -   id: python-no-log-warn # Check for log.warn
 - repo: https://github.com/psf/black-pre-commit-mirror
-  rev: 24.8.0
+  rev: 24.10.0
   hooks:
   - id: black
     args: [--line-length=120]
@@ -40,7 +40,7 @@ repos:
     - --force-single-line-imports
     - --profile black
 - repo: https://github.com/astral-sh/ruff-pre-commit
-  rev: v0.6.9
+  rev: v0.7.2
   hooks:
   - id: ruff
     args:
@@ -59,13 +59,8 @@ repos:
   hooks:
   - id: rstfmt
     exclude: 'cli/.*' # Because we use argparse
-- repo: https://github.com/b8raoult/pre-commit-docconvert
-  rev: "0.1.5"
-  hooks:
-  - id: docconvert
-    args: ["numpy"]
 - repo: https://github.com/tox-dev/pyproject-fmt
-  rev: "2.2.4"
+  rev: "v2.5.0"
   hooks:
   - id: pyproject-fmt
 -   repo: https://github.com/jshwi/docsig # Check docstrings against function sig

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -10,6 +10,16 @@ Keep it human-readable, your future self will thank you!
 
 ## [Unreleased](https://github.com/ecmwf/anemoi-graphs/compare/0.4.0...HEAD)
 
+### Added
+- feat: Define node sets and edges based on an ICON icosahedral mesh (#53)
+- feat: Add support for `post_processors` in the recipe. (#71)
+- feat: Add `RemoveUnconnectedNodes` post processor to clean unconnected nodes in LAM. (#71)
+- feat: Define node sets and edges based on an ICON icosahedral mesh (#53)
+- feat: Support for multiple edge builders between two sets of nodes (#70)
+
+# Changed
+- fix: bug when computing area weights with scipy.Voronoi. (#79)
+
 ## [0.4.0 - LAM and stretched graphs](https://github.com/ecmwf/anemoi-graphs/compare/0.3.0...0.4.0) - 2024-11-08
 
 ### Added
@@ -27,6 +37,7 @@ Keep it human-readable, your future self will thank you!
 - Added `CutOutMask` class to create a mask for a cutout. (#68)
 - Added `MissingZarrVariable` and `NotMissingZarrVariable` classes to create a mask for missing zarr variables. (#68)
 - feat: Add CONTRIBUTORS.md file. (#72)
+- Create package documentation.
 
 ### Changed
 
@@ -43,6 +54,8 @@ Keep it human-readable, your future self will thank you!
 - ci: extened python versions to include 3.11 and 3.12
 - Update copyright notice
 - Fix `__version__` import in init
+- The `edge_builder` field in the recipe is renamed to `edge_builders`. It now receives a list of edge builders. (#70)
+- The `{source|target}_mask_attr_name` field is moved to inside the edge builder definition. (#70)
 
 ## [0.3.0 Anemoi-graphs, minor release](https://github.com/ecmwf/anemoi-graphs/compare/0.2.1...0.3.0) - 2024-09-03
 

diff --git a/docs/cli/inspect.rst b/docs/cli/inspect.rst
@@ -4,7 +4,8 @@
 inspect
 ========
 
-Use this command to inspect a graph stored in your filesystem.
+Use this command to inspect a graph stored in your filesystem. A set of interactive and static visualisations
+are generated to allow visual inspection of the graph design.
 
 The syntax of the recipe file is described in :doc:`building graphs <../graphs/introduction>`.
 

diff --git a/docs/conf.py b/docs/conf.py
@@ -29,15 +29,15 @@
 
 project = "Anemoi Graphs"
 
-author = "ECMWF"
+author = "Anemoi contributors"
 
 year = datetime.datetime.now().year
 if year == 2024:
     years = "2024"
 else:
     years = "2024-%s" % (year,)
 
-copyright = "%s, ECMWF" % (years,)
+copyright = "%s, Anemoi contributors" % (years,)
 
 try:
     from anemoi.graphs._version import __version__

diff --git a/docs/graphs/edge_attributes.rst b/docs/graphs/edge_attributes.rst
@@ -20,8 +20,9 @@ coordinates.
 .. code:: yaml
 
    edges:
-     - ...
-       edge_builder: ...
+     - source_name: ...
+       target_name: ...
+       edge_builders: ...
        attributes:
          edge_length:
             _target_: anemoi.graphs.edges.attributes.EdgeLength
@@ -37,8 +38,9 @@ latitude and longitude coordinates of the source and target nodes.
 .. code:: yaml
 
    edges:
-     - ...
-       edge_builder: ...
+     - source_name: ...
+       target_name: ...
+       edge_builders: ...
        attributes:
          edge_length:
             _target_: anemoi.graphs.edges.attributes.EdgeDirection
diff --git a/docs/graphs/edges.rst b/docs/graphs/edges.rst
@@ -14,8 +14,8 @@ for each (`source name`, `target name`) pair specified.
    edges:
      - source_name: data
        target_name: hidden
-       edge_builder:
-         _target_: anemoi.graphs.edges.CutOff
+       edge_builders:
+       - _target_: anemoi.graphs.edges.CutOff
          cutoff_factor: 0.7
 
 Below are the available methods for defining the edges:
@@ -26,3 +26,19 @@ Below are the available methods for defining the edges:
    edges/cutoff
    edges/knn
    edges/multi_scale
+
+Additionally, there are 2 extra arguments (``source_mask_attr_name`` and
+``target_mask_attr_name``) that can be used in the edge configuration to
+mask source and/or target nodes. This can be useful to different use
+cases, such as Limited Area Modeling (LAM) where your decoder edges
+should only connect to the nodes in the limited area.
+
+.. code:: yaml
+
+   edges:
+     - source_name: hidden
+       target_name: data
+       edge_builders:
+       - _target_: anemoi.graphs.edges.KNNEdges
+         num_nearest_neighbours: 5
+         target_mask_attr_name: cutout
diff --git a/docs/graphs/edges/cutoff.rst b/docs/graphs/edges/cutoff.rst
@@ -39,13 +39,13 @@ YAML configuration:
    edges:
       -  source_name: source
          target_name: destination
-         edge_builder:
-            _target_: anemoi.graphs.edges.CutOffEdges
+         edge_builders:
+         -  _target_: anemoi.graphs.edges.CutOffEdges
             cutoff_factor: 0.6
 
 .. note::
 
-   The cut-off method is recommended for the encoder edges, to connect
+   The cut-off method is recommended for the encoder edge, to connect
    all data nodes to hidden nodes. The optimal ``cutoff_factor`` value
    will be the lowest value without orphan nodes. This optimal value
    depends on the node distribution, so it is recommended to tune it for

diff --git a/docs/graphs/edges/knn.rst b/docs/graphs/edges/knn.rst
@@ -1,11 +1,11 @@
-#####################
- K-Nearest Neighbors
-#####################
+######################
+ K-Nearest Neighbours
+######################
 
 The knn method is a method for establishing connections between two sets
 of nodes. Given two sets of nodes, (`source`, `target`), the knn method
-connects all destination nodes, to their ``num_nearest_neighbours``
-nearest source nodes.
+connects all target nodes, to their ``num_nearest_neighbours`` nearest
+source nodes.
 
 To use this method to build your connections, you can use the following
 YAML configuration:
@@ -14,12 +14,12 @@ YAML configuration:
 
    edges:
      -  source_name: source
-        target_name: destination
-        edge_builder:
-          _target_: anemoi.graphs.edges.KNNEdges
+        target_name: target
+        edge_builders:
+        - _target_: anemoi.graphs.edges.KNNEdges
           num_nearest_neighbours: 3
 
 .. note::
 
-   The knn method is recommended for the decoder edges, to connect all
-   data nodes with the surrounding hidden nodes.
+   The KNNEdges method is recommended for the decoder edges, to connect
+   all target nodes with the surrounding source nodes.
diff --git a/docs/graphs/edges/multi_scale.rst b/docs/graphs/edges/multi_scale.rst
@@ -5,7 +5,7 @@
 The multi-scale connections can only be defined with the same source and
 target nodes. Edges of different scales are defined based on the
 refinement level of an icosahedron. The higher the refinement level, the
-shorter the length of the edges. By default, all possible refinements
+shorther the length of the edges. By default, all possible refinements
 levels are considered.
 
 To use this method to build your connections, you can use the following
@@ -16,8 +16,8 @@ YAML configuration:
    edges:
      -  source_name: source
         target_name: source
-        edge_builder:
-          _target_: anemoi.graphs.edges.MultiScaleEdges
+        edge_builders:
+        - _target_: anemoi.graphs.edges.MultiScaleEdges
           x_hops: 1
 
 where `x_hops` is the number of hops between two nodes of the same
@@ -28,8 +28,12 @@ refinement level to be considered neighbours, and then connected.
    This method is used by data-driven weather models like GraphCast to
    process the latent/hidden state.
 
+.. csv-table:: Triangular refinements specifications (x_hops=1)
+   :file: ./tri_refined_edges.csv
+   :header-rows: 1
+
 .. warning::
 
-   This connection method is only supported for building the connections
+   This connection method is only support for building the connections
    within a set of nodes defined with the ``TriNodes`` or ``HexNodes``
    classes.
diff --git a/docs/graphs/edges/tri_refined_edges.csv b/docs/graphs/edges/tri_refined_edges.csv
@@ -0,0 +1,8 @@
+Refinement,Num Nodes,Num Edges,Num Multilevel Edges
+0,12,60,60
+1,42,240,300
+2,162,960,1260
+3,642,3840,5100
+4,2562,15360,20460
+5,10242,61440,81900
+6,40962,245760,327660
diff --git a/docs/graphs/introduction.rst b/docs/graphs/introduction.rst
@@ -65,6 +65,7 @@ following classes define different behaviour:
 
 -  :doc:`node_coordinates/zarr_dataset`
 -  :doc:`node_coordinates/npz_file`
+-  :doc:`node_coordinates/icon_mesh`
 -  :doc:`node_coordinates/tri_refined_icosahedron`
 -  :doc:`node_coordinates/hex_refined_icosahedron`
 -  :doc:`node_coordinates/healpix`

diff --git a/docs/graphs/node_attributes.rst b/docs/graphs/node_attributes.rst
@@ -21,3 +21,12 @@ nodes.
    :maxdepth: 1
 
    node_attributes/weights
+   node_attributes/zarr_attribute
+
+Additionally, different boolean operations have been implemented to
+support more complex use cases:
+
+.. toctree::
+   :maxdepth: 1
+
+   node_attributes/boolean_operations
diff --git a/docs/graphs/node_attributes/boolean_operations.rst b/docs/graphs/node_attributes/boolean_operations.rst
@@ -0,0 +1,12 @@
+####################
+ Boolean operations
+####################
+
+_anemoi-graphs_ package implements a set of boolean opearations to
+support these operations when defining node attributes. Below, an
+attribute `mask` is computed as the intersection of two other masks,
+that are generated as the non-missing values in 2 different variables in
+a Zarr dataset.
+
+.. literalinclude:: ../yaml/attributes_boolean_operation.yaml
+   :language: yaml
diff --git a/docs/graphs/node_attributes/weights.rst b/docs/graphs/node_attributes/weights.rst
@@ -2,7 +2,7 @@
  Weights
 #########
 
-The `weights` are a node attribute useful for defining the importance of
+The `weights` are node attributes useful for defining the importance of
 a node in the loss function. You can set the weights to follow an
 uniform distribution or to match the area associated with that node.
 

diff --git a/docs/graphs/node_attributes/zarr_attribute.rst b/docs/graphs/node_attributes/zarr_attribute.rst
@@ -0,0 +1,18 @@
+###################
+ From Zarr dataset
+###################
+
+Zarr datasets are the standard format to define data nodes in
+_anemoi-graphs_. The user can define node attributes based on a zarr
+dataset variable. For example, the following recipe will define an
+attribute `land_mask` based on the _lsm_ variable of the dataset.
+
+.. literalinclude:: ../yaml/attributes_nonmissingzarr.yaml
+   :language: yaml
+
+In addition, if an user is using "cutout" operation to build their
+dataset, it may be helpful to create a `cutout_mask` to track the
+provenance of the resulting nodes. An example is shown below:
+
+.. literalinclude:: ../yaml/attributes_cutout.yaml
+   :language: yaml
diff --git a/docs/graphs/node_coordinates.rst b/docs/graphs/node_coordinates.rst
@@ -24,6 +24,7 @@ a file:
 
    node_coordinates/zarr_dataset
    node_coordinates/npz_file
+   node_coordinates/icon_mesh
 
 or based on other algorithms. A commonn approach is to use an
 icosahedron to project the earth's surface, and refine it iteratively to

diff --git a/docs/graphs/node_coordinates/healpix.rst b/docs/graphs/node_coordinates/healpix.rst
@@ -10,7 +10,7 @@ to the number of refinements of the sphere.
 .. code:: yaml
 
    nodes:
-     data:
+     data:  # name of the nodes
        node_builder:
          _target_: anemoi.graphs.nodes.HEALPixNodes
          resolution: 3

diff --git a/docs/graphs/node_coordinates/hex_refined_icosahedron.rst b/docs/graphs/node_coordinates/hex_refined_icosahedron.rst
@@ -4,32 +4,56 @@
 
 This method allows us to define the nodes based on the Hexagonal
 Hierarchical Geospatial Indexing System, which uses hexagons to divide
-the sphere. Each refinement level divides each hexagon into seven
-smaller hexagons.
+the sphere. With each refinement, each hexagon into seven smaller
+hexagons.
 
 To define the `node coordinates` based on the hexagonal refinements of
 an icosahedron, you can use the following YAML configuration:
 
+***************
+ Global graphs
+***************
+
+The class `HexNodes` allows us to define the nodes over the entire
+globe.
+
 .. code:: yaml
 
    nodes:
-     data:
+     hidden: # name of the nodes
        node_builder:
          _target_: anemoi.graphs.nodes.HexNodes
          resolution: 4
        attributes: ...
 
-where resolution is the number of refinements to be applied.
+where `resolution` is the number of refinements to be applied.
+
+*********************
+ Limited Area graphs
+*********************
+
+The class `LimitedAreaHexNodes` allows us to define the nodes only for a
+specific area of interest.
+
+.. code:: yaml
+
+   nodes:
+     hidden: # name of the nodes
+       node_builder:
+         _target_: anemoi.graphs.nodes.LimitedAreaHexNodes
+         resolution: 4
+         reference_node_name: nodes_name
+         mask_attr_name: mask_name  # optional
+         margin_radius_km: 100  # optional
+       attributes: ...
+
+where `reference_node_name` is the name of the nodes to define the area
+of interest.
 
 .. csv-table:: Hexagonal Hierarchical refinements specifications
    :file: ./hex_refined.csv
    :header-rows: 1
 
-Note that the refinement level is the parameter used to control the
-resolution of the nodes, but the resolution also depends on the
-refinement method. Then, for the same refinement level, ``HexNodes``
-will have a higher resolution than ``TriNodes``.
-
 .. warning::
 
    This class will require the `h3 <https://h3.org>`_ package to be