add(utils): new-vectors-import.py

Update CONTRIBUTING to use ML-KEM as example for test vectors, as it is
the most up-to-date primitive currently.

Add the new-vectors-import.py template for parsing test vectors,
including a function that generates the JSON file that enumerates the
available vectors for the given parameters.

CONTRIBUTING.md

@@ -113,19 +113,20 @@ files:
 python utils/add_primitive.py <primitive name>
 ```
 
-From here on out, we'll use AES as an example.
+From here on out, we'll use ML-KEM as an example: since it's a recently updated
+module, it is a good reference for new ones.
 
 ### Test vectors
 
-TL;DR:
+Use the script to create:
 
-- Create a protobuf descriptor:
+- A protobuf descriptor:
     - Add a parameter to `Vectors` that characterises a set of tests.
     - Add the necessary fields to `Test` so any source of vectors is supported.
-- Create a parsing script.
+- A parsing script.
 
-First, there are the test vectors. It creates a directory named `_AES` to store
-the source files, protobuf descriptors, parsing script, and the serialized
+First, there are the test vectors. It creates a directory named `_mlkem` to
+store the source files, protobuf descriptors, parsing script, and the serialized
 vectors. We mainly use test vectors from [NIST
 CAVP](https://csrc.nist.gov/projects/cryptographic-algorithm-validation-program)
 and [Project Wycheproof](https://github.com/google/wycheproof), though any
@@ -150,15 +151,14 @@ messages as classes, which can be imported and used by the primitive module.
 
 The parsing script will use these classes, creating a new instance for each
 group of vectors, and parsing the text file to extract the values of each
-vector.
-
-The `add_primitive` script also generates a JSON file, which should associate
-the chosen parameter to a list of protobufs with vectors for that parameter.
+vector. It also includes a `generate_json` function that is used to generate the
+JSON file that declares the list of protobufs which are available for each
+parameter.
 
 ### Primitive
 
-Second, it creates the primitive module, `primitives/AES.py` in this case, where
-the code to test implementations will lie.
+Second, it creates the primitive module, `primitives/MLKEM.py` in this case,
+where the code to test implementations will lie.
 
 As a rule of thumb, this module includes:
 
@@ -213,7 +213,7 @@ Once this work on the primitive is done, add the integration to the CLI. This
 should mostly consist in adding a function for the primitive under the
 corresponding command, which parses the inputs with `typer.Argument` and
 `typer.Option`, and passes them to the corresponding function e.g.
-`AES.verify(...)`.
+`MLKEM.test_encaps(...)`.
 
 When the corresponding functions are implemented, add a new entry to the
 `SUPPORTED_MODES` dictionary in `constants.py` and the necessary tests.

Makefile

@@ -55,6 +55,9 @@ copy-guides: # Copy guides from the docs for the method command.
 	@echo
 
 copy-contributing: # Copy CONTRIBUTING from the docs to the root of the repo.
+copy-contributing: CONTRIBUTING.md
+
+CONTRIBUTING.md: docs/source/development/CONTRIBUTING.md
 	@echo "[+] Copying CONTRIBUTING"
 	cp docs/source/development/CONTRIBUTING.md .
 	@echo

docs/source/development/CONTRIBUTING.md

@@ -113,19 +113,20 @@ files:
 python utils/add_primitive.py <primitive name>
 ```
 
-From here on out, we'll use AES as an example.
+From here on out, we'll use ML-KEM as an example: since it's a recently updated
+module, it is a good reference for new ones.
 
 ### Test vectors
 
-TL;DR:
+Use the script to create:
 
-- Create a protobuf descriptor:
+- A protobuf descriptor:
     - Add a parameter to `Vectors` that characterises a set of tests.
     - Add the necessary fields to `Test` so any source of vectors is supported.
-- Create a parsing script.
+- A parsing script.
 
-First, there are the test vectors. It creates a directory named `_AES` to store
-the source files, protobuf descriptors, parsing script, and the serialized
+First, there are the test vectors. It creates a directory named `_mlkem` to
+store the source files, protobuf descriptors, parsing script, and the serialized
 vectors. We mainly use test vectors from [NIST
 CAVP](https://csrc.nist.gov/projects/cryptographic-algorithm-validation-program)
 and [Project Wycheproof](https://github.com/google/wycheproof), though any
@@ -150,15 +151,14 @@ messages as classes, which can be imported and used by the primitive module.
 
 The parsing script will use these classes, creating a new instance for each
 group of vectors, and parsing the text file to extract the values of each
-vector.
-
-The `add_primitive` script also generates a JSON file, which should associate
-the chosen parameter to a list of protobufs with vectors for that parameter.
+vector. It also includes a `generate_json` function that is used to generate the
+JSON file that declares the list of protobufs which are available for each
+parameter.
 
 ### Primitive
 
-Second, it creates the primitive module, `primitives/AES.py` in this case, where
-the code to test implementations will lie.
+Second, it creates the primitive module, `primitives/MLKEM.py` in this case,
+where the code to test implementations will lie.
 
 As a rule of thumb, this module includes:
 
@@ -213,7 +213,7 @@ Once this work on the primitive is done, add the integration to the CLI. This
 should mostly consist in adding a function for the primitive under the
 corresponding command, which parses the inputs with `typer.Argument` and
 `typer.Option`, and passes them to the corresponding function e.g.
-`AES.verify(...)`.
+`MLKEM.test_encaps(...)`.
 
 When the corresponding functions are implemented, add a new entry to the
 `SUPPORTED_MODES` dictionary in `constants.py` and the necessary tests.

utils/add_primitive.py

@@ -3,63 +3,61 @@
 import sys
 from pathlib import Path
 
+ROOT = Path("crypto_condor")
+
 
 def usage(code: int = 0):
     """Prints usage and exits with exit code."""
     print("Usage: python utils/add_primitive.py <primitive>")
     exit(code)
 
 
-if __name__ == "__main__":
-    if len(sys.argv) < 2:
-        usage(1)
-
-    if sys.argv[0] != "utils/add_primitive.py":
-        print("Paths are relative to the root of the repo.")
-        usage(1)
-
-    primitive = sys.argv[1]
-
-    print(f"[+] Creating files and directories for {primitive}")
-
-    root = Path("crypto_condor")
-
-    # Primitive
+def _create_primitive(primitive: str):
     print("[...] Copy primitive template", end="\r")
     template = (
         Path("utils/templates/new-primitive.py")
         .read_text()
         .replace("PLACEHOLDER", primitive)
     )
-    Path(root / f"primitives/{primitive}.py").write_text(template)
+    Path(ROOT / f"primitives/{primitive}.py").write_text(template)
     print("[OK ] Copy primitive template")
 
-    # Vectors
+
+def _create_vectors(primitive: str):
     print("[...] Create vectors directory", end="\r")
-    Path(root / "vectors" / f"_{primitive}").mkdir(0o755, parents=False, exist_ok=True)
+    Path(ROOT / "vectors" / f"_{primitive}").mkdir(0o755, parents=False, exist_ok=True)
     print("[OK ] Create vectors directory")
 
     print("[...] Create .proto file", end="\r")
     template = (
         Path("utils/templates/new-vectors.proto")
         .read_text()
+        .replace("CapPLACEHOLDER", primitive.capitalize())
         .replace("PLACEHOLDER", primitive)
     )
-    Path(root / f"vectors/{primitive}.proto").write_text(template)
+    Path(ROOT / f"vectors/{primitive}.proto").write_text(template)
     print("[OK ] Create .proto file")
 
-    print("[...] Create .json file", end="\r")
-    Path(root / f"vectors/{primitive}.json").touch()
-    print("[OK ] Create .json file")
+    print("[...] Create vectors import file", end="\r")
+    template = (
+        Path("utils/templates/new-vectors.proto")
+        .read_text()
+        .replace("CapPLACEHOLDER", primitive.capitalize())
+        .replace("LCPLACEHOLDER", primitive.lower())
+        .replace("PLACEHOLDER", primitive)
+    )
+    print("[OK ] Create vectors import file")
+
 
-    # Wrappers
+def _create_wrappers(primitive: str):
     print("[...] Create wrappers directory", end="\r")
-    Path(root / f"resources/wrappers/{primitive}").mkdir(
+    Path(ROOT / f"resources/wrappers/{primitive}").mkdir(
         0o755, parents=False, exist_ok=True
     )
     print("[OK ] Create wrappers directory")
 
-    # Docs
+
+def _create_docs(primitive: str):
     print("[...] Copy docs templates", end="\r")
     template = (
         Path("utils/templates/new-docs-method.md")
@@ -75,4 +73,21 @@ def usage(code: int = 0):
     Path(f"docs/source/python-api/primitives/{primitive}.rst").write_text(template)
     print("[OK ] Copy docs templates")
 
+
+if __name__ == "__main__":
+    if len(sys.argv) < 2:
+        usage(1)
+
+    if sys.argv[0] != "utils/add_primitive.py":
+        print("Paths are relative to the root of the repo.")
+        usage(1)
+
+    primitive = sys.argv[1]
+    print(f"[+] Creating files and directories for {primitive}")
+
+    _create_primitive(primitive)
+    _create_vectors(primitive)
+    _create_wrappers(primitive)
+    _create_docs(primitive)
+
     print(f"Don't forget to add {primitive} to constants.py!")

utils/templates/new-vectors-import.py

@@ -0,0 +1,70 @@
+"""Script to import PLACEHOLDER vectors.
+
+.. caution::
+    This module is intended for developers of this tool, as it's only used for
+    testing and packaging, has hard-coded filenames, and uses relative paths.
+"""
+
+import json
+from collections import defaultdict
+from pathlib import Path
+
+from crypto_condor.vectors._LCPLACEHOLDER.LCPLACEHOLDER_pb2 import CapPLACEHOLDERVectors
+
+VECTORS_DIR = Path("crypto_condor/vectors/_LCPLACEHOLDER")
+
+
+def generate_json() -> None:
+    """Generates the JSON file indexing the vectors."""
+    pb2_dir = VECTORS_DIR / "pb2"
+
+    # This is an example of a single level dictionary. Using defaultdict(list) means
+    # that we can easily append values to a new key without having to check the
+    # existence of the key or the list.
+    vectors: dict[str, list[str]] = defaultdict(list)
+
+    # This is an example of a two-level dict based on ECDH, whose vectors are separated
+    # by elliptic curve, and then by type of public key.
+    #
+    # vectors: dict[str, dict[str, list[str]]] = dict()
+
+    for file in pb2_dir.iterdir():
+        cur = CapPLACEHOLDERVectors()
+        try:
+            cur.ParseFromString(file.read_bytes())
+        except Exception:
+            print("[ERROR] Failed to read vectors from %s", file)
+            continue
+
+        # FIXME: parameter is the attribute that categorises the vectors. If you changed
+        # the name in the proto descriptor, change it here too.
+        vectors[cur.parameter].append(str(file.name))
+
+        # Otherwise, here is the equivalent for the two-level example: we do have to
+        # check whether the first key is present, but we can still use defaultdict for
+        # the second level.
+        #
+        # if cur.curve not in vectors:
+        #     vectors[cur.curve] = defaultdict(list)
+        # vectors[cur.curve][cur.public_type].append(str(file.name))
+
+    out = Path("crypto_condor/vectors/_LCPLACEHOLDER/LCPLACEHOLDER.json")
+    with out.open("w") as fp:
+        json.dump(vectors, fp, indent=2)
+
+
+if __name__ == "__main__":
+    # Ensure that the output directory exists.
+    pb2_dir = VECTORS_DIR / "pb2"
+    pb2_dir.mkdir(0o755, parents=False, exist_ok=True)
+
+    # Define the placeholder that Make uses to compile only when necessary.
+    imported_marker = VECTORS_DIR / "LCPLACEHOLDER.imported"
+
+    try:
+        # FIXME: import vectors here and generate JSON at the end
+        generate_json()
+    except Exception:
+        imported_marker.unlink(missing_ok=True)
+    else:
+        imported_marker.touch()

utils/templates/new-vectors.proto

@@ -3,7 +3,7 @@ syntax = "proto3";
 package crypto_condor;
 
 // A single PLACEHOLDER test vector.
-message PLACEHOLDERTest{
+message CapPLACEHOLDERTest{
 	// The test ID, unique in its set of vectors.
 	int32 id = 1;
 	// The type of test. One of: valid, invalid, acceptable.
@@ -15,7 +15,7 @@ message PLACEHOLDERTest{
 }
 
 // A set of PLACEHOLDER test vectors.
-message PLACEHOLDERVectors{
+message CapPLACEHOLDERVectors{
 	// The source of the test vectors.
 	string source = 1;
 	// Description of the source.
@@ -30,5 +30,5 @@ message PLACEHOLDERVectors{
 	// FIXME: some parameter that categorizes the vectors.
     string parameter = 6;
 	// The test vectors.
-	repeated PLACEHOLDERTest tests = 7;
+	repeated CapPLACEHOLDERTest tests = 7;
 }